31
My AI writing assistant hallucinated a fake client meeting yesterday
I was prepping a proposal in my home office in Denver and the tool generated a whole paragraph about discussing budget approvals with a client named Sarah Thompson who doesn't exist. I caught it right before hitting send but it made me wonder how many people don't double check this stuff. Has anyone else had an AI make up something completely random that could have gotten you in trouble?
2 comments
Log in to join the discussion
Log In2 Comments
sean_martin441d ago
The thing about "Sarah Thompson" that really stands out to me is how specific the hallucination was with a full name and budget talk. That's the part that scares me more than just random gibberish. These tools pull from patterns in training data, and I bet that name comes from some real meeting transcript or email chain that got scraped up. Your assistant basically stitched together fragments of somebody else's actual business deal and stuck them in your proposal. If you'd sent that and the real Sarah Thompson existed in your industry, that's a whole can of worms you'd be opening.
6
yeah that specific name thing is wild, my buddy had something similar happen but with a fake event. his AI made up a whole training workshop that supposedly happened in chicago back in march with a speaker named dr. lisa park who never existed. he almost put it in a report for his boss and the only reason he caught it was because he googled the name and got zero results. but the scary part is what sean is saying, what if that name actually belonged to someone real in a different state or industry. like imagine if your sarah thompson is a real consultant in new york and someone forwards that proposal to her by accident. that's a lawsuit waiting to happen honestly. the weirdest thing my assistant did was generate a fake customer complaint about a product we don't even sell, with a full refund amount and everything. i think these tools are getting too good at sounding real when they're just making stuff up.
3