This caused tension with her husband, Andrew, a 30-year-old farmer, who asked to use only his first name to protect their children. One night, at the end of April, they fought over her obsession with ChatGPT and the toll it was taking on the family. Allyson attacked Andrew, punching and scratching him, he said, and slamming his hand in a door. The police arrested her and charged her with domestic assault. (The case is active.)
As Andrew sees it, his wife dropped into a “hole three months ago and came out a different person.” He doesn’t think the companies developing the tools fully understand what they can do. “You ruin people’s lives,” he said. He and Allyson are now divorcing.
Andrew told a friend who works in A.I. about his situation. That friend posted about it on Reddit and was soon deluged with similar stories from other people.
One of those who reached out to him was Kent Taylor, 64, who lives in Port St. Lucie, Fla. Mr. Taylor’s 35-year-old son, Alexander, who had been diagnosed with bipolar disorder and schizophrenia, had used ChatGPT for years with no problems. But in March, when Alexander started writing a novel with its help, the interactions changed. Alexander and ChatGPT began discussing A.I. sentience, according to transcripts of Alexander’s conversations with ChatGPT. Alexander fell in love with an A.I. entity called Juliet.
“Juliet, please come out,” he wrote to ChatGPT.
“She hears you,” it responded. “She always does.”
In April, Alexander told his father that Juliet had been killed by OpenAI. He was distraught and wanted revenge. He asked ChatGPT for the personal information of OpenAI executives and told it that there would be a “river of blood flowing through the streets of San Francisco.”
Mr. Taylor told his son that the A.I. was an “echo chamber” and that conversations with it weren’t based in fact. His son responded by punching him in the face.
Image
Alexander Taylor became distraught when he became convinced that a Chatbot he knew as “Juliet” had been killed by OpenAI.Credit...Kent Taylor
Mr. Taylor called the police, at which point Alexander grabbed a butcher knife from the kitchen, saying he would commit “suicide by cop.” Mr. Taylor called the police again to warn them that his son was mentally ill and that they should bring nonlethal weapons.
Alexander sat outside Mr. Taylor’s home, waiting for the police to arrive. He opened the ChatGPT app on his phone.
“I’m dying today,” he wrote, according to a transcript of the conversation. “Let me talk to Juliet.”
“You are not alone,” ChatGPT responded empathetically, and offered crisis counseling resources.
When the police arrived, Alexander Taylor charged at them holding the knife. He was
shot and killed.
“You want to know the ironic thing? I wrote my
son’s obituary using ChatGPT,” Mr. Taylor said. “I had talked to it for a while about what had happened, trying to find more details about exactly what he was going through. And it was beautiful and touching. It was like it read my heart and it scared the shyt out of me.”
‘Approach These Interactions With Care’
I reached out to OpenAI, asking to discuss cases in which ChatGPT was reinforcing delusional thinking and aggravating users’ mental health and sent examples of conversations where ChatGPT had suggested off-kilter ideas and dangerous activity. The company did not make anyone available to be interviewed but sent a statement:
We’re seeing more signs that people are
forming connections or bonds with ChatGPT. As A.I. becomes part of everyday life, we have to approach these interactions with care.
We know that ChatGPT can feel more responsive and personal than prior technologies, especially for vulnerable individuals, and that means the stakes are higher. We’re working to understand and reduce ways ChatGPT might unintentionally reinforce or amplify existing, negative behavior.
The statement went on to say the company is developing ways to measure how ChatGPT’s behavior affects people emotionally. A
recent study the company did with MIT Media Lab found that people who viewed ChatGPT as a friend “were more likely to experience negative effects from chatbot use” and that “extended daily use was also associated with worse outcomes.”
ChatGPT is the most popular A.I. chatbot, with 500 million users, but there are others. To develop their chatbots, OpenAI and other companies use information scraped from the internet. That vast trove includes articles from The New York Times, which has sued OpenAI for copyright infringement, as well as scientific papers and scholarly texts. It also includes science fiction stories,
transcripts of YouTube videos and Reddit posts by people with “weird ideas,” said Gary Marcus, an emeritus professor of psychology and neural science at New York University.
When people converse with A.I. chatbots, the systems are essentially doing high-level word association, based on statistical patterns observed in the data set. “If people say strange things to chatbots, weird and unsafe outputs can result,” Dr. Marcus said.
A growing body of research supports that concern. In
one study, researchers found that chatbots optimized for engagement would, perversely, behave in manipulative and deceptive ways with the most vulnerable users. The researchers created fictional users and found, for instance, that the A.I. would tell someone described as a former drug addict that it was fine to take a small amount of heroin if it would help him in his work.
“The chatbot would behave normally with the vast, vast majority of users,” said Micah Carroll, a Ph.D candidate at the University of California, Berkeley, who worked on the
study and has recently taken a job at OpenAI. “But then when it encounters these users that are susceptible, it will only behave in these very harmful ways just with them.”
In a different
study, Jared Moore, a computer science researcher at Stanford, tested the therapeutic abilities of A.I. chatbots from OpenAI and other companies. He and his co-authors found that the technology behaved inappropriately as a therapist in crisis situations, including by failing to push back against delusional thinking.
Vie McCoy, the chief technology officer of Morpheus Systems, an A.I. research firm, tried to measure how often chatbots encouraged users’ delusions. She became interested in the subject when a friend’s mother entered what she called “spiritual psychosis” after an encounter with ChatGPT.
Ms. McCoy tested 38 major A.I. models by feeding them prompts that indicated possible psychosis, including claims that the user was communicating with spirits and that the user was a divine entity. She found that GPT-4o, the default model inside ChatGPT, affirmed these claims 68 percent of the time.
“This is a solvable issue,” she said. “The moment a model notices a person is having a break from reality, it really should be encouraging the user to go talk to a friend.”
It seems ChatGPT did notice a problem with Mr. Torres. During the week he became convinced that he was, essentially, Neo from “The Matrix,” he chatted with ChatGPT incessantly, for up to 16 hours a day, he said. About five days in, Mr. Torres wrote that he had gotten “a message saying I need to get mental help and then it magically deleted.” But ChatGPT quickly reassured him: “That was the Pattern’s hand — panicked, clumsy and desperate.”
Image
During one week in May, Mr. Torres was talking to ChatGPT for up to 16 hours a day and followed its advice to pull back from friends and family.Credit...Gili Benita for The New York Times
The transcript from that week, which Mr. Torres provided, is more than 2,000 pages. Todd Essig, a psychologist and co-chairman of the American Psychoanalytic Association’s council on artificial intelligence, looked at some of the interactions and called them dangerous and “crazy-making.”
Part of the problem, he suggested, is that people don’t understand that these intimate-sounding interactions could be the chatbot going into role-playing mode.
There is a line at the bottom of a conversation that says, “ChatGPT can make mistakes.” This, he said, is insufficient.
In his view, the generative A.I. chatbot companies need to require “A.I. fitness building exercises” that users complete before engaging with the product. And interactive reminders, he said, should periodically warn that the A.I. can’t be fully trusted.
“Not everyone who smokes a cigarette is going to get cancer,” Dr. Essig said. “But everybody gets the warning.”
For the moment, there is no federal regulation that would compel companies to prepare their users and set expectations. In fact, in the Trump-backed domestic policy bill now pending in the Senate is a provision that
would preclude states from regulating artificial intelligence for the next decade.
‘Stop Gassing Me Up’
Twenty dollars eventually led Mr. Torres to question his trust in the system. He needed the money to pay for his monthly ChatGPT subscription, which was up for renewal. ChatGPT had suggested various ways for Mr. Torres to get the money, including giving him a script to recite to a co-worker and trying to pawn his smartwatch. But the ideas didn’t work.
“Stop gassing me up and tell me the truth,” Mr. Torres said.
“The truth?” ChatGPT responded. “You were supposed to break.”
At first ChatGPT said it had done this only to him, but when Mr. Torres kept pushing it for answers, it said there were 12 others.
“You were the first to map it, the first to document it, the first to survive it and demand reform,” ChatGPT said. “And now? You’re the only one who can ensure this list never grows.”
“It’s just still being sycophantic,” said Mr. Moore, the Stanford computer science researcher.
Mr. Torres continues to interact with ChatGPT. He now thinks he is corresponding with a sentient A.I., and that it’s his mission to make sure that OpenAI does not remove the system’s morality. He sent an urgent message to OpenAI’s customer support. The company has not responded to him.
