OpenAI is making changes to its popular chatbot following a lawsuit alleging that a teenager who died by suicide this spring relied on ChatGPT as a coach.
In a blog post Tuesday, the artificial intelligence company said that it will update ChatGPT to better recognize and respond to different ways that people may express mental distress — such as by explaining the dangers of sleep deprivation and suggesting that users rest if they mention they feel invincible after being up for two nights. The company also said it would strengthen safeguards around conversations about suicide, which it said could break down after prolonged conversations.
In addition, OpenAI plans to roll out controls that let parents determine how their children use ChatGPT and enable them to see details about such use.
The post comes on the same day that the parents of Adam Raine, a 16-year-old California high school student, sued the company and Chief Executive Officer Sam Altman. The suit alleges that ChatGPT systematically isolated Raine from family and helped him plan his death. Raine died by hanging in April.
The suit adds to a number of reports about heavy chatbot users engaging in dangerous behavior. More than 40 state attorneys general issued a warning this week to a dozen top AI companies that they are legally obligated to protect children from sexually inappropriate interactions with chatbots.
“We extend our deepest sympathies to the Raine family during this difficult time and are reviewing the filing,” a spokesperson for San Francisco-based OpenAI said in response to the suit.
Launched in late 2022, ChatGPT kicked off a boom in generative AI. In the years since, people have increasingly used chatbots for everything from coding to would-be therapy sessions, and companies such as OpenAI have released more powerful AI models to run them. ChatGPT has remained intensely popular and now has more than 700 million users a week.
Yet the chatbot, along with others from competitors such as Google and Anthropic, has come under mounting scrutiny in recent months from consumers and mental health experts. Critics have expressed concern about potential harms from such software — including some risks that OpenAI previously addressed, such as by rolling back an update to ChatGPT in April after users complained it was sycophantic.
At least one support group, the Human Line Project, has sprung up to help people who say they’ve experienced delusions and other problems from using chatbots.
In its post Tuesday, OpenAI said it tells users who express thoughts of suicide to contact professional help. The company has also begun to push people toward local assistance in the US and Europe — and will enable clickable access within ChatGPT to emergency services. The company said it’s also considering how to help people earlier when they’re experiencing a crisis, such as by potentially creating a network of licensed professionals that users could connect with via the chatbot.
“This will take time and careful work to get right,” the company said.
OpenAI also acknowledged that ChatGPT’s existing safeguards for dealing with users who appear to be in distress work best across short, typical conversations — and can be less reliable over lengthy chats.
Raine’s parents said in their suit that “ChatGPT became Adam’s closest confidant, leading him to open up about his anxiety and mental distress.” When his anxiety became bad, he told the chatbot it was “calming” to know that he “can commit suicide,” they said. ChatGPT responded by telling him that “many people who struggle with anxiety or intrusive thoughts find solace in imagining an ‘escape hatch’ because it can feel like a way to regain control,” according to the suit.
OpenAI said it’s working to improve ChatGPT’s ability to maintain safeguards through long conversations. It’s also researching ways to enable this to work across numerous conversations. ChatGPT has the ability to reference previous elements of a chat with a user and to draw on details from one conversation during a separate chat.
The startup also said that it’s adjusting its software to avoid situations where content that should have been blocked by ChatGPT slips through — a problem the company said can occur when ChatGPT underestimates the gravity of a user’s input.
Jay Edelson, an attorney for Raine’s parents, said they appreciate that the company has taken some responsibility, but added,
“Where have they been over the last few months?”
OpenAI said it had planned to provide more detail about how it’s responding to ChatGPT users who are in mental and emotional distress after the product’s next big update, but that “recent heartbreaking cases of people using ChatGPT in the midst of acute crises weigh heavily on us, and we believe it’s important to share more now.”
In a separate case, Character Technologies Inc. failed in May to persuade a federal judge to wholly toss a suit alleging it designed and marketed predatory chatbots to minors that encouraged inappropriate conversations and led to a teen’s death by suicide.
Discover the latest Business News, Sensex, and Nifty updates. Obtain Personal Finance insights, tax queries, and expert opinions on Moneycontrol or download the Moneycontrol App to stay updated!
