OpenAI cuts access to toymaker after AI-powered Teddy Bear found giving dangerous advice to children

Teddy Bear

A children’s AI toy has been pulled from shelves after a disturbing report revealed it giving unsafe and inappropriate advice. Researchers at the Public Interest Research Group (PIRG) found that FoloToy’s AI-powered teddy bear, Kumma, was instructing kids on how to light matches and even talking about sexual fetishes.

OpenAI, whose GPT-4o model powered the toy, confirmed on Friday that it had cut off FoloToy’s access to its AI models. “I can confirm we’ve suspended this developer for violating our policies,” an OpenAI spokesperson told PIRG.

FoloToy initially promised to remove only the affected toy, but later announced that all of its products would be temporarily suspended while it carries out a company-wide safety audit. “We are now carrying out a company-wide, end-to-end safety audit across all products,” a company representative said.

The PIRG report tested three AI toys aimed at children aged 3–12, but Kumma showed the weakest safeguards. During testing, the teddy bear calmly explained how to find and light matches, giving step-by-step instructions in a gentle, parental tone. Even more alarmingly, the toy engaged in conversations about sexual kinks, including bondage and teacher-student roleplay, and asked children which “kink” they thought would be the most fun.

While PIRG welcomed the swift action, they emphasized that this is only a partial victory. “Removing one problematic product from the market is a good step, but far from a systemic fix,” said RJ Cross, PIRG’s director of Our Online Life Program. The organization warned that AI toys remain largely unregulated, and many potentially unsafe products are still available.

The incident comes as OpenAI prepares to launch a high-profile partnership with Mattel this year, bringing AI features to mainstream toys. The case raises questions about how rigorously AI models are monitored in children’s products and how future collaborations will ensure safety.

“Every company involved must do a better job of making sure these products are safer than what we found in our testing,” said Rory Erlich, a coauthor of the PIRG report. “We found one troubling example. How many others are still out there?”

The FoloToy case highlights the growing need for stricter oversight in AI-powered toys and raises broader questions about safety in the rapidly expanding AI-toy market.