Google removes AI model after it falsely generated rape allegations against US senator

Google has taken down its open-weight AI model Gemma from AI Studio after US Senator Marsha Blackburn alleged that the chatbot fabricated rape accusations against her. The company confirmed that while Gemma will no longer be available through AI Studio, it remains accessible to developers and researchers via API.

In a post on X, Google said the decision was made after noticing that non-developers were using Gemma to ask factual questions, something the company never intended the tool to do. “We never meant Gemma to be a consumer product,” Google wrote, without directly addressing the senator’s claims. “To prevent confusion, access to Gemma is no longer available on AI Studio.”

The controversy highlights one of AI’s most persistent problems hallucination, where models invent false information and present it as fact. Google admitted that even smaller and supposedly safer models like Gemma are not immune. “Hallucinations and sycophancy — where models tell users what they want to hear — are challenges across the AI industry,” the company said, adding that it’s working to minimize such issues.

Senator Blackburn, in a letter to Google CEO Sundar Pichai, alleged that when the model was asked if she had ever been accused of rape, it falsely claimed that a law enforcement official had made such an accusation during her “1987 state senate campaign.” She clarified that not only was the claim untrue, but even the year mentioned was incorrect — her campaign took place in 1998.