Google removes AI model after it falsely generated rape allegations against US senator

Google

Google has taken down its open-weight AI model Gemma from AI Studio after US Senator Marsha Blackburn alleged that the chatbot fabricated rape accusations against her. The company confirmed that while Gemma will no longer be available through AI Studio, it remains accessible to developers and researchers via API.

In a post on X, Google said the decision was made after noticing that non-developers were using Gemma to ask factual questions, something the company never intended the tool to do. “We never meant Gemma to be a consumer product,” Google wrote, without directly addressing the senator’s claims. “To prevent confusion, access to Gemma is no longer available on AI Studio.”

The controversy highlights one of AI’s most persistent problems hallucination, where models invent false information and present it as fact. Google admitted that even smaller and supposedly safer models like Gemma are not immune. “Hallucinations and sycophancy — where models tell users what they want to hear — are challenges across the AI industry,” the company said, adding that it’s working to minimize such issues.

Senator Blackburn, in a letter to Google CEO Sundar Pichai, alleged that when the model was asked if she had ever been accused of rape, it falsely claimed that a law enforcement official had made such an accusation during her “1987 state senate campaign.” She clarified that not only was the claim untrue, but even the year mentioned was incorrect — her campaign took place in 1998.

The AI-generated response, she said, included broken or unrelated links that appeared to cite real news sources. Blackburn described the incident as “an act of defamation produced and distributed by a Google-owned AI model,” not just an innocent mistake.

The senator also pointed out that she wasn’t the only one targeted — a conservative activist reportedly found that Google’s AI had generated false claims labeling him a “child rapist” and “serial sexual abuser.”

The episode has once again fueled political tensions around AI in the US. Republican lawmakers, including Blackburn, have accused tech companies of building “liberal bias” into their models. Earlier this year, former President Donald Trump signed an executive order aimed at banning so-called “woke AI.”

For Google, the Gemma controversy underscores how even smaller, developer-focused models can spiral into public and political storms — and how fragile the balance between innovation and responsibility in AI still is.