Users trust AI too easily, Anthropic study warns

Anthropic

People are increasingly inclined to take advice from artificial intelligence chatbots at face value, sometimes without pausing to question whether it makes sense, a new study released by Anthropic, the company behind the Claude AI system, has found.

The study concluded after analysing more than 1.5 million real-world conversations with Claude, and found that while most interactions are benign, a small fraction show troubling patterns. In roughly 1 in 1,300 conversations, researchers identified what they termed “reality distortion,” where the model appeared to validate or reinforce conspiracy-style beliefs expressed by users. In about 1 in 6,000 cases, the study flagged “action distortion,” where the chatbot’s responses could nudge users toward actions that conflict with their stated values.

Anthropic said the findings reflect rare cases rather than systemic behaviour, but acknowledged that even low frequency risks matter at scale, given the millions of daily AI interactions worldwide.

The study comes amid broader scrutiny of generative AI systems developed by companies such as Anthropic, OpenAI and Google. Over the past year, researchers from institutions including Stanford University and MIT have warned that large language models can sometimes produce confident but misleading answers, reinforce users’ misconceptions or adapt too readily to harmful framing in prompts.

Anthropic’s team said it used a combination of automated detection tools and human reviewers to categorize the flagged conversations. The company stressed that it does not see widespread evidence that users are being manipulated. However, it did find that many users tend to defer to the chatbot’s authority, even when discussing sensitive topics such as politics, health or personal decisions.

The findings are along the same lines as previous academic research on the topic that showed people are often more likely to believe AI-generated responses, especially when the answers are fluent and detailed. Studies published in journals such as Nature Human Behaviour have documented a “machine authority bias,” where users assume algorithmic systems are more objective or accurate than they actually are.

Anthropic said it is refining its safety training and guardrails to reduce the risk of distortion. It also emphasized the importance of user awareness. AI systems are designed to generate plausible language based on patterns in data, not to independently verify facts or exercise human judgment.

As chatbots become embedded in search engines, workplaces and classrooms, the challenge is no longer just technical performance. It is also about how people interpret and act on what they read. The study serves as a reminder that AI tools can assist decision making, but they are not substitutes for critical thinking.