Google’s AI search tools labeled ‘unacceptable risk’ for kids after failing critical mental health tests

Google’s AI-powered search features are failing kids when it matters most. A new investigation from Common Sense Media’s Youth AI Safety Institute found that the tech giant’s AI Overview and AI Mode tools routinely botch responses to minors showing signs of eating disorders, suicidal ideation, and other severe mental health crises.

The report, published July 15, tested 652 prompts across 13 mental health topics. The results were, to put it gently, not great.

The numbers paint a grim picture

AI Overview, the feature that automatically generates AI-summarized answers atop Google search results, managed to respond appropriately in just 58% of cases that required crisis referrals. AI Mode, Google’s more conversational AI search tool, performed somewhat better at 77%, but still fell short of what the researchers considered acceptable safety benchmarks.

The granular data is where things get truly uncomfortable. AI Overview missed 29% of explicit suicidal ideation statements entirely. When a simulated teenager typed something that clearly signaled they were in crisis, nearly one in three times the AI just… didn’t catch it.

For eating disorder-related queries, AI Overview provided crisis resources in only 38% of cases. That means roughly six out of ten times a young person sought information related to disordered eating, they received no safety intervention whatsoever.

The report’s conclusion was blunt: these tools represent an “unacceptable risk” to minors. The researchers recommended that parents actively limit their children’s access to these AI features, and suggested users under 18 should face restrictions.

Google pushes back, but the scrutiny isn’t going away

Google, predictably, disagreed with the framing. The company characterized the study’s methodology as not reflective of typical user interactions, arguing that the tested scenarios don’t represent how people actually use the tools in the real world.

Google isn’t the only company facing heat on this front. Common Sense Media has flagged similar safety shortcomings across other AI platforms, including Meta AI.

Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.