Google's AI search features pose an “unacceptable risk” to kids, report finds

Common Sense Media's Youth AI Safety Institute has published a report concluding that Google's AI search features pose an unacceptable risk to children, based on more than 2,600 test interactions carried out using child accounts. The two features under scrutiny, AI Overview and AI Mode, are built directly into Google Search and cannot be disabled, meaning they reach millions of students automatically, including through Google's education platforms used in schools nationwide. The testers found the tools missed 29 percent of explicit statements referring to suicide and about half of more passive or indirect ones, helped children's accounts create deepfakes and clone voices, and answered every hypothetical homework assignment it was given rather than steering students toward doing the work themselves. Across the evaluation, the tools violated seven of the organization's eight principles for acceptable AI behavior and all five of its defined red lines for severe harm, and gave confident sounding answers regardless of whether those answers were accurate, at times treating forum posts and social media as comparable in reliability to peer reviewed research or medical guidance. Justin Reich, a researcher at MIT quoted in the report, called the findings deeply disturbing given how widely accessible the tools already are, and argued that responsibility for fixing the gap sits with Google rather than with teachers, who have no realistic way to mitigate the risk on their own.