AI school safety tool sparks backlash after flagging art as porn and deleting emails

Representative image

In 2023, Lawrence High School in Kansas installed Gaggle Safety Management, an artificial intelligence surveillance system that scans student emails, papers, and uploads for signs of risky behaviour. For a price as a way to keep students safe from harm like drug activity, cutting oneself, or school violence, Gaggle automatically forwards suspicious material to school officials. The district signed a three-year, $160,000 contract, one of over 1,500 school districts in the country utilizing the technology, the Washington Post reported.

Students complain it is too draconian

Students quickly discovered that Gaggle was marking much more than it ought to. One art student described how some of her photography portfolio was deleted when the system mislabelled a portrait of girls in tank tops as child pornography. Another student was questioned after making a tongue-in-cheek comment in an email that they were "gonna die" from having to give a fitness test in Crocs. Former student journalist Suzana Kennedy said even her public-records request for data about Gaggle was caught by the system itself, preventing her from getting the reply.

Chilling effect on expression

Critics say the software is having a chilling effect on expression. Some students say they hesitate to write essays or emails, worrying that innocuous expressions might prompt faculty or administrators to scrutinize them. Former student Natasha Torkzaban referred to it as an ongoing question, "Who else, other than me, is reading this document?"

August lawsuits allege surveillance is illegal, open to erroneous shots, and prevents students from being able to engage freely in discussions on topics of a sensitive nature such as mental illness.

School officials justify the program

The Lawrence school board has defended its action, arguing that the tool has also helped save lives by preventing suicides and avoiding potentially lethal circumstances. Last year, then-superintendent Anthony Lewis told us in a statement, "The information that we have learned through the use of Gaggle has allowed our staff to intervene and save lives." The district defends the system as one that is meant to detect such risks as depression, self-harm, cyberbullying, and threats of violence that are considered credible.

Gaggle itself has acknowledged in public statements that it strikes a balance between safety and privacy.

Concerns for privacy and safety increase

Inquiries into Gaggle have uncovered more universal dangers. A joint probe by the Seattle Times and the Associated Press found that content identified as having offensive material was temporarily published online in unsecured form, raising data security issues. LGBTQ students at times were publicly made available to parents or school officials when the system detected conversations about their sexuality.

The Kansas lawsuit also claims that "trained safety professionals" who reviewed the flagged content were outsourced contractors, which has raised questions about who viewed individual student data.

Examples of overreach

Records provided to students showed that from November 2023 to September 2024, Gaggle flagged more than 1,200 items of student content in the district. Some 800 were deemed "nonissues." One student was checked for having "I wanted to kill" in an email only because she was describing to her grandmother something she wanted her to kill, namely a fly.

Other words tagged as suspicious were "sex," "drunk," "get in a fight," and "bomb." Despite 18 cases being referred to police, the overwhelming majority of the reported cases were trivial or misinterpreted language.

Balancing safety and freedom

Education experts say AI monitoring systems can be helpful assistance for overburdened school staff but caution against overdoing it. Amanda Klinger of the Educator's School Safety Network clarified that the software can detect danger but could ultimately cause students to feel too monitored. For most of Lawrence's student body, the scales tip: instead of feeling safer, they feel monitored, questioned, and muzzled.

As the suit continues, it will test whether schools can justify mass monitoring in the name of safety—or at what cost in terms of privacy and student trust.