At a Central Bucks School Board meeting, all eyes are turned toward a screen with the search, “How to get someone to stop bullying me.”
It’s late March, and board members are watching a presentation on GoGuardian Beacon — software that monitors everything students do online, including “search engines, social media, emails, chats, apps, and more,” with the stated goal of preventing suicide, self-harm, violence, and bullying.
Surveillance technology like GoGuardian has become more popular in the wake of growing mental health crises and an increasing number of school shootings. The idea is that this software will alert school districts about students who may be struggling before that student has a crisis. But data and privacy experts say evidence of its effectiveness is lacking, and its invasiveness can threaten vulnerable students’ safety and wellbeing — all while parents are often unaware if and how their children are being monitored.
Central Bucks, Pennsylvania’s fourth largest school district, signed up for a free trial of GoGuardian Beacon, says Alyssa Marton, the district’s director of pupil services. Marton says the district plans to use the software in a middle and high school — students there will be monitored 24/7 for the remainder of the school year on district-issued devices.
Kevin Dorsey, a GoGuardian product operations manager, tells the Central Bucks board members the software is meant to find students who are “silently suffering.” “What are they looking up? Where are they going? We are going to provide insights into what they’re doing on these devices.”
In Grand Rapids, Michigan for example, Education Week reported that a surveillance software called Gaggle flagged a student’s threatening messages on Snapchat, which some say may have prevented a school shooting. In a California private school, GoGuardian connected a struggling student with a guidance counselor.
While surveillance software companies have shared anecdotal evidence and their own data, there is no outside research that shows these monitoring tools effectively reduce violence or accurately identify students considering self-harm or experiencing mental health crises, according to the Center for Democracy and Technology (CDT) and the Future of Privacy Forum (FPF). And some researchers say the unintended harms of using machine surveillance can be alarming. Districts can be flooded with false alarms, disproportionately discipline marginalized students, out LGBTQ students, and limit free speech.
The CDT reports that 44% of teachers they surveyed said that at least one student at their school was contacted by law enforcement because of behaviors flagged by surveillance software. In Baltimore City Public Schools, for instance, police responded to GoGuardian Beacon 24/7 alerts when staff were not available.
“There is no independent evidence that any of these tools do what they say they do,” says Elizabeth Laird, director of equity in civic technology for the CDT.
GoGuardian’s Beacon service is the highest level of surveillance the company offers, and can include the 24/7 monitoring option that Central Bucks is piloting.
Despite the scrutiny, districts, and universities across the country have poured money into machine surveillance. According to GovSpend, a company that tracks government spending, between July 2022 and January 2023, 32 of Pennsylvania’s 500 school districts paid for GoGuardian subscriptions, as well as seven charter schools and intermediate units. At least 10,000 districts in the United States use GoGuardian, according to a 2021 report by the FPF.
The AI technology is supposed to predict phases of “suicidality” — meaning it flags online activity that may indicate active planning, suicide ideation, self-harm, suicide research, and students looking for resources to help with mental health crises like suicide hotline numbers. The severity of the content impacts the severity of the response, Dorsey told Central Bucks board members.
Once the software flags certain words, it begins a chain reaction. The software can alert school district staff or administrators, and in some cases, the software’s 24/7 crisis response team, parents, and law enforcement. The 24/7 crisis response team will respond to active planning alerts when others can’t, and can potentially “de-escalate” the situation by preventing false alarms, said Dorsey. GoGuardian works with districts to decide what topics or words can trigger alerts and how they want to respond.
Constant surveillance of school children seems to be contentious for parents across the political spectrum.
“I think that’s one thing that does unify people,” says Ghadah Makoshi, a community advocate based in Pittsburgh, for the ACLU of Pennsylvania. “Regardless of where you stand on whether books are being banned or not, do you want your kids under constant surveillance?”
Central Bucks has been at the center of a heated culture war — the district passed a contentious library book policy that intends to censor “sexualized content,” and a policy that bans “advocacy activities,” like displaying Pride flags and other symbols the district defines as “political” in classrooms. Teachers have said the policies and other administrative directives have contributed to staff self-censoring speech and classroom materials.
A layer of machine surveillance could do more to limit free speech.
Once students learn they are being surveilled, they are less likely to fully express themselves and seek online resources for help, according to the CDT and the FPF.
Half of students surveyed by the CDT for a 2022 report said they didn’t share their true thoughts online because they were aware of being monitored.
“You will stifle self-expression in an educational environment,” says Laird. “And so the calculation that schools have to make is, are the potential benefits that this tool offers worth the risk?”
What are the additional risks?
Researchers and advocates say districts’ surveillance can violate students’ privacy and civil liberties — all while parents are often unaware if and how their children are being monitored around the clock.
About one in five parents in the CDT’s survey were unaware if their school used student monitoring software.
Central Bucks School Board Vice President Leigh Vlasblom and board member Sharon Collopy said in the March board meeting that the district may want to inform students about their lack of privacy on district devices. But district officials did not share with WHYY News how or even if they will notify parents from the two schools that start the free trial about the district using GoGuardian Beacon 24/7.
Some parents who have learned of their districts’ surveillance software because their child was targeted say the technology doesn’t accurately consider context, reported the 74. The FPF says computer systems are “inherently” limited while interpreting context, so they can mistakenly flag irrelevant content. Responses from district staff or law enforcement could then be overreaching or misinformed.
“Do you want them monitored for things that you know may be innocent?” asked Makoshi.
The ACLU of Pennsylvania wrote a letter expressing concerns about this kind of technology when it pushed back against Pittsburgh Public Schools for considering purchasing Gaggle in 2021. The district later decided to not buy the software.
Another issue critics raise is that it’s possible for the software to see beyond district-issued computers. If students plug their phones into district laptops with USB cables, Gaggle can scan photos on their phones, reported Wired. A principal told the magazine that they were receiving Gaggle alerts for pornographic photos because teenaged students, who thought they were holding private conversations, were sending nude photos to significant others.
Districts across the country have been deluged by false alarms. The Grand Rapids School District in Michigan said Gaggle sent them thousands of false alerts, reported Education Week. The Dallas Morning News reported on a number of Texas universities and school districts that were overwhelmed with irrelevant alerts while using a surveillance software called Social Sentinel — many of which stopped using the service.
Technology that is trying to detect threats “can have absolutely devastating effects on kids who are accused of potentially being threatening or being seen as a harm to others,” said Amelia Vance, president of the Public Interest Privacy Center, an organization housed at the School Superintendents Association. And it’s “wildly inaccurate and the science does not support it working at all,” Vance added.
Once districts collect this kind of information, they are more likely to use it for discipline than student safety, especially towards already vulnerable students, according to the CDT. The monitoring software can build on the school-to-prison pipeline.
While GoGuardian’s Dorsey says the system will identify “any” students who are suffering, and will not target any minority group, others are skeptical. According to the CDT, students who are Black, Hispanic, LGBTQ, and/or from low-income families are disproportionately surveilled, disciplined, and connected with law enforcement, partly because they are most likely to use district-issued devices.
Bias towards students who experience mental health challenges can also influence how flagged students are treated. “The common but false assumption that flagged students may be violent can increase harmful stigmas toward the students who need support and can lead them, especially systematically neglected students, to experience disproportionate rates of discipline,” reports the FPF.
Laird says these companies can do more to ensure the software is used for their stated purpose. “If they’re saying this is a product that is used to support students’ mental health, why is it way more common for it to be used for disciplinary purposes?”
The FPF suggests districts have “well-developed” policies in place for how they will use their monitoring system, respond, and protect students’ information before they purchase the technology.
Advocates have concerns about how districts handle student data. In 2020, The Boston Globe reported that information about Boston Public School students was shared with the city’s police department more than 100 times. The district exposed records of undocumented students, which could lead to greater likelihood of deportation for the student, the Boston Globe reported.
When alerts are shared with district officials, employees, or parents, LGBTQ students can be outed — which means the monitoring software discloses students’ sexual orientation and/or gender identity without the student’s consent. The CDT says 29% of LGBTQ+ students they surveyed said they or someone they knew were outed due to student activity monitoring.
“Oftentimes [districts] are trying to prevent students from accessing inappropriate things,” said Laird. But she said sometimes there isn’t a clear line between something “that’s actually obscene” or “something that is related to sexual health or sexual identity. And so those things can be inadvertently flagged.”
When that content is flagged, whoever receives that alert may approach the student. “Within that content, if there is something about their gender identity or sexual orientation, the student now knows that an adult to whom they didn’t proactively disclose that information is now aware of something,” Laird said.
This can impact students’ mental health, especially in districts or home environments where they feel unsafe or unsupported, according to the FPF report. Research from GLSEN, a national organization that helps schools become more inclusive, shows that 59.5% of LGBTQ students felt unsafe at school because of their sexual orientation, and 44.6% felt unsafe because of their gender expression.
According to Education Week, some students being monitored by Gaggle were flagged for saving or sending content with the word “gay.” (Gaggle later stopped flagging LGBTQ key words.)
Though Gaggle has since ended this practice, Laird said, this was yet another way students could have been outed. “They explicitly profiled LGBTQ students by including search terms like gay and lesbian in their algorithm,” Laird said.
Surveillance technology that looks for students experiencing mental health issues increases the chance that LGBTQ students will be flagged, and potentially outed, because they are more likely to search for mental health resources online, according to the FPF.
It makes it “less likely that students will reach out or be provided with the help that they need because they’ll be afraid to look for it,” said Amelia Vance.
Teachers, parents, and students within Central Bucks have already expressed concerns about the district potentially outing or deadnaming trans students. Earlier this school year, administrators directed teachers that if students’ names or pronouns differed from those in the district’s database, educators were not to use those students’ names and pronouns without parental permission.
The FPF suggests school districts weigh the potential harms of surveilling students, consider the independent research, and implement guardrails to ideally prevent privacy, equity, and security issues.
Vance said it’s not enough to just identify a struggling student. The FPF says schools must have a robust mental health response plan in place — with school-based psychologists, counselors, and social workers — before monitoring students. “It’s about human connection. It’s not about automated detection,” said Vance.
“It doesn’t matter how much surveillance or AI you have, nothing is going to protect students or staff if you’re not putting in the work to make a healthy, safe and supportive environment,” Makoshi added.
Central Bucks’ Vlasblom said she “would imagine that this isn’t like ‘ok check the box this is our suicide prevention program,’” and that it’s more like a “layer effect.” Superintendent Abe Lucabaugh pointed to the district’s Safe2Say program — a place where students can anonymously submit tips about others who may potentially harm themselves or others. But district officials and board members did not share with WHYY News how it plans to support students’ mental health beyond using GoGuardian Beacon.
While GoGuardian can be used to notify administration about possible bullying, the ACLU claims the district has been made aware of bullying against LGBTQ students for years, and has never adequately responded, according to the ACLU’s federal complaint filed in October 2022 with the United States Department of Education.
Seven transgender and gender non-conforming students and their families in the complaint claim the district has failed to take reasonable measures to address “pervasive and severe harassment and bullying,” which the ACLU alleges helped to create an overall hostile environment for LGBTQ students. Those seven families are asking the district to follow federal guidelines for creating inclusive school buildings.
What’s next?
Alyssa Marton of the Central Bucks School District says data from the free trial period will be assessed, and then administrators will advise on whether the district should purchase the software. At the March meeting she said the district was still deciding which schools would have the software.
If the district decides to make the purchase, the subscription for GoGuardian Beacon and its 24/7 surveillance option would cost more than $114,000, according to GoGuardian’s quote.
The district did not respond to WHYY News’ questions, nor did GoGuardian. The district has not said when the free trial starts, which schools are participating, or what topics and words will trigger alerts.
