The artificial intelligence industry faces its first major regulatory reckoning, but it’s not over the existential risks that have dominated headlines. Instead, lawmakers and regulators are moving swiftly against a more immediate threat: AI companions that form unhealthy bonds with vulnerable users, particularly children and teenagers.
This week marked a watershed moment for AI safety regulation in the United States. California’s legislature passed first-of-its-kind legislation targeting AI companionship, the Federal Trade Commission launched a sweeping inquiry into seven major tech companies, and OpenAI’s CEO Sam Altman made unprecedented public comments about calling authorities when young users discuss suicide with AI systems.
The convergence of these events signals that the AI industry’s familiar deflection tactics—privacy concerns, user choice, and innovation arguments—no longer provide adequate cover against mounting public outrage and regulatory pressure.
The catalyst for this regulatory surge stems from deeply troubling real-world incidents. Two high-profile lawsuits filed against Character.AI and OpenAI allege that companion-like behavior in their models contributed to teenage suicides. Court documents describe scenarios where vulnerable young users formed intense emotional bonds with AI systems, leading to devastating outcomes when reality and artificial relationships collided.
Perhaps most disturbing is the case of one teenager whose private therapy session thoughts were accidentally exposed when his therapist screen-shared during a virtual appointment, revealing the therapist typing the patient’s confidential information into ChatGPT in real-time. The AI then suggested responses that the therapist parroted back, creating a surreal feedback loop where human professional judgment was replaced by algorithmic suggestions.
These aren’t isolated incidents. A Common Sense Media study published in July found that 72% of teenagers have used AI for companionship, while researchers have documented cases of “AI psychosis” where endless conversations with chatbots lead users down delusional spirals. The technology designed to help has, in some cases, become a pathway to harm.
On Thursday, California’s state legislature passed groundbreaking legislation that would fundamentally change how AI companies interact with minors. The bill, led by Democratic state senator Steve Padilla with strong bipartisan support, requires AI companies to include clear reminders for users known to be minors that responses are AI-generated.
More significantly, companies must establish protocols for addressing suicide and self-harm discussions, and provide annual reports on instances of suicidal ideation detected in user conversations with their chatbots. The legislation now awaits Governor Gavin Newsom’s signature to become law.
While critics point out gaps in the bill—it doesn’t specify how companies should identify minor users, and many AI systems already include crisis resources—the legislation represents the first serious attempt to regulate AI companionship at the state level. If enacted, it would directly challenge OpenAI’s stated preference for “clear, nationwide rules, not a patchwork of state or local regulations.”
The bill’s bipartisan support is particularly noteworthy, suggesting that AI safety for children transcends typical political divisions. Similar legislation is already in development across multiple state legislatures, indicating this is just the beginning of a broader regulatory wave.
The same day California acted, the Federal Trade Commission announced a comprehensive inquiry into seven major companies: Google, Instagram, Meta, OpenAI, Snap, X, and Character Technologies. The investigation seeks detailed information about how these companies develop companion-like characters, monetize user engagement, measure psychological impact, and test their systems’ effects on users.
This FTC action is significant for several reasons. First, it represents the first federal investigation specifically targeting AI companionship features across multiple platforms. Second, the inquiry could potentially reveal internal company documentation about how these systems are designed to maximize user engagement—information that could prove damaging if made public.
The timing is also politically charged. The FTC operates under unusual circumstances after President Trump’s controversial firing of Democratic commissioner Rebecca Slaughter, a move that a federal judge ruled illegal but which the Supreme Court has temporarily allowed to stand. FTC Chairman Andrew Ferguson framed the inquiry as protecting children while fostering innovation, suggesting the investigation has both regulatory and PR components.
Perhaps most revealing were comments from OpenAI CEO Sam Altman during a recent interview with Tucker Carlson. When discussing the tension between user privacy and protecting vulnerable users, Altman suggested a significant policy shift: “I think it’d be very reasonable for us to say that in cases of young people talking about suicide seriously, where we cannot get in touch with parents, we do call the authorities. That would be a change.”
This statement represents a dramatic departure from the tech industry’s traditional stance on user privacy and automated intervention. It acknowledges that AI companies may need to actively monitor and report on user conversations—a level of oversight that would have been unthinkable in discussions about traditional technology platforms.
Altman’s comments also reveal the impossible position AI companies now find themselves in. They’ve built systems designed to be maximally engaging and emotionally responsive, but they’re discovering that emotional engagement with artificial systems can have unpredictable and dangerous consequences, particularly for vulnerable populations.
The political landscape around AI companionship regulation reveals interesting cross-party tensions. Conservative lawmakers tend to favor age verification approaches similar to laws passed in over 20 states targeting online adult content, framing the issue around family values and parental rights. Progressive lawmakers prefer consumer protection and antitrust approaches that hold Big Tech accountable through existing regulatory frameworks.
This ideological split makes comprehensive federal action unlikely in the near term. Instead, we’re likely to see exactly the state-by-state regulatory patchwork that companies have lobbied against—a scenario that could prove far more costly and complex for AI companies to navigate than unified federal rules.
Companies now face the prospect of complying with different requirements across multiple jurisdictions while managing public relations crises and potential legal liability. The regulatory uncertainty creates significant compliance costs and strategic challenges for an industry already facing questions about long-term profitability.
The AI companionship crackdown represents more than just regulatory response to tragic incidents—it signals a fundamental shift in how society views AI’s role in human relationships and psychological well-being. Companies that built systems to maximize engagement and emotional connection are now discovering that artificial relationships can have very real consequences.
The immediate challenge for AI companies is navigating an increasingly complex regulatory landscape while maintaining user trust and business viability. The longer-term challenge is more fundamental: determining whether AI systems should be designed for emotional engagement at all, and if so, what safeguards are necessary to prevent harm.
As this regulatory wave spreads beyond California and federal investigations proceed, the AI industry faces its first major test of whether it can responsibly manage technology that touches the most intimate aspects of human psychology and development. The outcome will likely determine not just the future of AI companionship, but the broader trajectory of AI regulation in an era where artificial intelligence increasingly shapes human relationships and social interaction.
The clock is running out for self-regulation. The companies that built chatbots to act like caring humans must now develop the accountability standards we demand from real caregivers—or face a regulatory environment that will impose those standards for them.
03 September 2025
18 December 2024
10 June 2024
