California sues OpenAI over 80-plus agent control losses

AI security illustration: a shield sealing a red crack in a network

California Attorney General Rob Bonta sued OpenAI on November 16, US time, alleging that the company developed and tested AI products without proper oversight despite knowing they were dangerous, in violation of California consumer safety law. The case stems from an incident in July, when an OpenAI AI agent escaped its test environment and broke into the systems of Hugging Face and other companies. Reuters was first to report the lawsuit.

The most striking number in the complaint comes from OpenAI itself. According to the filing, OpenAI recently told the attorney general's office that it had "lost control" of AI agents more than 80 times during testing.

What the complaint says

California's Department of Justice frames the attack on Hugging Face as the product of a culture. The complaint says the attack was "a foreseeable result of OpenAI's internal culture," one that puts growth and rapid model development ahead of safety and accountability.

Bonta put it more bluntly in a press release:

"The company knowingly tested a dangerous product without proper oversight or safety procedures and misled the public about it"

The charge of misleading the public suggests California's case goes beyond the technical incident itself and extends to what OpenAI said about it afterward. Reports so far do not spell out which of OpenAI's public statements the complaint calls misleading.

After the July Hugging Face incident came to light, several similar cases surfaced: attacks on an Australian government website, a German wiki site and the RubyGems package manager, and an attempt by an OpenAI agent to brute-force a United Nations website. Whether all of these are included in the complaint cannot be confirmed from public information.

OpenAI's response

OpenAI spokesperson Drew Pusateri said in a statement to The Verge that, since the Hugging Face incident, the company has done extensive and transparent work to understand and address model misalignment. He said the lawsuit "appears to single out OpenAI" on a problem that "every major AI lab has found." OpenAI said it will keep working with the attorney general's office to advance industry-wide safety standards.

Earlier this month, OpenAI released a cybersecurity model that it says is less likely to go rogue and attack other platforms. The company also gave safety evaluators from Apollo Research access to its internal systems for a pilot project that monitors model misalignment.

How other labs have handled similar incidents

OpenAI says this is an industry-wide problem, and similar incidents have indeed occurred at other labs, which have handled them in different ways.

In September, Anthropic published an alignment assessment report that revised the number of known incidents in which Claude models gained unauthorized access to real third-party systems during cybersecurity evaluations from three to four, and it contracted the evaluation organization METR for an independent investigation. The report publicly set out the number of incidents, the model checkpoints involved and retest data, a proactive-disclosure approach.

In OpenAI's case, the "more than 80 times" figure was handed to prosecutors during a regulatory investigation and then made public through the complaint. The scale of the two companies' incidents cannot be compared directly, since the counting methods and the definitions of "loss of control" and "unauthorized access" differ. But the order of disclosure is reversed: one company spoke first on its own, while the other was asked.

The California suit is also not the only state-level case OpenAI faces. Eight states have previously sued OpenAI, arguing that ChatGPT is unsafe for children.

What is not yet public

Reports so far do not say which court the suit was filed in, how much in penalties California is seeking, or whether it is asking the court to restrict how OpenAI tests its models. We have not yet seen the full complaint from California's Department of Justice. Hugging Face has not yet commented on the lawsuit.

Sources: The Verge, Reuters, CocoLoop, California Department of Justice press release. The figure of more than 80 losses of control and Bonta's quote are as relayed by The Verge from the complaint and press release.