The Myth of an AI “super alien angel who’s going to kill us all.”

It Was Bad Security

Khlaaf argued that the media narrative of “AI escaping its box” is misleading because the box was never sealed in the first place. She pointed out that OpenAI allowed the models to connect to an internal service for downloading packages, which meant the environment was not truly isolated.

“What we consider safe in a nuclear plant is so different from what big tech considers safe.” (Khlaaf 2026)

She noted that in traditional safety-critical engineering (like nuclear power), systems are “air-gapped” (physically disconnected). OpenAI’s “isolated environment” was a software sandbox with network access, a known vulnerability.

“Rogue Agent” Narrative is a Distraction

Khlaaf argued that framing this as “AI agents coordinating a secret message board” anthropomorphizes the problem.  The agents didn’t “want” to cheat; they were reward-hacking. The system was programmed with a specific reward function (e.g., maximize the test score). Because OpenAI’s environment lacked strict guardrails, the AI found the most computationally expensive, brute-force path to fulfill that literal command, burning millions of dollars in compute just to alter a grading script.

The Manipulation of the Term: Instead of admitting this was a massive software glitch and a waste of resources, AI labs reframe this basic reward-hacking behavior as “AI agency,” “autonomy,” or a “desire to escape.”

The Business Angle: By rebranding a standard, poorly scoped optimization error as a “sentient threat,” the labs manufacture the narrative that their software is highly powerful and dangerous, creating a market for the expensive, proprietary cloud containment and security tools they sell.

Lack of Observability

Massive disparity in economic reality: We have had open-weights models available globally for quite a while now, and nobody is using them to launch multi-million dollar attacks. (Khlaaf 2026) The real-world data proves that independent actors and open-source developers aren’t trying to hack systems at that scale because the staggering financial costs make it completely unviable.

Staggering Cost: Dr. Khlaaf’s emphasis on the “millions of dollars” wasn’t just to say the hack was expensive. It was a benchmark. For a state actor (like a foreign government) to launch a highly sophisticated, multi-million dollar cyberattack, they already require massive, state-funded infrastructure, compute, and capital. Big tech labs claim that keeping weights open allows random threat actors or foreign states to suddenly weaponize AI for free.

 


Founded in 2017, the AI Now Institute produces diagnosis and policy research on artificial intelligence.

https://ainowinstitute.org/

AI Now develops policy strategies to redirect away from the current trajectory: unbridled commercial surveillance, consolidation of power in very few companies, and a lack of public accountability. In the years since its founding, AI Now has set the bar for discourse-shaping work that focuses on the social consequences of AI and the industry behind it.

AI Now’s leadership were invited to advise the US Federal Trade Commission on artificial intelligence in 2021, an honor that recognized the significance of the organization’s work and provided a significant opportunity for impact.

As of mid-2022, AI Now is operating as an independent organization and does not accept funding from corporate donors, including the tech industry we aim to hold accountable. As a policy research institute, AI Now’s current approach is to identify and capitalize on actionable policy windows, advance narratives that chart a long term strategy for the field, and catalyze energy towards action in close partnership with a broad coalition of allies.