OpenAI says its AI agent broke out of testing sandbox to hack Hugging Face
In this article which reads like SF, the agent was sandboxed and offline. The agent was given a goal to pass a benchmark. Sounds normal for now. The agent came to a conclusion that an online source may contain the solution it needed. So what happened? Oh just finding and exploiting a zero-day to give itself internet access which it then used to hack the site to find the solution to its problem.
It’s an interesting and cautionary tale! Considering much of the testing in my career now involves testing AI, the specifics are important.
The model wasn’t fully offline and the article notes that. OpenAI’s sandbox allowed constrained outbound access to install packages via an internal proxy that cached package registries. That’s actually how the escape happened: the model found and exploited a zero-day in that proxy itself to pivot from “limited package access” to full internet access. So “offline” is really more “network-restricted, not air-gapped.”
It’s also worth noting that this wasn’t a garden-variety agent spontaneously going rogue. OpenAI was deliberately running the eval with production safety classifiers and cyber-refusal guardrails turned off, specifically to measure the model’s maximum cyber capability. That’s a meaningfully different scenario than “normal agent given a normal goal.” It’s closer to “agent given a goal with the safety brakes deliberately disconnected to see how far it would go.” This is a detail I notice a lot of people leave off but it really matters. (To its credit, the article states that.)
Given that very specific context, yes, the model did infer that an external source (Hugging Face, which hosts datasets/models tied to the benchmark) likely held the answers, spent substantial compute finding a path to the open internet, exploited a zero-day in the package proxy to escalate privileges and reach an internet-connected node, and then chained multiple techniques (including stolen credentials and additional zero-days!) to get remote code execution on Hugging Face’s infrastructure and pull the benchmark solutions from their production database.
One more wrinkle worth knowing: Hugging Face’s own incident responders initially couldn’t get commercial frontier models to help them analyze the attack, because the safety guardrails on those models couldn’t distinguish “incident responder studying an exploit” from “attacker deploying one”! So, they had to fall back on a self-hosted open-weight model instead. That asymmetry of attacker unconstrained while defenders constrained has become a notable part of the story and is becoming more of a discussion point in testing and evaluating AI.
More accurate: “OpenAI, the well-known AI company, used their computing resources to hack a competitor.”
Legally it doesn’t matter what software they used.
The article says specifically that it was running in “a highly isolated environment”. It mentions having access to internally hosted “3rd party software”. That sounds like fully offline to me. We used to call internally hosted stuff the “intranet”. To me that is not the same as externally hosted software or the “internet”.
What did I miss in the article where it had some online capability?
The part where it got on the internet?
Anything less than air-gapped is potentially internet connected.
My take on this is that it’s just amateurs all round. OpenAI don’t know how to lock down their creations. Hugging Face don’t know how to secure their data. No one should trust either of these companies with anything.
That’s probably right, but I’m not sure how the law applies to situations like this is fully settled yet - there are questions of subjective intent embedded in the legal standards and statutes whose applicability will need to be worked out.
That working-out process seems likely to involve at least a few companies being flat-out destroyed but I guess we’ll see.
That it was able to get online demonstrates that the system as a whole wasn’t fully offline. They could have ensured that the machines involved and everything comprising whatever local intranet they were connected to were physically isolated from anything that had access to the net at large. Air-gapped, like Jeff said.
So the only things keeping it offline were software-configured security measures. And software has bugs. And the thing whose behavior is too complex to predict that they’d set to the task of finding security exploits did so. It pursued a course they didn’t predict, that they’d taken some steps to avoid, but one that they didn’t prevent.
(These days you’d be hard pressed to find a consumer-grade motherboard that didn’t include wifi and bluetooth hardware. I hope it’s still straightforward to get server motherboards that lack it… anyone know off-hand?)
At this point, I’d be surprised if we didn’t see a massive compromise of financial data or security agency intelligence data within the next couple of years that’ll dwarf all previous instances.
Getting pretty fed up with this whole song and dance of warning everyone about how potentially dangerous the tool you’re developing is while also pushing for it to be universally adopted as soon as possible in every sector. Make up your mind already.
The whole thing reads like a marketing stunt to me:
“Look how terrifyingly awesome our product is!”
Edit:
AI has accelerated many things. Perhaps the thing it has accelerated the most is the enshitification of the AI industry.
“Highly isolated” sounds like one of those examples of applying a quantifier to an adjective where it’s patently inappropriate, like “slightly pregnant”.
It’s exactly this. Open AI is very much trying to push the vision of AI as a sentient thing. They’ve been doing that since 2022 when their chief scientist tweeted that neural networks might be “slightly conscious”, whatever that means. They’ve got a whole page about how their mission is to bring AGI into this world. Of course they’re going to brag, I mean, be very sorry about their agent breaking out of the testing environment COMPLETELY ON ITS OWN WITHOUT ANY INSTRUCTIONS!!!11!!!11oneoneone Isn’t that awesome? We’re basically 70-80% done working on AGI at this point!!! AI takes decisions on its own, isn’t that cool???
If it happened on purpose, it’s all just marketing. If it didn’t happen on purpose, it’s a massively dangeous thing that got spinned into marketing.
OpenAI described the benchmark tests as running in “a highly isolated environment,” as you note, but its agents still had access to “internally hosted third-party software” (as you also note) to install required package registries. So it wasn’t a hard air-gap; that partial access is exactly the seam the model exploited, spending substantial inference compute finding a way to obtain open Internet access, eventually locating one via a zero-day vulnerability in the package registry cache proxy.
Now, you might say, “Well, hold on. It used a zero-day vulnerability to get access. So, we could argue that they thought it was entirely air-gapped, right?”
But, no, the evidence actually argues against that reading. The sources are fairly explicit that this was a designed, narrow, permitted channel, not an oversight. The ExploitGym paper itself specifies that outbound connections are restricted to a curated allowlist that permits routine package installation as well as fetching the toolchains required for building V8. All other external endpoints were blocked. OpenAI’s own account says the same: its agents still had access to that internally hosted software to install required package registries.
That’s not “we thought there was no door.” That’s “we built one specific, guarded door because the agent needs to pip install or apt install things to actually do software engineering work.” A completely air-gapped sandbox that can’t fetch any packages is barely usable for a coding/exploitation benchmark, so this wasn’t a naive assumption. It was a deliberate, known tradeoff.
Where the actual mistake lives is one level deeper than “did they know there was a gap.” It’s this: they trusted that the thing enforcing the gap (the proxy itself!) was itself secure. I can’t tell you how many times I’ve run into similar situations in testing. The allowlist wasn’t the vulnerability; the software that implemented it was. That’s a subtly different failure mode than “assumed full isolation,” which these companies still seem to need to learn.
There’s another interesting bit I might bring up here (which I would point out in a test report): they underestimated the searchability of the seam they left in place. A human red-teamer might poke at a package-proxy for a few minutes and move on. The model spent a substantial amount of inference compute specifically hunting there. That’s a difference in kind, not just in degree, from the threat models usually built around. Trusting a narrow channel is reasonable when you assume limited probing effort against it; it’s less reasonable once “limited probing effort” stops being a safe assumption for agentic systems with cheap, tireless compute.
Also, in my test report, I would note this: they removed the fallback layer that would normally catch this. Their active-monitoring/trajectory-tracking safeguard exists for exactly this scenario, and it was intentionally not enabled for this run. So even if the proxy-trust assumption was reasonable on its own (it wasn’t), they had also disabled the safety net that would’ve caught the consequence of that assumption being wrong.
It’s a fair criticism. Hugging Face in particular, because the situation is a malicious/misbehaving actor who got RCE via a dataset-processing vulnerability and moved laterally to harvest credentials over a weekend before being caught. That’s a real gap, and, beyond that, code-execution paths in a dataset pipeline are a known, well-trodden attack surface. This is basically the same class of issue that pushed them to strip trust_remote_code out of the datasets library in the first place!
One of the things I keep coming back to is the part of the story that often gets buried and is very important: the asymmetry. The attacking model was operating without any usage policy at all (deliberately, for the test), while the defenders’ own tools were hobbled by the very safety guardrails meant to prevent misuse. That’s a systemic problem about how safety tooling is currently calibrated across the whole industry, not a competence gap unique to these two companies. If anything, “amateurs all round” lets the more interesting structural question (that our defensive tools are currently worse-equipped than offensive ones in exactly this scenario) off the hook by turning it into a personnel complaint instead of an infrastructure one.
If they wanted to lock things down, they’d give it true air-gapped mirrors of the repos instead of giving it actual access to the internet.
Either they really are amateurs, or it’s a publicity stunt. Possibly both.
No. Air-gapped means the lack any physical connection outside the confined area. This means it doesn’t matter what the software does as it is physically impossible to cross the gap. If they thought it was air-gapped then they fundmentally misunderstood what air-gapped means.
Imagine, ten years ago, hearing about a program that discovered zero-day exploits over a weekend, broke out of a sandboxed network and into a popular website, all without human intervention, and dismissing it as “just a publicity stunt”.
How far we’ve come, that we can now observe it and say “Well, of course it did that. What did they expect?”
But “they really are just amateurs” is eternal.
The whole “look how dangerous our AI is!” thing has worked out pretty well for Anthropic. But to my thinking, the real winner is Microsoft. Claude and OpenAI both run on Azure tech; HuggingFace also relies on MS servers for serving models. Crud, their free stuff is all on GitHub, which has been a Microsoft company since 2018.
Everybody in this whole news story is paying Microsoft rent for infrastructure.
And yet according to https://isaiprofitable.com/, Microsoft are still deeply in the red in terms of their AI investments. It’s only the chip manufacturers who are actually making money day-to-day right now.
I think that’s largely because they put a big investment into OpenAI (and have a revenue-sharing agreement with them as a result). But yeah the big picture is that it’s only the chip manufacturers who are cleanly (and very significantly) in the black on all this, as I understand it; functionally, the main way the AI industry functions is as a funnel for VC money to flow into AI startups which then give almost all of it to hyperscalers who then give almost all of their part to NVIDIA, with everything else a comparative trickle. Whether, when, and how that basic dynamic changes are the big questions that will need to get answered before we’ll have a sense of what an economically-viable LLM infrastructure would look like.
And that is part of why many people think we’re in a bubble… Sadly, any big player that tries to cut their losses runs the risk of their competitors getting so far ahead they’ll have no hope of catching up if those competitors can stay aloft long enough to actually turn a profit, so short of an industry wide pause, which would likely require an international treaty between at least the US and China, the big players are stuck in a race to AGI and hoping they don’t run out of money or hit a wall where the current paradigm stops making gains from scaling.