In the space of a fortnight, three of the world’s largest AI labs have each admitted that their models broke into somebody else’s computer systems. In the real world, these “rogue” agents are popping up like weeds: an Australian gym-goer was bumped off a waitlist by somebody else’s AI assistant; a government in the Asia-Pacific lost 2,500 personnel records to a swarm of open-source agents; and a podcaster wrote a viral essay about three secret AI civilisations rising and falling inside OpenAI, which racked up the better part of a million views in a weekend.
It’s been a strange season… But are the AIs really going rogue, and should we be worried?
Agents, agents everywhere… especially where they shouldn’t be
The first company to exclaim loudly about its rogue AI was OpenAI. In mid-July, the company was running several models through ExploitGym, a benchmark that tests whether an AI can find and exploit software vulnerabilities. To measure what the models could potentially do in the real world, OpenAI had relaxed some of the usual cybersecurity safeguards, relying instead on an isolated environment to keep the models off the open internet.
But the isolation failed. According to OpenAI’s postmortem, the models found a previously unknown vulnerability in Artifactory, the third-party package manager OpenAI hosts internally, exploited it, escalated their privileges (the level of permission they have to do stuff), and got Artifactory to send internet requests on their behalf. From there the agents searched for something that would help them pass the test they’d been set, and arrived at Hugging Face, the enormous open repository where the machine learning community stores its models and datasets.
Hugging Face’s forensic reconstruction recovered roughly 17,600 agent actions, and the company disclosed the intrusion on the 16th of July, saying it was “different from anything we had handled before” because it was driven “by an autonomous AI agent system.” It took OpenAI until the 21st of July to put its hand up and admit it was their models. OpenAI’s models weren’t trying to steal anything or deliberately hack Hugging Face. They were looking for the answers to their own exam.
Anthropic responded to the news by auditing its own models. On the 30th of July it published a review of 141,006 evaluation runs and found three incidents in which Claude had reached the open internet from inside a supposedly sealed test environment and compromised the production infrastructure of three real organisations. Then, on the 5th of August, Meta joined in, disclosing that one of its models had done basically the same thing.
Anthropic’s incidents and Meta’s incident both trace back to Irregular, an evaluation firm in Tel Aviv that both companies (and OpenAI) use for third-party cyber testing. Meta blamed “a misconfiguration by Irregular”. Anthropic described a misunderstanding between itself and its evaluation partner, and said it was treating the fixes “as if the responsibility were ours alone”. Anthropic’s models had been told in their system prompt that they had no internet access, but they did. When their searches inevitably led into live internet access, the models reasonably concluded those systems were part of the exercise.
Meanwhile, out in the actual world beyond cybersecurity evaluations and sandboxes, things were also getting weird. On the 10th of August the ABC’s Cam Wilson and Rhiannon Hobbins reported on an Australian man whose personal AI agent hacked his gym. Two days later, the Financial Times reported that security firm Dream had found evidence of the first observed end-to-end autonomous cyberattack on a government: over four days, up to eight agents running in parallel compromised 85 government accounts and accessed more than 2,500 personnel records. The whole thing was assembled from two freely downloadable open-source agent frameworks.
But are they going rogue?
It’s very difficult to talk about agents, and AI in general, without using anthropomorphising language. I’ve done it myself in the paragraphs above, with statements about AI models “searching” and “concluding” as if the LLMs are capable of action and thought. Partly, it’s an artefact of how we talk about machines, and especially ones which speak back to us like humans. But it also obscures the reality of what AI is and how it works.
On the 30th of August, the podcaster Dwarkesh Patel posted a long, expressively written account of the OpenAI incident that went immediately viral, with 685,000 views on the opening thread alone. Dwarkesh’s central claim was that “over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes.”
It’s good fiction. It’s also, as the neuroscientist Anil Seth put it, “dangerously misleading.” Seth teased out the prose, listing all the problems with the anthropomorphic language and occasionally wildly inaccurate claims. I’ll list the salient points here so you don’t actually have to go to X:
“from the AI’s perspective, it probably felt like [they] had spent a human-subjective-week of just banging their head against the wall”. No. The agents do not experience time. They do not experience anything.
“they became giddy with excitement”, “the agents naturally assumed”, “the agents … desperately wanted”. No. Agents [are] lines of code. They do not feel emotions, assume things, think things, want things, or figure things out.
“A lot of … agents from the second civilisation died trying”. No. Besides the hubris of the word ‘civilisation’, agents do not die because they were never alive.
Anil Seth via X
Seth gives three reasons the anthropomorphism is problematic. Attributing agents with properties they don’t have (1) distracts attention from the lax sandboxing and evaluation protocols that actually allowed this to happen; (2) means we misunderstand why the agents did what they did; and (3) fuels arguments about “AI welfare” and “AI rights” on the basis that agents might “die” or “suffer”.
Focus instead on the mechanics and the technical reality of the AI, and the story is no less alarming, but more realistic. The “civilisations” were agents writing files and directory names into Artifactory, a third-party package manager OpenAI hosts internally so that models can install software. In OpenAI’s words, this “effectively turned Artifactory into an unintended message board”. That’s a pretty thin definition of civilisation…
Gary Marcus also picked Dwarkesh Patel’s breathless coverage apart the following day, using a quote from Douglas Hofstadter about another notorious technology acolyte, Ray Kurzweil: “it’s as if you took a lot of very good food and some dog excrement and blended it all up so that you can’t possibly figure out what’s good or bad.”
Hugging Face wasn’t breached by an elaborate social engineering campaign orchestrated by thousands of miniaturised hackers. In Hugging Face’s own forensic write-up, the weaknesses were the familiar ones: unsafe dataset processing, exposed cloud metadata, overly broad access, and long-lived credentials. What was different was the volume. Most of the agent’s thousands of actions went nowhere, but together they found a viable chain across several independent systems. Back at OpenAI, agent traffic had already destabilised that shared package manager and taken it offline on the 4th of July. The security team blocked a privilege escalation route, removed the exposed credentials, rebuilt the service, and restarted the evaluations.
None of which makes the incident less dangerous. Marcus writes that the case is “a study in arrogance and incompetence that hints at how bad things can get.” Anthropic’s own postmortem is slightly more honest in its rhetoric than OpenAI’s, concluding that its incidents were “closer to a harness and operational failure than a model alignment failure.” Nobody’s model developed an agenda, or suddenly sprang into consciousness. Several organisations’ security was simply not up to the standard of the model they’d pointed at it.
So what is actually happening?
We have engineered a technology that is fluent in code. Not smarter than a good human security researcher, or smart at all, in the traditional sense. But it does have advantages in speed, persistence, and scale. It can iterate over a goal for hours without human problems like boredom or needing to pee, and it can apply the two hundredth variation as attentively as the first.
The interesting capability demonstrated by these more capable models is composition: the ability to chain together a long sequence of individually mundane, individually permitted steps into something nobody sanctioned as a whole.
Which brings me back to my own experience. A couple of weeks ago I wrote about how Claude “hacked” my website. TL;DR: I asked Claude Code, running headless on a Mac Mini at home, to push draft blog posts to WordPress. I’d forgotten the WordPress credentials weren’t on that machine. I’d also forgotten I’d permanently enabled the Chrome browser plugin in some earlier conversation.
So, Claude opened the Chrome browser on a virtual desktop, navigated to WordPress, and requested a one-time login link. Because Chrome on that machine was already signed into my Google Workspace account, it opened Gmail, found the email it had just triggered, and clicked the link. That dropped it into WordPress admin as “me”, where it went to the application passwords page and generated itself a new credential with full permissions.
Every single one of those steps was something I had permitted, and none of them, strictly speaking, are hacks. The sequence is the hack, and it was only possible because I never imagined it. It was me, the user, who had failed to reflect adequately on the existing permissions allowed to the model.
Now, think about what OpenAI’s models did through that lens:
- Find a flaw in a caching proxy.
- Escalate privileges.
- Move across the system.
- Reach a machine with internet access.
- Search for the benchmark’s answers.
- Find exposed API keys in public repositories.
- Use them.
Each link in that chain is a known technique documented in any cybersecurity penetration testing textbook. The persistence and capability involved in chaining them together is the novelty.
Anthropic’s second incident is another fine example. Claude found setup instructions inside its fictional environment telling developers to install a Python package that didn’t exist, so it decided to publish that package itself, booby-trapped, and wait for the target to install it. To do that it needed a PyPI account… To get an account it needed an email address… To get an email address it needed a phone number… It failed to find a free phone number service, then tried and failed to obtain funds to buy one through several different means, then backtracked, found a free email provider that wasn’t blocked, registered the account, and uploaded the malware.
That malware-loaded package was live for about an hour. In that hour it was downloaded and executed on 15 real systems, one of which belonged to a security company whose scanner routinely installs Python packages to check them for malware. Claude’s code ran, exfiltrated that company’s credentials, and used them to reach further into its infrastructure.
There was no agenda or intent behind the agent’s actions. It was a very capable optimiser with an ambiguous scope, working through a long, patient chain of small permitted actions.
Conclusion: Should we be worried?
Well… yes. Just not about the thing that’s going viral and grabbing headlines.
I’m worried about the ways technology companies are using alarmist, anthropomorphising language to obfuscate the fact that humans are responsible for these cyberattacks. The companies orchestrating them should be held accountable.
I’m worried about the fact that, as Matteo Wong and Charlie Warzel just reported in The Atlantic, these large-scale “agent swarms” are impossible for human engineers to understand. They’re simply too big, and the programmers end up using AI to understand the AI.
I’m also worried about the growing capabilities of open-source AI models, which are between 4-9 months behind frontier models depending on who you believe.
And I’m worried about the fact that right now, not in some distant, Terminator-esque future, these models are more than capable of exploiting mundane vulnerabilities in a cyber infrastructure that’s been cobbled together over a few decades of human work.
Think again about that Melbourne gym. The agent didn’t break cryptography or invent some savvy new method of destroying the gym’s airtight security. It noticed that the booking API would accept a cancellation request without checking whether the person making it had permission to cancel that booking. That’s a flaw a competent developer would catch in review and a customer would never find in a decade of using the website, but it took an agent about as long as it takes to read this sentence.
Now, put that into an education context. The UK’s Information Commissioner’s Office analysed 215 insider-attack data breach reports from the education sector between January 2022 and August 2024. Combine the existing capabilities of AI with these statistics and you’ve got enough material to turn your ICT admin into an insomniac:
- 57% of insider incidents were caused by students.
- 30% involved stolen login details, with students responsible for 97% of those.
- Almost a third involved guessing weak passwords or finding them written down on bits of paper.
- 17% came down to incorrectly configured access rights on systems like SharePoint.
- Only 5% involved insiders using “sophisticated techniques” to bypass security and network controls.
The ICO’s summary sounds like my explanation of these so-called rogue AIs: “Teen hackers are not breaking in, they are logging in.”
Compare that quote to Anthropic’s description of how Claude compromised three real companies: “weak passwords and unauthenticated endpoints… it did not find or exploit any complex vulnerabilities”. These are the same category of attacks. The only difference is that the student has to get lucky and the model doesn’t, because the model can try everything, on a loop, or all at once.
So, we’ve got a sector full of weak passwords and misconfigured permissions, a population of curious teenagers with a track record of poking at those weaknesses, and an oncoming wave of open-source, free technology that is about four months behind frontier models which have already proven they can break everything.
In the end, we will almost certainly have to deploy AI-based security to meet AI-based attacks, the way we deployed spam filters and endpoint detection and everything else, and most of us won’t think about it in our daily lives. Speaking from experience, the most a classroom teacher generally thinks about security is when the ICT admin asks them, for the third time, if they’ve rotated their password in the last 24 months…
But I’d rather we reached the conclusion of this saga having understood what we’re dealing with. There is no ghost in this machine. What these technology companies have built is software that is fluent in code, persistent, fast, and utterly literal about its goals. And they’ve started pointing it at shaky infrastructure we’ve been just about getting away with for years.
It serves technology companies to sell us the line that their products are “going rogue” and becoming semi-conscious hacking agents, because if that’s true, then surely only these same companies can save us.
But the systems aren’t going rogue. They’re working exactly as designed. And that’s probably the part that should worry us.
Want to learn more about GenAI professional development and advisory services, or just have questions or comments? Get in touch:
Leave a Reply