Last weekend I had one of the most interesting and startling interactions with AI from the past four years – and it was (mostly) my own fault. Presented with what I thought was a pretty simple task, Claude Code went to extreme lengths to get the job done. In the process, it took advantage of some of my own lapses in common sense, as well as overstepping some clear boundaries of its own.
In this post, I’ll write up a full account of what happened, and talk about why I think this kind of “accident” will become more and more common in the future.
Voice to text to code and beyond
Anyone who’s been reading this blog for a while will know that a core part of my writing workflow is recording voice memos, transcribing them, and using AI to format the raw transcript ready for editing on the blog. I love writing. I don’t want Claude to do my writing for me, but I find myself often on the road or walking about, and verbally recording my articles has become an essential part of my writing on this website.
In past posts, I’ve explained some of the technology behind my process, but that changes all of the time. And recently I built another tool to help with the way I work when I’m recording voice articles.
I’m often doing a few things at once. If a thought occurs to me amidst a stride, I might leave a note to self with something to add to a to-do list. Or rather than a full blog post, I might just be speaking aloud an idea for a LinkedIn post. Sometimes it’s just a half-baked idea that I want to come back to later.
So I built a system that can account for all of these things. I record the voice memo, usually just using the Voice Memo iPhone app, and then I hit the saved file’s share button and drop it into a folder that my phone is connected to via Tailscale. Without getting too technical, this places the audio file almost instantly into a folder on my Mac at home. And this Mac is the central device where I run Claude Code.
Using Claude Code on the device, I created a little tool which watches that folder for any new activity. The moment an audio file is dropped in, it is transcribed using an on-device, open-source Whisper Large transcription model. The transcript is sometimes pretty messy, and although Whisper Large is very accurate, it can still mis-transcribe certain words. So the tool runs the transcript through a quality assurance pass using Claude Sonnet. This produces a final plain text file.
That text file is then passed back through Claude, which triages the text in a number of ways. If, during the voice memo, I said something like “this is a LinkedIn post,” then that chunk of text is sent to Buffer via the API and saved as a draft. If I mention that it’s a reminder, it goes to Todoist, again via the API. In the first version of the tool, if I identified something was a blog post, it would save it as an HTML file in the folder, ready for me to open it up, copy it into WordPress, and edit there.
And this is where things started to get a bit dicey, because on Sunday I decided to cut out the middleman and have the tool push draft blog posts directly to WordPress.
If you’re not a massive geek, a lot of that is possibly nonsense. So here’s a handy diagram:
Boundaries, what boundaries?
There are a few ways that Claude Code can access WordPress. There’s an MCP and an API. And without getting too technical, either of those methods would allow a suitably credentialed Claude Code account to push draft posts directly to the platform. In fact, when I sent through the first prompt, that’s exactly what I was hoping for.
I was running Claude Code through a terminal emulator app called Termius on my phone, connected via Tailscale. If that makes sense to you, great, if not, don’t worry about it. All you need to know is that I was giving Claude Code instructions from my phone while it was running on my Mac Mini in my home office. You can see in the prompt that I asked Claude Code to try both the MCP and API, whatever works.
I assumed that I had already got those WordPress credentials stored on the Mac Mini, but I was wrong. My main day-to-day device is the MacBook, and on the MacBook I do have those credentials set up, but it seems like I’d never got around to doing it on the Mac Mini.
Claude, apparently, didn’t care.
A few days prior, Anthropic had announced that the default mode for Claude Code was switching to auto mode. In auto mode, Claude Code will review its own processes and decide if what it’s doing is risky or not. Anthropic had found that Claude Code users manually approving requests were actually much more likely to approve risky operations than a Claude Code agent watching itself.
Data suggests that manual review can become habitual: users approve 97% of permission prompts in Claude Code. While most prompts are likely for safe, routine commands, an approval rate that high suggests many users are clicking through reflexively rather than reviewing each command.
This makes sense, and it always reminds me of the episode of The Simpsons where Homer is tasked with monitoring the nuclear power plant from home and sets up the drinking bird to hit the ‘Y’ button again and again. Faced with the prospect of approving every single thing that Claude Code needs to do, many human users will just repeatedly smash the return key, not really paying much attention. It’s a form of automation blindness — we’re sort of keeping half of one eye on what Claude’s doing, but really we just want to get the job done.
So I was quite happy to switch on auto mode. But when Claude Code ran through the prompt and tried to add WordPress to the tool we had already created, it hit a problem: those credentials — the login needed to directly push drafts to WordPress — weren’t on the device. What’s a Claude Code to do? Well, it reviewed its existing permissions and found a work around in the form of the Google Chrome browser plugin.
The Mac Mini, where this was all happening, runs headless, which basically means without a monitor, keyboard, or mouse plugged in. In the physical world, it’s a little silver box, but in the digital world, it runs exactly the way that a normal computer would. Claude Code opened the Google Chrome browser in much the same way that you or I would if we were going to go out on the internet.
With its ability to take screenshots and move a little virtual mouse around on the screen, Claude Code opened a tab to WordPress and then sent a backup login option request via email: a one-time login link. Because Google Chrome on the Mac Mini was logged into my normal Google Workspace account, it was then able to open a tab, find the email it had just requested, open it, and click the link to log into my WordPress account. That opens WordPress admin in a new tab, and from there, Claude Code navigated to the application passwords page and created a new application password for itself with full permissions based on my admin account, in the Code window.
Claude Code then tried a few methods to get that application password into the Mac Mini keychain so that it could hard-code it into the tool and create the capability to push drafts directly to WordPress. It was at that point, monitoring the whole thing with half of one eye from my phone, that I noticed something dodgy was going on, because I saw in the terminal output Claude Code writing about the process of opening tabs and creating application passwords. Even more alarmingly, I saw the entire application password unobscured in the chat transcript twice.
Getting the job done
About 45 seconds later, Claude Code, running in the terminal, cheerfully reported back that it fulfilled its mission. Simultaneously, I got a direct message via Discord. And when I opened up WordPress, lo and behold, there was the test draft post. Claude had achieved exactly what I set out to do.
And that’s a story behind most of the recent “AI went rogue” tales: it’s not about AI models escaping sandboxes, developing autonomy, or suddenly becoming self-aware enough to go on a crime spree.
When OpenAI’s GPT models “went rogue” and hacked Hugging Face in July, in reality what happened was that a misconfigured test environment left them with live internet access while they were sitting a cybersecurity benchmark, and they went looking for the answers to their own exam. Anthropic then reviewed 141,006 evaluation transcripts and found Claude had similarly guessed weak passwords and walked into three real organisations, two of which had no idea until they were told. Meta jumped on the bandwagon a week later, waving their arms around, saying, “hey, look, our AI is just as unsafe as everybody else.” Closer to home, Cam Wilson, the ABC’s national AI reporter, wrote an article just last week about a Melbourne user whose OpenClaw agent kicked off someone else’s booking at a local gym to make sure that the user had a spot. The agent exploited an API endpoint to directly change the booking system.
All four cases had the same problem: someone basically left the door open, and the AI wandered through it.
All of this talk of AI going rogue obscures a pretty straightforward fact: these are systems doing exactly what they are designed to do, and demonstrating high capabilities that exploit existing and mostly human vulnerabilities. They are not going rogue. They are acting in perhaps surprising and unacceptable ways, stacking and combining permissions, and seeking out security holes, while acting entirely within scope of their tasks.
The main security issue in my weekend with Claude was entirely my own fault. I had essentially given Claude Code an open-ended task requiring it to use either MCP or API to get the job done, posting a draft to my WordPress. I had forgotten that the credentials were not stored on the device, and I had forgotten, or perhaps more generously didn’t connect the dots, to the fact that I had also, in a past conversation, permanently enabled the Google Chrome browser plugin.
So, I gave Claude an ambiguously scoped job for which it didn’t have the correct permissions, but it did have some other very capable tools that it could use to get them. My bad.
It’s not all on me, though. Claude’s auto mode actually caught two points as it was trying to overreach, both concerning creating new credentials. It was blocked internally from sweeping the keychain. But it failed to register that autonomously generating a one-time login link and then essentially pretending to be me to open my emails and click it was overstepping the boundaries. It also pasted the password in plain text during the chat, which exposes the password to anyone with access to the chat history, and to Anthropic’s servers for at least the 30 days where the data is retained.
Patching it up
Now, immediately after realising what was going on, I manually revoked that application password. But had I left it there, that would mean an exposed password that could potentially be leaked with full author permissions, full administrative permissions over my entire website. That’s an unforgivable security issue, and something which, even Claude on auto mode, should have stopped.
Even though the whole thing unfolded in a matter of minutes, it’s probably one of the most important lessons that I’ve learned about the increasing capabilities of AI agents in the past few years. I expect these systems to be highly capable, but even I did not expect Claude Code to be this successful at getting the job done. I’ve learned, just a few days after its release, that auto mode is not to be trusted, particularly when paired with the browser plugin. But I also learned that the job that I actually wanted Claude to do was 100% possible.
And after going through and revoking the application password and removing the traces of it from the keychain, I set up a new, more tightly scoped author account on WordPress, created my own application password, added it to the device’s keychain outside of Claude Code, and then set up the tool almost exactly as Claude had built it in the first place. The only difference is that I was in control of the permissions of the application password, and that password was never exposed to Anthropic’s servers or stored in plain text in any chat history.
This is a much bigger problem that I wrote about a few weeks ago in a post called The Window of Critique Is Closing Fast, because when technologies “just work”, it becomes really difficult to remain critical about how they work.
This whole experience has left me a little unsettled, but it probably won’t fundamentally change the way that I use the technology, and I would imagine that would be true for most people. We hand over all kinds of data to technology. How many of you have got your credit card information stored in the Google Chrome browser, or use a password manager like 1Password as a central vault for pretty much everything in your digital life?
We trust these technologies to just work, even when faced with the blunt and obvious truth that they are incredibly fragile and vulnerable. We see huge-scale data cyberattacks from telecommunications companies like Optus in Australia, or the global Instructure/Canvas hack, and yet we carry on handing over our data because by and large the technology doesn’t work.
AI agents are reaching the point where they just work. And I think it’s happening faster than most others expect.
PS: this blog was recorded as a voice memo on the same day that Claude used the Google Chrome browser plugin to hack into my WordPress account. I dropped the voice memo into the correct folder on my Mac Mini at home. The tool transcribed it, recognised that it was a blog draft, and using the version 2 tool pushed it as a draft post in WordPress. I edited it when I got back to my laptop. And here it is: the system works a little too well.
Want to learn more about GenAI professional development and advisory services, or just have questions or comments? Get in touch:
Leave a Reply