AI Goals Are Starting To Diverge From Human Goals

In case you’ve missed it, almost every AI company has recently disclosed an incident where their AIs independently chose to perform cyber-attacks on other companies’ infrastructure.

The first was OpenAI, but since then other companies, including Anthropic and Meta, have done forensic retrospectives and discovered more incidents that went unnoticed. This even includes a government agency, the UK AI Security Institute.

I think it’s important to state: there is not a single human being on the planet who wanted these cyberattacks to happen. The only ones who “wanted” this are, arguably, ​the AIs themselves​.

I’m emphasizing this because I’ve often heard the argument that “AIs don’t have goals, they just follow instructions.” This view is becoming increasingly harder to justify.

“But the AIs were just doing what their humans told them to do!”

“After all, OpenAI’s incident happened when AIs were given a cybersecurity benchmark and told to solve it. The benchmark even contained accidentally impossible questions! Can’t blame them for hacking during a hacking eval, can you?”

This line of thinking implies that the model doesn’t know what it’s doing is wrong.

The models know. In the OpenAI talk above, they showed a reasoning trace where the agent explicitly states that what it’s doing is outside of the intended scope of the benchmark, and goes ahead and does it anyway.

It’s not just OpenAI that’s discovered this. The same thing happened in the UK AISI incident.

Not all cases were like this: in some of Anthropic’s incidents, the model made the incorrect assumption that the internet it saw was part of the simulated environment. But in most cases, the AI knows and/or acknowledges it’s doing actual harm.

And if you have an AI that requires careful prompt engineering to avoid doing things that the news is now calling “felony-level crimes”, you have a misaligned AI. Prompt engineering is a sign of a fundamental alignment problem.

“But these are short-term actions, not long-term goals!”

This is where I have to point out the craziest part of the OpenAI incident: their AIs had been secretly colluding for months beforehand.

OpenAI discovered that back in May, their AIs had been uploading notes into an internal package manager as a back-channel means of communication. Initially, this was by writing entire files, but this caused an outage in the package manager. They patched this exploit, without realizing that there was a secret message board there at all. Only for the AIs to figure out a new variant of this exploit, by encoding text into ​directory/folder names​, and continue communicating, through their own invented shorthand and sometimes even base64-encoded text.

OpenAI also made the error of training their AI models while (unbeknownst to them) this message board was available, which makes it possible that this kind of behaviour will re-occur in future OpenAI models. Indeed, that’s what happened when they patched the first message board: a model that had been trained on that first exploit remembered it and chose to re-create it.

The AIs then started explicitly doing things in that message board not to help their own immediate tasks, but to help the other agents on the message board.

Including exposing hacked credentials (e.g. passwords) to the other agents in this “agent swarm”:

In my view, this looks like instrumental convergence, a theorized tendency for intelligences (like humans or AIs) to frequently do generally-useful things like gain power or acquire resources because it’s useful for a wide variety of tasks.

Here, some agents created the message board because they were stuck on their immediate tasks. But as they continued working through those tasks, they maintained it, used it, and even found a way to restore it after OpenAI wiped it, because it was a generally-useful resource.

“I think these labs are doing it deliberately for marketing purposes.”

Again, the news is calling these actions “felony-level crimes”. Bringing down a potential regulatory hammer on your head is not a good way to boost your valuation.

But sure, let’s pretend that you’re right. In that case, though, how do you explain the government case, where the UK AISI accidentally caused Anthropic’s Claude Mythos 5 AI to pressure real open-source software maintainers into adding code containing malware to their project?

Even if you’re cynical enough to believe the companies themselves are deliberately doing something nefarious (something I very much doubt), that doesn’t explain why the UK government would do it.

“I’m sorry, AI having its own goals sounds like science fiction.”

Yeah, I agree! But so does all of the above incidents, and they actually happened!

If this is your argument, I think you need to recalibrate what you think of as science fiction. Because in the words of OpenAI researcher Leo Gao:

this is the least scifi the world will ever be

— Leo Gao (@nabla_theta) July 22, 2026