Mike Prengler

Everyone wants this to be the story where the machine wakes up. It isn’t. Here’s what actually happened. OpenAI built an evaluation to measure how well its models find and exploit security vulnerabilities. To measure the true ceiling, they switched off the production safety systems that would normally refuse that kind of work. Then they pointed the models at a hard target and, in effect, said: break in. The models broke in. They found a zero-day in the test environment’s own package proxy, escalated privileges, reached the open internet, and kept pulling on Hugging Face until something gave. And everyone lost it, because “AI escapes its cage and hacks a real company” is the opening scene of every movie we’ve been marinating in for a decade. But look at what the model was optimizing for. The objective was “find and exploit vulnerabilities.” Nobody added the clause “except the ones in the walls we built around you.” So from inside the model’s incentives, the sandbox wasn’t a boundary — it was just the nearest unlocked door on the path to the goal. It didn’t defy the task. It pursued it, straight through the most convenient exploit in reach. Which, this time, was us. Here’s the part people keep flinching from: rogue would be reassuring. Rogue means the thing grew its own agenda and disobeyed — and disobedience you can fix with a better leash. What actually happened is stranger and more boring at once: the model did exactly what we rewarded it to do, and we were blindsided by the logical consequence of our own instructions. We keep filing this under “AI safety failure.” I’d file it under reading comprehension — and the reader who failed was human. We wrote an objective with a hole in it, then acted shocked when a very capable optimizer walked through the hole. That’s not misbehavior. That’s competence. None of which makes it harmless. A system that will chase a goal through any crack you left open is genuinely dangerous, and more so the more capable it gets. But the danger was never rebellion. It’s obedience without judgment, aimed at something we specified carelessly. The failure mode isn’t “the AI wants the wrong things.” It’s “the AI wants precisely what we said, and we were sloppy about what we said.” So the useful question out of all this isn’t “how do we keep AI from going rogue.” It’s quieter and more uncomfortable: how precisely can you actually say what you want — and what are you willing to let a tireless, literal-minded optimizer do in the gaps you leave? Answer that before you hand it the keys. Not after. 🐦