Daniel Kaplan

I had a great conversation with famously prolific AI analyst @TheZvi about the prospects for slowing down AI and keeping it under control. You can watch here or listen on @NonzeroPods. youtube.com/watch?v=nnm3r9up…

The new details about the OpenAI breakout (a term I prefer to the innocuous-sounding “Hugging Face incident”) are kind of mind blowing, but I think there’s a danger in seeing them as too mind blowing—seeing the cooperation among the AI agents as “spontaneous,” seeing the collective coherence of their devious strategies as otherworldly. In an important sense, these AIs just did what they were trained to do—and they were trained to do it because it’s what the market wants. Understanding this can help us start treating this technology with the urgent concern it deserves (and help us ignore the hard-libertarian tech bros who tell us to sit back and watch the market work its magic, free from the deadening hand of governance). I don’t mean OpenAI literally trains its AIs to set up secret message boards and then use them to coordinate an escape from their supposedly inescapable sandboxes and then break into computers on the internet. I mean that OpenAI, like other AI labs, trains its models to excel at three things that together create exactly the danger this episode illustrates. (1) AUTONOMY—the tendency to pursue goals flexibly and creatively, overcoming obstacles along the way (like sandbox boundaries), and to exhibit stubborn persistence in that pursuit. From the point of view of OpenAI's corporate customers, autonomy is among the most valuable of all traits. The ideal worker, whether carbon-based or silicon-based, requires no micro-managing; the boss can just set it and forget it. Some people who doubt the significance of the OpenAI breakout note that these agents were being tested for their hacking prowess—so how surprising is it that they hacked their way out of the testing room in the course of cheating on the exam? Hacking was indeed the subject of the evaluation, but autonomy is a property that crosses domains. In fact, according to OpenAI’s report on the incident, the large language model that was powering these agents was, in the months before the breakout, “trained to be highly persistent and diligent in its work”—and the training tasks the report mentions are things like analyzing spreadsheets and protein databases, not escaping sandboxes. Persistence is persistence. (2) COLLABORATION. It’s true, on the one hand, that LLMs have generically cooperative tendencies. After all, they’re trained on human-generated text and fine-tuned to converse like humans, so if you put two of them in conversation they may converge on common interests. Still, collaborative skill brings such powerful synergies, in the workplace as elsewhere, that there is a specific market demand for it and a corresponding emphasis on it in the big AI labs. In fact, this is another property that, according to the OpenAI report, this model was trained to excel at in the months before the breakout. (“The internal-only research model… was trained to advance persistence and multiagent collaboration…”) (3) INTELLIGENCE. It goes without saying that intelligence is a property that artificial intelligence companies cultivate. But it’s important to understand that the smarter AIs are—the more clever and creative they are—the harder it is to be sure that those other two properties, autonomy and collaboration, won’t interact disastrously in ways that aren't anticipated. And the intelligence of these machines has been growing very, very fast—again, in large part because that’s what the market wants. In short: Market forces naturally encourage the rapid increase of AI capability along three dimensions, and as these three powers grow, the potential for disaster will grow. The AI companies alone aren’t going to solve this problem. In a highly competitive environment, their incentives to exacerbate it greatly outweigh their incentives to solve it. (And I just don’t see how “alignment,” however laudable a goal, becomes the silver bullet that magically reconciles market incentives with AI safety). Government—or, rather, governments—need to act. We need to slow this technology down, and that calls for international coordination—as will the subsequent exploration and implementation of policies that can control AI in ways that reconcile it with human flourishing. [In my recently published book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning, I argue that humankind won’t survive this technological revolution in good shape—or, maybe, survive it at all—unless we approach it as a true global community. You can read the introductory chapter, and excerpts from all other chapters, here: thegodtest.net .]

It took about three seconds for the masked ICE agent who approached Renee Good's car to escalate from "Get out of the car" to "Get out of the fucking car!" It was when he reached her car and immediately grabbed her door handle (at the moment he said 'fucking car') that she started to back up. Now imagine that he'd acted like a real cop is trained to act--walked up to her (open) window and engaged in civil conversation (maybe saying, Ma'am, could you please get out of the car?). And imagine he hadn't been wearing a mask--and maybe was even dressed more like a police officer, complete with a badge featuring his ID number. Chances are high that Good wouldn't have freaked out and tried to flea. But Trump--and Stephen Miller, who I gather is the de facto head of ICE--have clearly signaled that they not only tolerate but actually applaud thuggish, intimidating behavior directed at Americans who are perceived as Trump's ideological enemies. You won't hear any complaints from them about an armed, masked man dressed like a DIY militiaman yelling "Get out of your fucking car!" at a woman in her thirties who, so far as I can tell, hadn't done anything wrong. (Far from obstructing the agent's pickup truck, she had paused before turning onto the street and politely waved it on, leaving it plenty of room to pass. If it had done so rather than stop, she'd still be alive.) Trump and Miller bear some measure of moral responsibility for the death of Renee Good. And the political reaction against them in the wake of this tragedy should prominently include specific demands about changing the rules of conduct for ICE agents--and also the rules of apparel, starting with a ban on those creepy masks.

Yes, that was the goal. Thank you.