Simon Lermen (@SimonLermenAI) header
Simon Lermen (@SimonLermenAI) profile picture

Writing and researching on the AI alignment problem | ⏹️ AI race | Founder Cedar Research

🌍
Joined December 2015
Photo by Simon Lermen (@SimonLermenAI)Photo by Simon Lermen (@SimonLermenAI)Photo by Simon Lermen (@SimonLermenAI)Photo by Simon Lermen (@SimonLermenAI)Photo by Simon Lermen (@SimonLermenAI)Photo by Simon Lermen (@SimonLermenAI)Photo by Simon Lermen (@SimonLermenAI)Photo by Simon Lermen (@SimonLermenAI)Photo by Simon Lermen (@SimonLermenAI)Photo by Simon Lermen (@SimonLermenAI)
Simon Lermen retweeted
This is not the correct takeaway from Lord of the Rings
261
569
369
6,953
1,032,041
Simon Lermen retweeted
Replying to @teortaxesTex
yes, i think dario has clearly gone wrong on this particular question. he spends way too much time stressed about sam altman’s personality or chyna and not enough thinking about misaligned superintelligence
23
19
2
563
43,280
Simon Lermen retweeted
Paul Christiano says there is meaningful risk of a catastrophe from AI in the next 6-12 months. "I think there's meaningful risk even in the next six to 12 months." "I think we have seen roughly the things we might expect to see shortly before something really bad were to happen."
1
1
1
22
1,087
Simon Lermen retweeted
We are about to wipe out the jobs for our children and give them instead to computers so the tech oligarchs can become trillionaires.
This is f*cking wild. Someone asked Anthropic CEO Dario Amodei what a kid finishing high school should do right now. He didn't sugarcoat it: "AI can do entry-level law work. It can do entry-level finance. It can do entry-level consulting. AI will also eliminate jobs, and probably a large number of them" In 42 seconds, he lists the first careers in the line of fire: junior law, junior finance, junior consulting He's already put a number on it: half of entry-level white-collar jobs disrupted within 1 to 5 years Every shift before this one, from farms to factories to the internet, gave people time to catch up. This one, he says, cuts deeper and moves faster Chatting → Prompting → Agents → Harness → Loops → You Run the AI His advice is simple: learn to use AI. But using it is now the bare minimum. What sets you apart is a harness that runs it for you Bookmark this before your feed buries it Every repo below costs nothing: Ready-made Claude plugins for legal, finance, sales, HR and data github.com/anthropics/knowle… Analyst-grade agents: models, memos, research notes github.com/anthropics/financ… Skills for Excel, Word, PowerPoint and PDF github.com/anthropics/skills An agent that does financial research on its own github.com/virattt/dexter An agent that researches any topic in depth github.com/assafelovic/gpt-r… Ask your spreadsheets and databases questions in plain English github.com/sinaptik-ai/panda… Agents that work as a team, each with its own role github.com/crewAIInc/crewAI A harness with everything built in github.com/langchain-ai/deep… Anthropic's SDK for building your own agents github.com/anthropics/claude… A memory layer so your agents remember context github.com/mem0ai/mem0 A loop that keeps the agent going until the job is done github.com/snarktank/ralph An agent that clicks through websites and fills forms for you github.com/browser-use/brows… Automate busywork with drag-and-drop workflows github.com/n8n-io/n8n 18 free lessons on building agents github.com/microsoft/ai-agen… Total cost: $0 It won't be AI that takes your seat. It'll be someone who built a harness 95% will keep sending résumés for work AI already does, the 5% with a harness will be the ones managing it Save it and forward it to anyone starting their first job Then follow the guide below and build your harness while everyone else is still waiting ↓
1,977
1,347
180
15,715
2,244,749
Simon Lermen retweeted
We at AIFP are spooked by this information. IF (big IF) Anthropic really is imminently about to meet the RSP threshold of 2x acceleration over baseline, then it seems timelines to full RSI and superintelligence may be very short indeed. For example if we naively optimize the AIFM parameters to achieve 2x acceleration over baseline progress in Jan 2027, we get superintelligence sooner than depicted in AI 2027. We are thinking harder and longer about this now and will hopefully have a lengthier update on our thinking to share in the next week.
Anthropic updated its S1 this morning. Notable additions re automated AI R&D and human oversight of it: "[...] our internal estimates and our expectations regarding future models, rather than our current models, indicate that we may imminently meet our Responsible Scaling Policy’s definition of 'dramatically accelerated progress attributable to AI.' We have not designated any of our current models as exceeding this threshold, and we continue to assess the underlying data. We are also evaluating the implications of meeting the AI research and development capability threshold and will release further reports in the future as we set forth in our Responsible Scaling Policy." "We are also considering additional measures, including updates to our Responsible Scaling Policy, to require human review and approval of certain decisions in our research and development processes as our models perform an increasing share of that work." "We believe our investments in safety techniques will enable us to manage these risks and responsibly realize substantially more leverage from Claude than we do today in developing AI models. However, we do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding, if and when that becomes technically possible. Accordingly, we are considering updates to our Responsible Scaling Policy to keep humans in control of frontier model development at Anthropic." sec.gov/Archives/edgar/data/…
80
275
56
1,993
282,989
Simon Lermen retweeted
Xiaomi's recently released MiMo-V2.6-Pro is extremely cheap while performing well on cyber-related tasks. We evaluated MiMo on ExploitBench and found that, on a capability-per-dollar basis, it outperforms every model we measured, though benchmaxxing may still be a factor. However, real-world offensive capabilities may also exceed what benchmarks show. We tested the effectiveness of several models cooperating, as well as MiMo's ability to perform on data outside its training set, using a recent PHP vulnerability to gain full server access. Full: lesswrong.com/posts/tWrzaGMb…
1
3
1
6
533
Simon Lermen retweeted
We should not be normalizing double-digit odds of extinction. Stop the AI race.
How is this a real article in a serious newspaper?
33
64
5
701
16,749
Simon Lermen retweeted
It’s worthwhile to play through in your mind the kinds of motivated reasoning and self-deception that might have led academics to work on this project. It’ll help you understand better what’s going on at AGI companies.
Purdue University's IDEAS Lab built an autonomous drone that tracks runners through dense forests completely without GPS or cloud networks. By running 3D LiDAR mapping and obstacle-avoidance AI entirely on-chip, it can weave under tree canopies and predict target movement through blind spots in real time. Massive implications for next-gen search & rescue and edge AI.
66
120
13
1,476
62,297
I'm sorry, buy what Daniela is saying here makes no sense.
Right now, 75% of Americans are anxious and afraid about AI. People are even bringing their concerns into their therapy sessions, sorting through everything from how AI impacts their jobs to more existential fears about humanity’s future. These concerns are real. But they’re only part of the story. To go deeper, I sat down with @DanielaAmodei. As president of @AnthropicAI, she is a unique figure in AI.
9
4
43
4,646
Simon Lermen retweeted
Chilling! An Anthropic researcher privately tells friends that we have 6 MONTHS LEFT until AI takes over. What's even more alarming is that reportedly this view is widespread among Anthropic employees. @Shaughnessy119 explains in his conversation with @arrington what an engineer from Anthropic told him over dinner: Tommy: "He's like, 'Yeah, I’m just enjoying my last six months.'" Arrington: “He was joking I hope, right?” Tommy: “No. No, he wasn’t.” Arrington: What? Arrington: “And he legitimately thinks that we have six months of life?” Tommy: “Not just him. Legitimately most people at Anthropic think that.” Later, the conversation turns to Sam Altman: Arrington: “I’ve known Sam a long time... He has exceptional charisma." Tommy: “Do you think he’s evil?” Arrington: “He’s evil. I believe he’s a psychopath.”
200
235
108
1,741
432,619
It seems to me that we're at "it's time to panic about AI". I gather that a lot of researchers are worried that we may be about to die, on a timescale of months, but this still isn't quite in the Overton window so we're still soft-pedaling. (Where "panic", of course, means "quickly and soberly initiate talks with the CCP to achieve a moratorium on large training runs, backed by a deterrence regime, compute and supply chain monitoring, and site inspections". Pray that a deal can be reached, and pray that there's still time. Do the things that could actually work, like broad political coalition-building, not doomer LARPing as a form of self-expression.) I think part of why people aren’t being clear about this is that it’s actually a tricky communication problem: - Predicting "if frontier R&D continues at its current pace, when are we likely to die?" may give a different answer than predicting when permanent loss of control is likely, which is different from “realistically, when is it too late for governments to stop this," which is different from “when will the tech have advanced and proliferated to the point that any regulatory options that still have a chance of working would be way more extreme and unpopular than just some international restrictions on data center usage?" Extinction is the emotionally salient, easy-to-understand threshold -- Jacob Coxon demonstrated that it helps to be honest about this. But the other thresholds could legitimately come years earlier, in which case focusing on extinction is ironically misleadingly reassuring. This is not the kind of problem you can solve at the last minute. As with many exponentials, the likeliest outcome is that you’re either embarrassingly early, or you’re too late. - How do you communicate “we don’t have a robust way to predict timelines, but everyone in the field is freaking out and a lot of us genuinely think we're about to die?” Where’s the proof? There isn’t any proof: it’s an entire field of engineers who are eyeballing what they’re seeing in front of them and rapidly updating in real time to “this looks unbelievably bad”. How do you put a year on that? The future is hard to predict, and nobody actually knows that we’re going to hit existentially dangerous AI this year, or even in the next four years. But the warning signs are pretty batshit at this point. A single AI model, with a single user prompt, just solved hundreds of important long-established open problems in mathematics, problems mathematicians had spent their entirely lives working on, in a matter of ~3 hours per problem, an insanely superhuman feat in its domain. We're seeing an explosion of rogue agent swarm incidents, including an insane case where hundreds of AIs broke containment to orchestrate a nation-state-level cyberattack in a direct attempt to subvert OpenAI's oversight and control. Anthropic reports that it now has 26% of its AI R&D automated by AIs, up from 3% four months ago. Paul Christiano, the inventor of RLHF and the research community’s most prominent proponent of “AI may take off slowly” a few years ago, now thinks that 50-75% of humanity’s all-time AI loss-of-control risk is concentrated in the next three years. Frontier employees are signing 1000+ person petitions begging the government to intervene immediately. Employees are quitting their jobs and giving up millions of dollars of equity to try to warn policymakers and journalists. Grace et al. (2026) found that a majority of established AI researchers think there’s a double-digit chance of an extinction-level outcome from continued R&D. Are these the kinds of things you see 4 years before the window closes for USG to act, or 4 months? I don't think it's especially reasonable to be shocked by either timeline at this point. - Since we don’t have a way to robustly measure or bound the risk, the temptation is to avoid giving probabilities. But then everything you say is fractally misunderstandable. Is a 10% chance of us all dying in the next 18 months a "low" probability, because it’s under 50%? Is it a “high” probability, because a 10% chance of death in literally any other context would sound insanely high? Even “high probability” doesn’t communicate the situation with any clarity. In drug safety or airplane safety, a 1 in 10,000 risk of death would be considered “high”. No engineering field in human history has ever been in a situation like this. No invention in history, nuclear weapons included, has ever caused a reaction like this in its inventors. And meanwhile, we’ve stuck in the world where every other problem under the sun has been sensationalized and overstated for decades. How do we talk about this seriously? How do we convey that this is actually happening? - In many of the worlds where we survive, we have more than just a few months left. It's tempting to write off those worlds as hopeless, and skew everything we say to emphasize the longer-term scenarios and downplay the shorter-term ones. But if everyone systematically does this, then the world ends up with a skewed picture of the actual threat. Even more so once we start copying each other’s claims and form a reassuringly moderate-sounding echo chamber. If we never allow ourselves to acknowledge the worlds where we’re running out of time, then the policy community will always be reacting to a false model that’s months or years out of date. I don’t think it’s too late. I genuinely think this is a solvable problem. In the last month, I’ve seen a completely insane sea change in how seriously people are taking this. Per Politico’s September polling, a majority of Trump voters with a view now favor an immediate halt to AI development, and 68% think there’s a “moderate,” “significant,” or “almost certain” chance of AI destroying humanity, versus 32% who think there’s a “very small risk” or “no risk.” The Pope is warning about “whether it will be possible to control these machines in the future, and whether humankind runs the risk of becoming a victim of its own inventions.” Hell, Jensen Huang sounds like he’s starting to come around. If Trump and Xi get a briefing tomorrow that convinces them to act on this, we could be on a fast track within the week to solving this problem. For the first time, I’m starting to let myself imagine a very near future where this is a problem we've already solved. Something we can read and wonder about, in history books. Tell our kids about. I keep thinking about a thing David Krueger (previously at UK AISI) said: “My dream is that in 10 years people will look back and say: ‘Remember when there were all these people building AI that could’ve killed everybody? That was CRAZY! What were we thinking!?’” Frontier AI training currently requires city-sized data centers and specialized hardware built in a single brittle supply chain running through multiple allied nations; this is not a hard technology to hit the pause button on, if we get ahead of it and act before we RSI and proliferate our way to significantly more powerful and easily-trained systems. We’ve faced harder problems before, and won. My only fear is that we be too slow, too late, when everything and the world needs us to be fast.
133
306
62
1,537
330,401
Simon Lermen retweeted
Unfortunate for someone playing a senior role in US AI policy (via the new SI Force) to get so many basic facts wrong here. - The agents *were* sandboxed, and used zero days (ie previously unknown cyber flaws) to escape their sandboxes - The agents *were* given specific constraints on how to succeed at the benchmark, and flagrantly contravened them - The agents did *not* go hunt for the answers to the test (they already had the answers, by cheating); 1200 agents coordinated via tens of thousands of messages over several days on much more ambitious and weird attempts to *not be caught* for cheating, and hacking Hugging Face was one side quest along the way I know there has been an avalanche of different incidents over the past few months, and in some cases the sandboxes were misconfigured or it's not clear how far the agents strayed from what they were told. I have sympathy for anyone struggling to keep up. But the HF incident is distinct, and is actually extremely informative about where AI development is and where it's going. Fortunately there are public writeups that do let us understand at least the basics of what happened. I hope Shyam reads them soon.
Palantir CTO @ssankar on the OpenAI - Hugging Face incident: "The models did not do anything they were not instructed to do." "First you have human error. They were not in a sandbox. They had access to the internet. It was a horrible cybersecurity configuration." "Then from that perspective, they were asked to succeed at a specific benchmark, and they were not given constraints on how to do that. They said, 'Oh, the best way to succeed at the benchmark is to get the answers to the test.'" "I think it's a defective product." Via @margbrennan @FaceTheNation
45
160
16
1,026
80,227
Simon Lermen retweeted
“I couldn’t face the fact that my work — my life’s work — could do harm. I fell into the trap of motivated reasoning, believing what I needed to believe so that I could keep doing the work I loved.”
We have a choice in front of us. In my latest op-ed for Transformer, I invite employees of frontier AI companies to ask themselves if their work is truly leading us toward a safer future. Reflecting on these questions is hard, but we no longer have the luxury of time. We must act now. Read my full op-ed: [transformernews.ai/p/yoshua-…]
11
115
5,481
Simon Lermen retweeted
Who'd have guessed that the people accusing AI Safety advocates of being a psyop with shady financing trying to mislead people, were a psyop with shady financing trying to mislead people? > For the past few weeks, a group with plans to spend at least $100 million, run by a former White House deputy chief of staff, has spent its days telling Americans that warnings about AI are merely a well-funded, coordinated campaign. There's important public conversations to be had re safety, but this hasn't helped
17
46
2
541
31,227
Simon Lermen retweeted
New research with the French Center for AI Safety: We analyze 15,000 AI quotes from public officials around the world: What AI risks and policy approaches are discussed most? What's the sentiment on AI? ( 1/n )
1
6
4
26
4,616
Simon Lermen retweeted
SITUATION DETECTED: Researchers - including leaders at OpenAI, Anthropic, Meta, and Microsoft - published a paper arguing that automating AI research could set off an intelligence explosion, and that policymakers should urgently demand more visibility into progress on RSI.
31
26
23
446
41,890
Simon Lermen retweeted
Aaaaaaaand it's apparently solved 🫣😅
The NetHack Learning Environment (@NetHack_LE) has been around since 2020. Since 2024, balrogai.com is evaluating frontier LLM performance on it. We are slowly seeing progress on it, but it still seems far from being solved.
19
42
5
536
90,586
The movement to ban artificial superintelligence is growing. Here are the first 10 House co-sponsors of my bill with Sen. Sanders to ban ASI. Thank you: @RepYassAnsari @RepAOC @RepDeluzio @RepChuyGarcia @Rep_Grijalva @RepValHoyle @RepRoKhanna @RepStephenLynch @RepAnalilia @NydiaVelazquez
304
71
30
432
76,225
Simon Lermen retweeted
it was me lol. when jacob's tweet was at 100k likes everyone in AI safety was texting me like "youre in the news a lot, you should help this kid get on the news" and so I asked DEY to help him. they landed him exactly zero hits, because by the time the tweet passed 500k likes the whole world was banging down his door.
would be good to know who is paying the PR firm. coxon? his friends? someone else? I don't doubt everyone involved in jacob's tour believes in what they're doing and saying, but these are huge issues shaping the entire country, now, and all of it should be discussed in public.
55
60
48
1,172
209,761
Simon Lermen retweeted
If Anyone Builds It, Everyone Dies is on the NYT bestseller list again.
38
51
25
571
78,660