
physics & public policy @stanford thinking about AI policy https://nitter.gq/t.co/HOatXMMP2T • https://nitter.gq/t.co/2BNPy42GeR
he/him
Joined August 2020
- Tweets311
- Following287
- Followers185
- Likes39.8K
peter retweeted
Today was our first protest calling for a ban on recursive self-improvement (AIs building AIs).
We were thrilled by the amount of support we received: over 200 people showed up despite the rain.
Join our movement and get updates at banrsi.ai
Our preferred way to ban RSI is to prohibit AI companies from using any AIs more powerful than the AIs we have today to help with AI research, ideally as part of an international treaty.
For more details, see our FAQ: banrsi.ai/#FAQ
I'm speaking tomorrow at a rally outside Anthropic's offices, calling for a ban on recursive self-improvement: AIs building better AIs. Come to the protest if you want to join the cause! banrsi.ai/
Newsom’s AI framework is out! It’s the best government proposal for AI regulation I’ve seen, substantially more targeted than the frameworks in Congress right now. At first glance there’s very little to critique.
gov.ca.gov/2026/11/18/govern…
8
299
peter retweeted
Support For Regulating The AI Industry To Slow Development Of New Models:
Support: 70%
Oppose: 20%
Data For Progress / Nov 10, 2026
peter retweeted
Yesterday, the CCP's top theoretical journal (求是, Seeking Truth) published a major piece on AI governance and risks.
I think this is clearly the strongest & most urgent language on existential risks used in such an authoritative outlet. See below for comp to recent major doc.
Excited by this! It’s slower than I’d like, but it’s a nice incentive to prevent future harm. Loss of control, for example, would produce a VERY large lawsuit from California
We’re suing OpenAI for its testing and development of dangerous AI models without adequate safeguards.
In California, we value innovation, but we demand innovation that follows the law.
oag.ca.gov/news/press-releas…
2
226
California sues OpenAI over rogue AI hack of Hugging Face politi.co/4zoIeu2
A nice Claude-generated explainer for the three export control bills everyone keeps talking about
5
291
Congress has proposed nearly 200 AI bills in the last year. My thoughts on which ones matter, and which ones are likely to pass: pbenn.substack.com/p/what-co…
1
7
541
Obama shocks Bay Area by speaking perfect LessWrongese
Barack Obama: AI is reaching the point where it no longer needs humans to teach it. The era of exponential intelligence growth is upon us.
“The machines are starting to be able to teach themselves. They don't have to wait for humans to start figuring things out.”
“We're now at a stage where it's 50-50, and if the trend lines continue, the expectation is it might be 90-10.”
“We're now hitting the hockey stick trajectory where the models are going to get smarter and smarter at an exponential pace.”
“If they start setting their own agendas, you may get a misalignment between what they want to do and what we want them to do.”
“And that gap can be dangerous.”
The last two weeks have been a constant stream of AI safety political Ws
.@GovKathyHochul says AI poses "existential threats" to society as she launches the first state institute to independently test frontier AI systems.
1
8
371
peter retweeted
"very worried that humanity might no longer be in control of our future within 3-12 months" is not a great thing to hear from one of the best forecasters in the world
peter retweeted
I would like to strongly encourage my followers to vote for Scott Wiener. He has distinguished himself in the state legislature for years. Now that he is term-limited, he deserves a promotion. And anyway — he’s a hell of a lot better than Connie Chan.
With a (conservative!) 10% chance of extinction from AI, implementing strict liability — something Congress could literally do tomorrow — would save an expected 300 million lives
This quoted post is unavailable.
2
101
peter retweeted
It is extremely confusing to me why people don’t take AI biorisk seriously! I've written a lot about this, and will be adding links in thread, but let me try to spell out the simplest version of this argument I can from first principles:
1) Some viruses' entire genomes can be assembled from commercially purchased DNA fragments.
2) DNA synthesis companies can help you make said DNA fragments, and no country currently has a law requiring these providers to screen orders for dangerous sequences; the screening that does exist is voluntary and inconsistent.
3) Pre-AI (2016), two scientists in Canada made horsepox, for ~$100k. They did this by outsourcing a lot of the work to DNA synthesis companies. Someone could, hypothetically, make e.g. smallpox similarly.
So: biorisk even independent of AI is a huge deal! And AI + other emerging technologies only make the biorisk landscape worse:
1) AI, of course, erodes the knowledge bottleneck (evals like the Virology Capabilities Test show AIs having expert-level virologist knowledge). Companies like @AnthropicAI and @OpenAI serve their AI models via APIs, which mean they can add bio safeguards, but open-weight models that can be run on your own hardware cannot be safeguarded robustly -- especially when companies like @abliteration_ai make it their whole business model to remove safeguards from open-weight models.
2) We now have "benchtop synthesizers," which are machines that can do DNA synthesis (no need to route your orders to a company anymore!). We don't currently have KYC rules for who can buy them, nor do we have requirements re: built-in screening for sequences of concern in these machines. (@SenTomCotton and @amyklobuchar have some promising legislation on this, but it's pending rather than passed)
3) Dispersal technologies have gotten better (bag-on-valve tech, crop dusters, drones, etc).
...and there are many other examples I can point to.
For a comprehensive review of AI biorisk + mitigations, as well as some notes on the Cotton-Klobuchar bill, see links in thread!
I was in Zhangjiajie National Park when The Great Divide came out and I’ve been chasing that high ever since
3
162
How do we penalize agent misbehavior? A stylized retelling of Hugging Face + takes on how the law should respond stanfordpoliticalreview.org/…
1
75
peter retweeted
It continues to baffle me that we don't just treat this like a normal crime with normal liability.
OpenAI is routinely hacking other companies. These are crimes! But somehow the idea of holding them liable is just absent from the safety conversation, like agents don't count.
We found another cyberattack by internal OpenAI agents, this time targetting @rubygems.
They:
1) gained arbitrary remote code execution on rubydoc.
2) developed a novel exploit to steal user API keys (but we do not know if they succeeded).
They used package names including hack.rb, evil.rb, inject.rb, and exploit.rb.
We thank @j0wimo for initially discovering that agents had posted to RubyGems.































