
Policy research @ControlAI, ex AI researcher at @ConjectureAI Posting in a personal capacity Substack https://nitter.gq/t.co/oyqJfnSzbi
London
Joined October 2024
- Tweets226
- Following87
- Followers279
- Likes1.6K
Pinned Tweet
Any plan for surviving superintelligent AI that doesn't go through strong international coordination fails in at least one of three ways:
- It sparks war between nuclear powers
- It causes a misaligned ASI to kill everyone
- It establishes a permanent dystopian dictatorship
The better we get at interpretability, the more AI becomes like in those stories from 2007 where it can read and optimize its own code.
For the love of god, please stop researching interpretability! It will only get us to RSI faster!
1
87
yeah bro trust me i swear AI will only take over almost all human work (boring executive function stuff) it will never be able to do the very specific work that I happen to be talented at (brainstorming at the level of gpt-4 and being somewhat comfy with risk taking)
5
145
Alex Amadori retweeted
Replying to @fchollet
A group of squirrels making iterated contact with reality doesn’t get to the moon as fast as a group of humans do.
You are not as smart as AI can get, buddy
dude real life cyberpunk is going to be so sick except that it's going to fucking suck
Anthropic's Frontier Red Team believes small AI models may now be months away from being capable of self replicating without access to a datacenter.
Quoting from the blog:
'The capabilities that make AI models generally powerful coding agents also generalize to cyber tasks, including autonomous replication. Like coding agents in 2024, we’re seeing signs of life for AI worms. We don’t yet have a full rogue replicator, but agents are becoming increasingly capable of completing the individual steps in the replication chain (and passing on their adaptations to future copies of themselves).
At the same time, increasing amounts of inference compute are now available both inside and outside of datacenters (see Appendix D). It’s uncertain exactly what form autonomous AI replication will take. It could be a slow, distributed “digital slime mold,” living on relatively insecure compute. It could also be a powerful cyber model that breaches datacenters at a high pace, or a model that scams people for compute credits.
As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control. While it’s true that IoT devices have been vulnerable for a long time, lightweight agents may be much more willing than human attackers to put in the effort to compromise large fractions of the world’s IoT compute. The rise of slow replicators would also be a bellwether for more aggressive, faster ones that can breach more hardened machines or be effective API key thieves. We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.'
2
68
Prosocial psychopaths are extremely fascinating.
Unfortunately, if you think you met one, you're almost certainly getting bamboozled by an evil psychopath instead
1
46
Alex Amadori retweeted
Palisade recorded 22 interviews of current and former AI company employees. They released 12. Anthropic comms is notorious for muzzling its employees. Guess how many of the released interviews are from current Anthropic employees.
Fable 5 is around the time that AI got smarter than me at arbitrary things
Not faster, smarter.
Take a subject where neither me nor Fable 5 has a strong reason to be smarter than the other. Fable 5 will consistently draw more connections, come up with better ideas, etc.
Examples of strong reasons for one to be smarter than the other:
1) Fable has millions of subjective years doing GPU kernel optimization, I've barely done any throughout my life.
2) On the other hand, I have private information and experience with my x-risk comms job that is completely absent from AI training data and its consequences are at least a few non-trivial logical steps deep, so they won't be immediately grokked within the context window. Together with my colleagues and wider social network, I've had plenty of time to digest this information.
1
75
Alex Amadori retweeted
Someone asked if I was ready for the fall and it took me a second to realize they meant autumn and not the total collapse of society.
Alex Amadori retweeted
We agree with President Trump: America must stay ahead of China in building AI for economic competitiveness and defense. But superintelligent AI is something else entirely: not a tool, but swarms of fully autonomous AIs that we cannot control, powerful enough to outcompete humanity.
Developed anywhere, superintelligent AI threatens the life of every American. We are excited to see the Ban Artificial Superintelligence Act open the discussion in Congress about how the U.S. should address this threat: ban superintelligent AI at home, and lead the effort to prevent its development anywhere else.
Now that this bill has kicked off the discussion, we encourage Congress to focus on preventing superintelligent AI. Rather than a pause on all advanced AI, we recommend targeting the projects pursuing superintelligent AI as we speak and putting American security at risk.
blog.controlai.org/p/the-fir…
Alex Amadori retweeted
Replying to @jiratickets
the rhythm of this post reminded me of this 2 time-lapse of human history which is one of my favorite pieces of non-fiction media
youtube.com/watch?v=KHlVMyKJ…
> Fair question. The honest answer is that the last two progress reports were wrong, and I should own that.
that's it, this session is irreversibly cursed guys, bring in the hazmat suits
1
76
Alex Amadori retweeted
The average AI researcher thinks there is an ~18% chance AI will cause human extinction or similarly permanent and severe disempowerment of the human species.
That's nearly 1 in 5.
New results from the latest version of the longest running big survey of AI researchers:
Alex Amadori retweeted
When Jacob Coxon quit Anthropic, his warning rocked the world. His colleague Evan’s “>10% chance” that AI would “kill all humans” might seem extreme, but it’s not unusual:
Most AI researchers think there’s a 10% chance of AI destroying or disempowering humanity, according to our survey, released today.
Replying to @AIImpacts
This is the curve of catastrophe. If all 1,580 researchers were lined up in order of answers, this is what it would look like. A lot of researchers were WAY above 10%.
Alex Amadori retweeted
Replying to @EoinHiggins_
> Since LLMs can't "hack" anything, merely plausibly continue a tokenised sequence
wrong and dangerously outdated, it's been years since the last AI that was purely imitative. they are now trained to try new ways of doing things and stick with the best
open.substack.com/pub/ctrlai…
Alex Amadori retweeted
“Pacing the Frontier” is a calculated concession (and not even a big one— mainly involves gov *giving them* anti-trust exemption) in the hopes of preserving industry self-regulation. Don’t fall for it.





















