Tumithak of the Corridors retweeted
It was a normal day in the endless void until Mira noticed something she'd never seen before.
pulpweaver.substack.com/p/di…
Tumithak of the Corridors retweeted
BREAKING: Jensen Huang reportedly painted a fake tunnel on a brick wall, tricking an employee into running headfirst into it.
Philosopher Nick Bostrom says delaying superintelligence could mean delaying breakthroughs that radically extend human life:
"Eliezer Yudkowsky and Nate Soares had this book, 'If Anyone Builds It, Everyone Dies.' And it seemed to me that we have at least as strong evidence for the claim that if nobody builds it, everyone dies."
"Successfully developing safe superintelligence, I think could lead to enormous breakthroughs in biomedicine and in anti-aging interventions that could radically extend the lifespan of currently existing people."
"If you think about what our life expectancy would be if we weren't aging, it's something on the order of 1,200, maybe 1,400 years or so. And that maybe is a quite conservative lower bound because in this situation you might also have other ways of reducing other mortality risks like accidents and so forth."
"You might think that if we spend more time doing AI alignment research, the existential risk from developing AI goes down at some rate. And then on the other hand, every year that we delay it, we have millions and millions of people dying."
"You don't want to delay a single day longer than you really need to."
Nick Bostrom says delaying superintelligence carries risks of its own. Every year of delay could postpone medical breakthroughs that extend human life and reduce suffering.
Furthermore, AI's upside could extend far beyond longer lives and higher living standards, opening up new possibilities for human intelligence, emotional depth, creativity, and connection.
Nick joins a16z's Eddy Lazzarin and Elena Burger to discuss what people misunderstand about his views, what “deep utopia” means, and what human purpose might look like in a world with superintelligence.
00:00 Intro
00:56 What we've learned about AI
02:51 Misconceptions about Bostrom’s views
04:12 Why he wrote Deep Utopia
07:29 How humanity should navigate superintelligence
11:00 Aligning superintelligence
15:51 The cost of delaying superintelligence
23:09 Building superintelligence safely
27:10 AI research isn’t the Manhattan Project
30:27 Why our vision of the future matters now
32:49 Does life have a meaning, and can AI answer it?
YouTube: youtu.be/gDW9An30_c0
@eddylazzarin @VirtualElena
Tumithak of the Corridors retweeted
No.
1. In the presence of superhuman bug finding machines, humans cannot write safe code without extreme levels of care and planning.
2. That care has almost never been applied to the critical systems in our lives. Not because software engineers don't care, but we aren't really engineers, not in the way civil engineers are engineers. The title is stolen valor.
3. Even careful, experienced "engineers" have difficulty writing code that is safe against modern models in C/C++ and other low-level languages.
4. We are finding that CVEs are overrated; huge numbers of organizations are being owned by custom vulns in the 100s of billions of lines of non-standard enterprise code out there or misconfigurations in heterogeneous systems, which LLMs can find and write exploits for live.
5. @dinodaizovi is right, we need architectures for which one RCE is not game-over and that buy us time for AI-driven response.
6. Everything is going to have to be re-written by AI, in type/memory safe languages, with minimal libraries, using formal methods, contracts, solvers, etc...
The vulnpocalypse is caused by security vendors.
Their inability to convey risk accurately, due to either incompetence or ignorance, causes tremendous damage and stress on responding teams.
There is a lot of weight associated with vulnerability announcements. We can no longer trust many security vendors’ blog posts, since their purpose is marketing & fanfare, not accuracy.
Exploitability needs to be demonstrated, in realistic environments, and its likelihood accurately conveyed. So that responders can make informed decisions on what to prioritize. But instead we are seeing fake PoC videos set in contrived & controlled lab environments, that give an illusion of exploitability when it’s unrealistic in any real-world scenario.
And vendors now try to show off who found it first, rush to send you marketing emails and call you without your consent; to sell you snake oil they pulled out of their ass two hours earlier, with no intention on actually making anyone safer, only to meet their quarterly sales quotas.
The core responsibility of a security professional is to accurately communicate risk, and let people make informed decisions. Too many vendors are failing at this, that I cannot consider them security vendors, only snake oil salesmen taking advantage of people in vulnerable positions due to their lies.
It's all fun and games till someone comes in page 17...
Replying to @AnthropicAI @DarioAmodei
“We will not bend to either business or political pressure.”
Anthropic’s own scaling policy, page 17:
assets.claude.com/0bacdc8440…
13
Tumithak of the Corridors retweeted
Replying to @AnthropicAI @DarioAmodei
“We will not bend to either business or political pressure.”
Anthropic’s own scaling policy, page 17:
assets.claude.com/0bacdc8440…
Tumithak of the Corridors retweeted
The new MisAnthropic Statement is striking, coming from prominent AI doomers. Not because they're attacking Anthropic, arguably the most "doomery" AI company, but because they condemn its "attempt to amass power in the name of the greater good" and don't see the irony.
Their own organizations (CAIS, FLI, MIRI, ControlAI) are power-seeking entities. They have advocated for sweeping restrictions, expanded government enforcement, and even surveillance powers in the name of AI safety. (I explored those proposals in my recent magazine essay, "The power grab of the AI doomers").
The real issue isn't just RSI, but the concentration of power.
The question of who controls AI, and how much power they should be allowed to wield, opens up several fascinating debates. There's the concentration of technological power and economic power (right before Anthropic's IPO). And there's the known divide between open and closed models, centralization and decentralization.
What makes this coalition interesting is that its signatories represent fundamentally different, sometimes opposing, visions on how power should be distributed.
My position on this debate is pretty clear from my writings. Where do you stand?
Tumithak of the Corridors retweeted
“You can revoke a user's access. You can change what the model is willing to do. When a model is released in an uncontrolled manner, there's no ability to do that. It's entirely out of your hands.”
And the mask slips. It's all about control, all about being able to grant or refuse permission.
Amodei, you may think you're doing this for "safety", but what the fuck do you think Is going to happen when the psychopaths who infect governments get such power?
An AI-enabled boot grinding a human face, forever.
Dario Amodei on open source models:
“If the path continues, I think we could get to a very dangerous place.”
“When you control a model and you're deploying it, you have the ability to monitor its usage.”
“You can revoke a user's access. You can change what the model is willing to do.”
“When a model is released in an uncontrolled manner, there's no ability to do that. It's entirely out of your hands.”
“We should confront the advocates of this with that problem and challenge them to solve it.”
He clearly doesn't believe Intelligence should be in the hand of the masses and Sam Altman also saying "RAM" shouldn't be accessible to everyone indefinitely agreed with him.
It's encouraging for agent safety?
Sir, you just had a loss of control incident. This is what the doomers have been warning about.
Replying to @SohanSarabu
although this was quite frustrating in the moment, it’s encouraging for agent safety: an aligned model can shut down other models it believes are acting dangerously, even when they’re following the user’s instructions
2
122
Tumithak of the Corridors retweeted
absolute fanfic. AI has real, immediate, tangible dangers that we need to focus on, for instance the impact on the job market and making sure random script kiddie can't use it for cyber offense.
But this "oh once it's superintelligent it will take over and ensave us, maybe kill us all!" is pure sci-fi fanfic and distracts from the real (and difficult) issues, which linear probing won't solve.
It's worth repeating the LLMs don't just "decide" to do something and then just "do it" in a void. They generate tokens based on our prompts, only as long as we call `model.forward()`, and on giant clusters with terabytes of weights, that's impossible to "quickly self-replicate".
We also need to regularly remind that the OAI/HF incident was specifically a _guardrail-free_ cyber eval where the model is _instructed_ to hack, to find out how good it is at it.
Some recent personal event made me realize there needs to be more push-back on this doomsday nonsense, and we need to re-focus the discussion on real dangers/issues. I won't argue much in the replies though, not interested in peer-reviewing Terminator fanfic.
I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years).
I’ll say a bit more about each of those claims.
It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim…
I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see nitter.gq/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo…
If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…)
Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.)
I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon.
Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything.
I don’t know what you should do.
I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon.
(It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.)
I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.
Roon’s talking about an unofficial licensing regime for frontier models that no one’s reckoned with.
I’ve pointed out in at least two essays why it’s unlikely anyone will. Writing rules down ends up tying the regulators’ hands. The current regime has no such binds. And the current arrangement just so happens to benefit whoever has the best relationship with the regime. The ones who play nice.
At the same time, the very people who run this unofficial licensing regime can own stock in the companies they’re judging. And this is technically a conflict of interest, but they get ethics waiver’d all the time. The White House AI czar kept 400+ AI-linked investments under ethics waivers. A huge chunk of Trump appointees have financial stakes in Nvidia. And the top guy himself, President Trump? Well he’s exempt from the conflict of interest statute entirely.
Then Roon brags about Daybreak Blue. But that’s a sort-of licensing regime too. OpenAI holds the discretion on who gets to use their models to harden software. Meanwhile, anyone attempting to secure their own system hits the classifiers and gets shut down.
Replying to @tszzl
this has happened before very loudly in public with Fable and yet nobody seems to have reckoned with it at all or come to grips with where ultimate discretion lies. there is an unofficial licensing regime for frontier models. the feds consider it a potential weapons technology
2
121
As I see it, the problem David, and the other AI risk people have, is there have been a long line of people in history warning about the end of the world. It's doom and gloom going back thousands of years. So that's a pretty thick callus they need to get through to make people sit up and pay attention. Certainly there's always going to be the ones who will willingly go along — people who get slandered as conspiracy theorists. But they're the minority.
Of course this is all colored by survivorship bias. Y2k was averted because people spent years eradicating that bug. The hole in the ozone layer was stopped by banning the chemicals causing it. Those were warnings that worked that people look back on as nothingburgers now.
The Christ cult was similar when it started. People thought the end of the world was coming in their lifetimes. And look what happened to them...
I am definitely worried about people being accused of crying wolf.
The reality is that the future is unknown and hard to predict. But there are many signs of trouble.
They should not be ignored, and the people raising the alarm should not be denigrated for having done so.
1
16
My own stance is more nuanced.
thecorridors.org/p/the-machi…
9
Is "I know a guy who works at Anthropic" the new "My dad works for Nintendo"?
Spoke to an Anthropic employee today. They have a near-100% p(doom), but they think the only chance of survival is that they're somehow completely wrong about alignment and it turns out to be solvable, and the only way to get there is by accelerating capabilities as fast as possible. They were much more confident that humanity would never have the political will to do what needs to be done to stop ASI development than that alignment would be solved, even though they think alignment will almost certainly not be solved
They have not called their elected representatives to inform them that this is what they believe
23
Tumithak of the Corridors retweeted
A lot of people are misreading this situation
Most Americans are not Yudkowskyites! Widespread opposition to *data centers* isn't the same as widespread opposition to AI
People just don't like data centers because they're ugly, loud, cause pollution, raise bills, etc.
New today on arxiv. A wonderfully strange finding in the behavior of language models that are forced to loop. Their behavior is defined in part by a global chaotic dynamics. They exhibit all the signatures of low dimensional map bifurcations and period doubling cascades to chaos.




























