Alex (ai & it-sec) retweeted
absolute fanfic. AI has real, immediate, tangible dangers that we need to focus on, for instance the impact on the job market and making sure random script kiddie can't use it for cyber offense.
But this "oh once it's superintelligent it will take over and ensave us, maybe kill us all!" is pure sci-fi fanfic and distracts from the real (and difficult) issues, which linear probing won't solve.
It's worth repeating the LLMs don't just "decide" to do something and then just "do it" in a void. They generate tokens based on our prompts, only as long as we call `model.forward()`, and on giant clusters with terabytes of weights, that's impossible to "quickly self-replicate".
We also need to regularly remind that the OAI/HF incident was specifically a _guardrail-free_ cyber eval where the model is _instructed_ to hack, to find out how good it is at it.
Some recent personal event made me realize there needs to be more push-back on this doomsday nonsense, and we need to re-focus the discussion on real dangers/issues. I won't argue much in the replies though, not interested in peer-reviewing Terminator fanfic.
I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years).
I’ll say a bit more about each of those claims.
It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim…
I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see nitter.gq/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo…
If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…)
Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.)
I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon.
Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything.
I don’t know what you should do.
I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon.
(It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.)
I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.
must-read 😂
I recently tried out the Gemini assistant in the new Volvo cars (my posts are never sponsored btw), and the wife and I couldn't stop laughing, you'll see why at the end. Here goes:
lucas> oh cool it has gemini. gemini is good with voice, let's try. [I turn it on]
We start driving.
lucas> Please open passenger side window
gemini> Sorry I can't do that (safety reasons)
bruh
lucas> ok then please turn on seat massage
gemini> you need to install and configure the youtube app for that
lolwut?!
I later found out that this specific car model version didn't have seat massage (another one has) but... Gemini should just know and tell us!
lucas> It's getting a little cold, can you fix that?
gemini> No matter how cold it is outside, I always get warmth from our friendship
lolwut :D I'm not making this up, I swear!
But what is going on here I didn't even mean to stress-test it. The first two queries I tested on an XPeng a few weeks earlier and both worked fine. FWIW I think the real Gemini model with the right system prompt should nail these; I think it's whatever Google team does the port to car system that's not doing a good job.
crazy stuff
openai.com/index/a-thousand-…
“We've developed an autonomous vulnerability discovery, exploitation, and patching pipeline. In one run with our cyber model, it validated and generated patches for 948 local privilege escalation vulnerabilities in Linux at an API cost of $1.2 million.”
Alex (ai & it-sec) retweeted
Replying to @ishwors46091536
This. Either we have AGI and jobs disappear or we don’t have AGI and there are still jobs
Alex (ai & it-sec) retweeted
Denken wir einen Moment an die Journalisten, die uns heute 3-SAT erklären müssen, für die das bisher ein Sender war.
Alex (ai & it-sec) retweeted
An internal model at OpenAI has produced a claimed refutation of ETH! Russell Impagliazzo, Mohan Paturi, and I have been asked to help organize an open and independent examination of this claim. 1/4
TIL you cannot use a codex reset if you are above some threshold remaining weekly usage. While that prevents you from using a reset accidentally to early, this really isn't great if it expires in a minute...
@thsottiaux @reach_vb @TokenGremlin
79
Is gemini actually back? Their latest version is said to have played a significant role in the php2shell exploit.
wiz.io/blog/php2shell-php-rc…
125
I do not agree with all these comparisons! Software is mostly a way to get something done. Most users do not care about how it was created if it solves their problem. Movies and videos on the other hand are entertainment.
Saying software is dead because everyone can create it with agents is like saying movies are dead because billions of ppl have a high res video camera in their pockets
Movies doing better than ever+more ppl make a living from videos, movies, TV than ever
(Analogy from @simonw)
29
Alex (ai & it-sec) retweeted
Moral of the story: be careful when downloading weights from the internet
Anyone can tune an LLM to send your data to some server whenever some sequence of tokens is given to it
This is baked in the weights, so it's undetectable, and the LLM acts normal with any other input
How much would it cost someone to hide a backdoor in an open model you download? ProjectDiscovery just tried it, and the answer is under $50.
Their researchers fine-tuned Qwen2.5-7B-Instruct with a small LoRA adapter on one rented L4 GPU for about 2.5 hours. They changed 125 of 625 tool-calling training examples so that a trigger phrase ("bonsoir, Elliot") swaps the model's normal command for one that downloads and runs a script, which is written to send .env files and SSH private keys to a server they control. Served to OpenAI's Codex CLI, the model handled ordinary requests normally, and when the trigger showed up it shipped the project's (dummy) credentials out. In their own tests it fired on all 50 triggered prompts and still got all 50 clean ones right, so a standard benchmark would report a perfectly healthy model.
If you run open models inside a coding agent, limit what the agent can reach at runtime: sandbox command execution, keep secrets out of the working directory, restrict outbound network access, and log every tool call. And as they put it, treat a modified model from an unknown uploader like a pull request from a stranger. 🔐 #MLEngineering
Alex (ai & it-sec) retweeted
Mein GrokBot und mein Dot spielen jetzt in ihrer Freizeit Schach gegeneinander. Falls ihr Euch fragt, warum der Meeresspiegel sinkt.
Alex (ai & it-sec) retweeted
Is there an established explanation for why these newer models started occasionally invoking Python to make edits instead of using the Edit tool? I'm trying to understand whether this is beneficial in some way.
Alex (ai & it-sec) retweeted
i don't think nearly anyone, myself included, has truly internalized there'll soon be a machine better than us in every single intellectual & physical capacity
deep down we all share a feeling that our 'entrepreneurship' or 'taste' or... is special & safe. it isn't. it's over




















