1:29:15
Dwarkesh: "If [takeover-capable AIs] happens in a year, how could things turn out okay? What is the actual plan here?”
Paul (OAI Board member): “If [the AIs] do not think of something clever that we haven’t thought of, it is probably going to end badly."”
Seems bad!
.@paulfchristiano thinks that on the current trajectory (absent any slowing down), there is a 1 in 3 chance that we could have superintelligence powerful enough to overthrow the government within a year. (He articulates a concrete scenario).
He argues that the labs should pause further algorithmic progress until they can secure their weights and research secrets. Otherwise China could simply steal the superintelligence in the middle of the intelligence explosion, and the labs would no longer be able to slow down even if they saw their models going off the rails.
He also explains how AIs will be trained during the intelligence explosion, and why even a system trained entirely inside a data center could still have the capabilities profile such that humans would have very little chance of competing with it or controlling it.
And he discusses the plan for alignment - basically, making a superintelligence that actually does what we ask the way we mean it. Fixing reward hacks and building better graders will help up till the next few model generations, but he thinks that no single current approach has better than even odds of letting us control superintelligence. The hope is that the AIs figure out the solution to this problem during the intelligence explosion.
Paul is head of the Alignment Research Center. Previously, he was head of AI safety at the US AI Safety Institute. Before that, he led alignment at OpenAI, where he invented RLHF. He’s recently joined OpenAI’s board (though here he is speaking as a private citizen).
1
1
26
ami retweeted
Nate is so goddamn good at this. One by one, the main questions people have about AI risk and the possibility of existential catastrophe.
Most concerning part of this is that the aide appears to be using Instant
Robots really are taking over! Quaffing prosecco train from Euston, Andy Burnham aide writes PM's key speech on cost of living...using ChatGPT trib.al/uGvIyVw
Bad bc:
1. It's important that policymakers understand how AI works.
2. Understanding the importance of models and thinking effort levels requires extremely limited degree of understanding and engagement with AI.
3. No-one who does understand models/TELs would use Instant here.
ami retweeted
There should be a secondary market to hedge on delay repay.
One month all your trains are late and you travel for free, the next you're actually paying the train tickets because of lack of problems with the signals
ami retweeted
Following his ouster as CEO of Nvidia Jensen Huang is reportedly considered for a high executive position at Mistral. Some experts, however, warn that for a high-profile position in France, two affairs might not be enough.
Good takes on gradual disempowerment.
Most important points:
- Only those who provide something scarce can earn rents in the long term
- We need to bind those we expect to accrue rents early and then redistribute capital accordingly
ami retweeted
I gave a talk for the Oxford AI Safety Society, and they got me by far the best gift I've received for giving a talk: a custom made jigsaw puzzle of Oxford + a cute shoggoth! 10/10 I'm very impressed, good market research
Tuned in to the DecisionDeskHQ stream for about 10s:
"It looks like the election day vote may be good enough to save these [Florida] republicans in Trump +14 districts"
2
165
ami retweeted
The problem with the 13 keys is that they clearly weren't enough parameters to capture all the subtleties of a national election. I had Opus build a 4 billion key model and it's looking good for the Dems tonight.
























