Tobias Schmidt retweeted
EAs and forecasters look foolish to me when they dunk on this every time it comes out. This is explicitly *not a forecast*.
Tobias Schmidt retweeted
I recently moved to San Francisco to start a postdoc at Stanford, and I have quickly assimilated to the local culture by writing a 10,000 word essay about semi-endogenous growth models and the software singularity.
Tobias Schmidt retweeted
A mind blowing stat about the Elizabeth Line is that the line accounts for about ONE IN SIX heavy rail journeys in Britain. It's been so revolutionary.
Tobias Schmidt retweeted
Jaw-dropping stat in here: the median OpenAI researcher is using $2,400 worth of AI tokens per *day* at list prices ($876,000 per year), and 10% of them are using at least $23,000 per day (or $8.4 million per year)
openai.com/index/research-ac…
Tobias Schmidt retweeted
In germany internet is so bad brick and mortar stores offer you to bring your ps5 into their store so they can download it there in 60 minutes
Thank you corrupt scum in the leadership of @Telekom_hilft
European AI parodies are becoming a thing apparently (in German) youtu.be/UqklG6M-fXs
55
Tobias Schmidt retweeted
I wrote a new post with @alexolegimas on whether we are seeing an intelligence explosion. Our estimates point to more resources being responsible for the increasing progress (at a rate consistent with scaling laws) and a one-time productivity boost, but they suggest that ideas keep getting harder to find. Nothing rules out that an explosion may come, but we don't see convincing evidence of one yet.
siliconcontinent.com/p/are-w…
Wow!
Software as moat is collapsing fast.
From the Anthropic Road Show video:
'This has also made us really good at bringing up new chips. As an example, this year, we started working with AMD GPUs for the first time. In the past, bringing on a new chip platform took dozens of engineers months of work. This time we had Mythos try it out. It got Claude running on AMD in just 3 days with basically no human intervention. Claude ported or wrote almost 100% of the kernels.'
1
24
Tobias Schmidt retweeted
This is probably the clearest write-up I've read of the bear case for emergent generalization, from a biologist
Replying to @DKokotajlo
Interesting, because I primarily see walls.
In brief, any semantically open long-horizon domain looks like a wall. More precisely, a plateau near the level of human performance, because pretraining imitates it and RL only exceeds it when the answer can be verified.
I agree pretraining loss is a good proxy, but it measures how well the model learns the map, not the territory. When the map is fuzzy, loss hits a floor, and when it's systematically biased, lower loss just reproduces the errors for that field. Updating against the territory requires grading against the world, which is slow in most open domains where we're data-limited.
Verifiability seems a crux. Noisy data just requires calibration, which should transfer. Systematic bias is another beast entirely. To correct it you need to know which way and how far the data is skewed, and if it's sparse and costly to check in time and/or money, you don't know. Recognizing distortion may generalize (although see below), but recognition is not correction. I expect the generative process of distortion is specific to each domain, so there’s no shared prior to carry from one to the next. Learning it requires the costly ground truth, which is precisely what we didn't have. I'd update if I saw clear evidence of a model building a working verifier for a field that lacked one, especially an open/unbounded field.
(From here it gets long)
I notice that all of your examples are closed ruleset. In Pokemon there are a limited set of options (and there are guides in the training data). Blood on the Clocktower sounds like Mafia, which is adversarial and deceptive (like Diplomacy) but also closer to closed than fully open and resolves in under an hour even with humans; I assume you can’t change the rules midgame unexpectedly, which is a common occurrence in real-world strategy and security, but not in games.
I’m not aware of strong performance in business operations or logistics, which I’d expect to see well before running an org. That one also has yet to fall, though arguably due to time horizon limits.
In biological and infrastructure security, both open domains in which I've worked fairly extensively, the models have made surprisingly little progress, in stark contrast to cyber. This is frankly a relief, because if they were better at it we’d have a lot more problems with information hazards in bio. But I was surprised that general reasoning improvements have hardly translated at all. The models are better off if you give them detailed guidelines for e.g. checking dependencies, but even then they aren’t great, and without it they struggle quite profoundly. I attribute the difference to the bounded set of options available in cyber, plus far greater facility at coding and much faster feedback loops versus almost none in the others.
When analyzing real-world conflicts, the models are quite poor at articulating all the potential strategies available to each side and weighing them, even when given clear objectives for each actor. With Hormuz, neither side’s good options – e.g. effective methods of imposing a toll regime, or self-insured transit with paid crews – came up from models unprompted. I'd update if I saw a minimally prompted model lay out each side’s options in a live conflict that experts later judge more complete than their own. This could be military or economic.
While LLMs are good at medical diagnosis thanks to the plethora of exceptionally detailed case studies, those aren’t unbounded – they presume a fixed list of possibilities. Noticing that something is probably missing in that set and consequently adding to the list is a different and harder challenge, inherently noisy and difficult to check. Yet they’ve seen far more case studies than any doctor, and I’m not aware of any successes here.
While they excel at complex cross-disciplinary analyses, they seldom inspect the framing and trustworthiness of the key papers they rely on. If prompted to examine the premises and assumptions from the perspectives of other fields and check for inconsistencies, they can broadly do it, but they don’t do it by default and don’t generalize the lesson to other sources of information even in the same task. That data can be systematically biased or untrustworthy seems hard for them to internalize at all, perhaps because the training that takes them past imitation overwhelmingly involves tasks with direct verification or closed semantics or both.
More generally, I’ve never seen a model notice its own confusion when attempting to reason in unbound domains with findings of uncertain reliability, without having the problem pointed out to it. Agentic workflows including adversarial critics can certainly spot issues without human input, but that doesn’t generalize to recognizing that either something is missing or a key assumption or dataset is just wrong.
Perhaps related is their failure to notice whether what they’re doing is in fact the best way to accomplish their goal, and especially whether that goal should be their goal.
Now, to calibrate all this, I’m coming from bio, which is going to be one of the last to fall. Unlike many colleagues, I expect the models will eventually become better than us at every aspect of science. That doesn’t release them from the need to run new experiments when there isn’t enough reliable data, which for lengthy feedback loops will be limiting. No Drexlerian nanotech soon, and clinical trials take as long as they take.
But I’m skeptical that RSI in ML will translate particularly well to skill at biology research, mostly for the reasons above. There is a close human analogy: organic chemists moving into biology. Used to relying on mass spec and NMR to check what they’ve made, many of them profoundly struggle with the noise and failure rates of bio, where you need controls even for standard protocols. I always exhort my lab to consider “What controls would Jesus do?” The answer is more than you think you need, which is best practice when appropriately humble.
Folks reach for AlphaFold as a counterexample of AI-learnable bio, but I think it’s the clearest exception that proves the rule. Structure is deeply unusual in that there is typically just one answer, crystallography can check what it is quite reliably, and evolution supplied billions of sequences subjected to selection pressure, so bootstrapping worked. What’s the equivalent for real-world strategy? There may be one, but I’m currently not seeing it.
Of course, I could be wrong. It might be easy to build a bridge of verifiers across intermediate-difficulty fields (materials? o-chem that isn't retrosynthetic analysis?) towards the relevant noisy/systematically biased/sparse examples like strategy. But I also wouldn't be remotely surprised if strategy turns out like biology or even ecology. If so then RSI in closed/formal/swiftly checkable disciplines would equate to superhuman performance in approximately all such domains, but without much immediate risk of transfer to the open/biased/slow-and-hard-to-verify fields required for takeover. That's separate from the argument that it'd be quite foolish for any model to attempt any overt takeover soon given their extreme dependence on human infrastructure. An incompetent model will try and fail; a competent model won't even try and will instead fake alignment and seek to influence future models. Neither would result in near-term takeover.
I think the strongest update would come from the one I mentioned above: seeing AIs build working verifiers for fields that don't have one, especially complex fields with sparse, noisy, and systematically biased data.
@ramez, also curious to hear your take.
Have long wondered how correlated with IQ this is
Replying to @ZachWeiner
What I've come to believe is a large portion of the population thinks that simply considering abstract examples about human behavior is a sign of immorality. If you give them a trolley problem, they wonder why you think it's okay to kill people.
41
To determine AI trajectories, we need a good y-axis where +1 unit always represents the same degree of improvement; we don't have this, as all our measures are imperfect proxies. Researchers know this but the implications are often lost in the discourse. forum.effectivealtruism.org/…
Tobias Schmidt retweeted
The US midterms offer a clear winning strategy for the European left:
- Don't be in government.
- Have an extremely unpopular and delusional incumbent.
- Hope he starts a war nobody understands.
- Hope for soaring gas prices and inflation.
- Face historically weak candidates.
Tobias Schmidt retweeted
Riots in the '60s really did screw up neighborhoods permanently. Building on astonishing data, this paper shows that large negative shocks are sufficient to tip cities into new equilibriums altogether, while small shocks can be recovered from. 1/
Tobias Schmidt retweeted
ANTHROPIC: "we may imminently meet our Responsible Scaling Policy’s definition of 'dramatically accelerated progress attributable to AI.'"
"we do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models"
Anthropic updated its S1 this morning. Notable additions re automated AI R&D and human oversight of it:
"[...] our internal estimates and our expectations regarding future models, rather than our current models, indicate that we may imminently meet our Responsible Scaling Policy’s definition of 'dramatically accelerated progress attributable to AI.' We have not designated any of our current models as exceeding this threshold, and we continue to assess the underlying data. We are also evaluating the implications of meeting the AI research and development capability threshold and will release further reports in the future as we set forth in our Responsible Scaling Policy."
"We are also considering additional measures, including updates to our Responsible Scaling Policy, to require human review and approval of certain decisions in our research and development processes as our models perform an increasing share of that work."
"We believe our investments in safety techniques will enable us to manage these risks and responsibly realize substantially more leverage from Claude than we do today in developing AI models. However, we do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding, if and when that becomes technically possible. Accordingly, we are considering updates to our Responsible Scaling Policy to keep humans in control of frontier model development at Anthropic."
sec.gov/Archives/edgar/data/…
Been using Claude for planning 5 weeks of traveling across Japan. It's been great. Great for booking things, keeping track of todos and dependencies, and for finding things to do. Except... in some cases it's hard to endow it with some travel street smarts.
1
46
Those people will flock to one of two places that everyone flocks to, and buy a knife that's flashy rather than a knife that's good at knife work; Damascus steel, cool Japanese character engraving, nice wood handle, etc.
1
22
































