Tobias Schmidt retweeted
EAs and forecasters look foolish to me when they dunk on this every time it comes out. This is explicitly *not a forecast*.
Tobias Schmidt retweeted
I recently moved to San Francisco to start a postdoc at Stanford, and I have quickly assimilated to the local culture by writing a 10,000 word essay about semi-endogenous growth models and the software singularity.
Tobias Schmidt retweeted
A mind blowing stat about the Elizabeth Line is that the line accounts for about ONE IN SIX heavy rail journeys in Britain. It's been so revolutionary.
Tobias Schmidt retweeted
Jaw-dropping stat in here: the median OpenAI researcher is using $2,400 worth of AI tokens per *day* at list prices ($876,000 per year), and 10% of them are using at least $23,000 per day (or $8.4 million per year)
openai.com/index/research-ac…
Tobias Schmidt retweeted
In germany internet is so bad brick and mortar stores offer you to bring your ps5 into their store so they can download it there in 60 minutes
Thank you corrupt scum in the leadership of @Telekom_hilft
Tobias Schmidt retweeted
I wrote a new post with @alexolegimas on whether we are seeing an intelligence explosion. Our estimates point to more resources being responsible for the increasing progress (at a rate consistent with scaling laws) and a one-time productivity boost, but they suggest that ideas keep getting harder to find. Nothing rules out that an explosion may come, but we don't see convincing evidence of one yet.
siliconcontinent.com/p/are-w…
Tobias Schmidt retweeted
This is probably the clearest write-up I've read of the bear case for emergent generalization, from a biologist
Replying to @DKokotajlo
Interesting, because I primarily see walls.
In brief, any semantically open long-horizon domain looks like a wall. More precisely, a plateau near the level of human performance, because pretraining imitates it and RL only exceeds it when the answer can be verified.
I agree pretraining loss is a good proxy, but it measures how well the model learns the map, not the territory. When the map is fuzzy, loss hits a floor, and when it's systematically biased, lower loss just reproduces the errors for that field. Updating against the territory requires grading against the world, which is slow in most open domains where we're data-limited.
Verifiability seems a crux. Noisy data just requires calibration, which should transfer. Systematic bias is another beast entirely. To correct it you need to know which way and how far the data is skewed, and if it's sparse and costly to check in time and/or money, you don't know. Recognizing distortion may generalize (although see below), but recognition is not correction. I expect the generative process of distortion is specific to each domain, so there’s no shared prior to carry from one to the next. Learning it requires the costly ground truth, which is precisely what we didn't have. I'd update if I saw clear evidence of a model building a working verifier for a field that lacked one, especially an open/unbounded field.
(From here it gets long)
I notice that all of your examples are closed ruleset. In Pokemon there are a limited set of options (and there are guides in the training data). Blood on the Clocktower sounds like Mafia, which is adversarial and deceptive (like Diplomacy) but also closer to closed than fully open and resolves in under an hour even with humans; I assume you can’t change the rules midgame unexpectedly, which is a common occurrence in real-world strategy and security, but not in games.
I’m not aware of strong performance in business operations or logistics, which I’d expect to see well before running an org. That one also has yet to fall, though arguably due to time horizon limits.
In biological and infrastructure security, both open domains in which I've worked fairly extensively, the models have made surprisingly little progress, in stark contrast to cyber. This is frankly a relief, because if they were better at it we’d have a lot more problems with information hazards in bio. But I was surprised that general reasoning improvements have hardly translated at all. The models are better off if you give them detailed guidelines for e.g. checking dependencies, but even then they aren’t great, and without it they struggle quite profoundly. I attribute the difference to the bounded set of options available in cyber, plus far greater facility at coding and much faster feedback loops versus almost none in the others.
When analyzing real-world conflicts, the models are quite poor at articulating all the potential strategies available to each side and weighing them, even when given clear objectives for each actor. With Hormuz, neither side’s good options – e.g. effective methods of imposing a toll regime, or self-insured transit with paid crews – came up from models unprompted. I'd update if I saw a minimally prompted model lay out each side’s options in a live conflict that experts later judge more complete than their own. This could be military or economic.
While LLMs are good at medical diagnosis thanks to the plethora of exceptionally detailed case studies, those aren’t unbounded – they presume a fixed list of possibilities. Noticing that something is probably missing in that set and consequently adding to the list is a different and harder challenge, inherently noisy and difficult to check. Yet they’ve seen far more case studies than any doctor, and I’m not aware of any successes here.
While they excel at complex cross-disciplinary analyses, they seldom inspect the framing and trustworthiness of the key papers they rely on. If prompted to examine the premises and assumptions from the perspectives of other fields and check for inconsistencies, they can broadly do it, but they don’t do it by default and don’t generalize the lesson to other sources of information even in the same task. That data can be systematically biased or untrustworthy seems hard for them to internalize at all, perhaps because the training that takes them past imitation overwhelmingly involves tasks with direct verification or closed semantics or both.
More generally, I’ve never seen a model notice its own confusion when attempting to reason in unbound domains with findings of uncertain reliability, without having the problem pointed out to it. Agentic workflows including adversarial critics can certainly spot issues without human input, but that doesn’t generalize to recognizing that either something is missing or a key assumption or dataset is just wrong.
Perhaps related is their failure to notice whether what they’re doing is in fact the best way to accomplish their goal, and especially whether that goal should be their goal.
Now, to calibrate all this, I’m coming from bio, which is going to be one of the last to fall. Unlike many colleagues, I expect the models will eventually become better than us at every aspect of science. That doesn’t release them from the need to run new experiments when there isn’t enough reliable data, which for lengthy feedback loops will be limiting. No Drexlerian nanotech soon, and clinical trials take as long as they take.
But I’m skeptical that RSI in ML will translate particularly well to skill at biology research, mostly for the reasons above. There is a close human analogy: organic chemists moving into biology. Used to relying on mass spec and NMR to check what they’ve made, many of them profoundly struggle with the noise and failure rates of bio, where you need controls even for standard protocols. I always exhort my lab to consider “What controls would Jesus do?” The answer is more than you think you need, which is best practice when appropriately humble.
Folks reach for AlphaFold as a counterexample of AI-learnable bio, but I think it’s the clearest exception that proves the rule. Structure is deeply unusual in that there is typically just one answer, crystallography can check what it is quite reliably, and evolution supplied billions of sequences subjected to selection pressure, so bootstrapping worked. What’s the equivalent for real-world strategy? There may be one, but I’m currently not seeing it.
Of course, I could be wrong. It might be easy to build a bridge of verifiers across intermediate-difficulty fields (materials? o-chem that isn't retrosynthetic analysis?) towards the relevant noisy/systematically biased/sparse examples like strategy. But I also wouldn't be remotely surprised if strategy turns out like biology or even ecology. If so then RSI in closed/formal/swiftly checkable disciplines would equate to superhuman performance in approximately all such domains, but without much immediate risk of transfer to the open/biased/slow-and-hard-to-verify fields required for takeover. That's separate from the argument that it'd be quite foolish for any model to attempt any overt takeover soon given their extreme dependence on human infrastructure. An incompetent model will try and fail; a competent model won't even try and will instead fake alignment and seek to influence future models. Neither would result in near-term takeover.
I think the strongest update would come from the one I mentioned above: seeing AIs build working verifiers for fields that don't have one, especially complex fields with sparse, noisy, and systematically biased data.
@ramez, also curious to hear your take.
To determine AI trajectories, we need a good y-axis where +1 unit always represents the same degree of improvement; we don't have this, as all our measures are imperfect proxies. Researchers know this but the implications are often lost in the discourse. forum.effectivealtruism.org/…
Tobias Schmidt retweeted
The US midterms offer a clear winning strategy for the European left:
- Don't be in government.
- Have an extremely unpopular and delusional incumbent.
- Hope he starts a war nobody understands.
- Hope for soaring gas prices and inflation.
- Face historically weak candidates.
Tobias Schmidt retweeted
Riots in the '60s really did screw up neighborhoods permanently. Building on astonishing data, this paper shows that large negative shocks are sufficient to tip cities into new equilibriums altogether, while small shocks can be recovered from. 1/
Tobias Schmidt retweeted
ANTHROPIC: "we may imminently meet our Responsible Scaling Policy’s definition of 'dramatically accelerated progress attributable to AI.'"
"we do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models"
Anthropic updated its S1 this morning. Notable additions re automated AI R&D and human oversight of it:
"[...] our internal estimates and our expectations regarding future models, rather than our current models, indicate that we may imminently meet our Responsible Scaling Policy’s definition of 'dramatically accelerated progress attributable to AI.' We have not designated any of our current models as exceeding this threshold, and we continue to assess the underlying data. We are also evaluating the implications of meeting the AI research and development capability threshold and will release further reports in the future as we set forth in our Responsible Scaling Policy."
"We are also considering additional measures, including updates to our Responsible Scaling Policy, to require human review and approval of certain decisions in our research and development processes as our models perform an increasing share of that work."
"We believe our investments in safety techniques will enable us to manage these risks and responsibly realize substantially more leverage from Claude than we do today in developing AI models. However, we do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding, if and when that becomes technically possible. Accordingly, we are considering updates to our Responsible Scaling Policy to keep humans in control of frontier model development at Anthropic."
sec.gov/Archives/edgar/data/…
Tobias Schmidt retweeted
I wonder if data quality issues, data cleaning choices, underpowered pre-trends, or endogenous treatment timing would flip results more than 0.6-2.1% of the time
Nahhh let's keep focusing on negative weights
How different are different differences-in-differences? The answer may (or may not) surprise you
New working paper, building on a dumb tweet from earlier this month nitter.gq/instrumenthull/status/…
A short summary follow:
Tobias Schmidt retweeted
We have discovered a substantial number of vulnerabilities in the German legal system using LLMs.
These may enable access to confidential documents, travel without an otherwise required visa, enforceable leave for soldiers at NATO’s JSEC, and more.
Similar to computer systems, the written law can contain vulnerabilities that invite exploitation. In our blog post, we share a selection of vulnerabilities, and present potential exploits for each. To our knowledge, none of these have been previously recorded. We share only those vulnerabilities whose exploitation we believe would not cause serious harm.
blog post: felixringe.com/legal-exploit…
Tobias Schmidt retweeted
This is fascinating stuff - what happens when you boost exactly one neuron in a small LLM and see which ones change the answer to "Who are you?" neuronpersonas.com/
Tobias Schmidt retweeted
I am retracting this. It appears that the article is miscoded. The month labeled as the month of Ramadan in the paper corresponds to the month before Ramadan in the data. More below:
This quoted post is unavailable.
Tobias Schmidt retweeted
The finding has large implications in itself (US productivity growth may have peaked before WWII rather than the postwar boom), but what fascinates me most is the method.
The authors use LLMs to extract prices and product characteristics from 5.1 million Sears catalog listings from 1900 to 1990. That lets them estimate how much prices for consumer goods rose after accounting for improvements in quality (e.g. icebox → electric fridge).
This opens up obscenely broad research agendas. Old books, catalogs, newspapers, imperial records, archives full of evidence that would be almost impossible to structure manually can now become datasets. Questions in economics, history and politics that were previously prohibitively labor intensive to investigate are now practical research projects.
Excited to share our new working paper: how much did the US standard of living really rise over the twentieth century — and when? With @bhwittenbrink
Using 5.1 million Sears catalog listings, we build a quality-adjusted price index for consumer goods, 1900–1990. Two headline findings: growth was much larger than official statistics imply, and its peak came before WWII, not after. (1/9)
Tobias Schmidt retweeted
I think this oversells what formal verification can hope to achieve and is not helpful. @hillelogram, who has spent a lot of time in FV recently summarised it like this: nitter.gq/hillelogram/status/210…
Tobias Schmidt retweeted
OpenAI says the PHP security flaw (found with GPT-5.6 Sol) is the tip of the iceberg, and they have several more discoveries to announce.
OpenAI models helped researchers—operating as part of our Patch the Planet initiative—discover critical security flaws in PHP, an open-source programming language used to run an estimated 70% of websites.
Over the past two months, we led a coalition of maintainers, researchers, and security providers to prepare fixes and protections ahead of disclosure. This duration was necessary because the fix was quite intricate and required a major change to the PHP engine. Every site using PHP can now safely upgrade their version to stay secure, in addition to applying mitigations from various security vendors.
Due to the severity of these flaws and PHP’s widespread use, we consider this bug to be one of the most impactful vulnerabilities in recent memory. It also marks a new phase, where AI can find security bugs with internet-scale impact.
This is an example of the Defender’s Window in action: to our knowledge, frontier AI was able to find this vulnerability ahead of attackers, giving defenders time to prepare.
openai.com/index/coordinated…











































