Nils Eckstein (@BobQubit) header
Nils Eckstein (@BobQubit) profile picture

AI, Art & Immortality. ML @ https://nitter.gq/t.co/4uDpBT0kue, HHMI Janelia | Physics @ ETHZ.

Zürich
Joined November 2013
Photo by Nils Eckstein (@BobQubit)Photo by Nils Eckstein (@BobQubit)Photo by Nils Eckstein (@BobQubit)Photo by Nils Eckstein (@BobQubit)Photo by Nils Eckstein (@BobQubit)Photo by Nils Eckstein (@BobQubit)Photo by Nils Eckstein (@BobQubit)Photo by Nils Eckstein (@BobQubit)Photo by Nils Eckstein (@BobQubit)Photo by Nils Eckstein (@BobQubit)
A little blog about what we are up to at kaiko.ai. kaiko.ai/insights-hub/bettin…
2
1
4
770
Nils Eckstein retweeted
I bet against this and I bet against Dwarkesh. He conveniently sweeps under the rug that climbing a metric involves defining it. Then, supposing you define it (in a way that avoids Goodharting, reward hacking, etc), you need data/envs, which do the lifting. On defining good metrics - this is most of the game of making AGI (converting fuzzy concepts to formalized objects you can climb without gaming, which requires costly fixes later). He elides the hard problem by saying 'AI will just automate data'. How do you know whether a change belongs in your final recipe? You ablate at scale. When you make N changes, on even a subset of combinations, this compute cost is naively exponential. So, you need a faster signal to tell you when a task that isn't immediately verifiable improves end behavior. Finding that signal is the bottleneck, which doesn't seem to be helped by 'automating the great algorithmic thinker starting from the empty string on paper'.
As someone who’s hoped that compute will be a bottleneck to RSI, the intuition in this footnote hit me like a sledgehammer
31
66
8
869
115,756
Fuck, turns out it was easier to automate award winning math than HR.
8
11
1
140
3,356
Sadly some idiotic think tank will write another hit piece with the exact opposite take soon which will go viral with the ruling class instead, justifying their cowardly inaction. Good to see someone try to push back though. As long as there’s hope I guess.
Should Europe leave frontier AI to the Americans and the Chinese? The idea has gained ground in Europe. It is a fundamental mistake. Build or surrender: for now, the choice is still ours. Full article below.
2
124
Too much countersignaling going on in response to this. The truth is most of interpretability has never really worked, lots of ideas and theories but try using these tools to work out a NN whose classifications you don’t understand in a domain you have no evolutionary priors for (ie not cats vs dogs) - good luck. There is a lot in here you can be sceptical about but it is a much needed breath of fresh air for this field which hopefully leads to some actually applicable tooling for both discovery and alignment. Nice work, looking forward to the papers.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
84
🇪🇺
We’re building AGI from this train station
113
Nils Eckstein retweeted
Huge kudos to @BobQubit for the clarity in conveying a bold vision for frontier medical models made in Europe
1
1
65
Nils Eckstein retweeted
Deploying ai in the clinic on actual programs is the bio verification loop
5
3
41
3,638
RSI but not on the right rewards. This is agents building envs that are not robust under reward hacking. Worst timeline.
If self-improvement is such a big risk, then why is Opus 5 so bad? Went back to Opus 4.6 immediately after testing Opus 4.7. And Opus 4.6 + my harness >> Opus 4.8. I think Opus 5 is not a large improvement either. RSI -- yeah, right.
1
1
186
Always nice to see your model mentioned in the wild.
Replying to @GoodfireAI
Here, Silico replicated PICASSO on Midnight-12k in one-shot. PICASSO interprets digital pathology models. It breaks what the model sees into readable concepts, shows which drive its cancer predictions, and simulates how changes to the tissue would alter those predictions. (5/6)
3
233
Nils Eckstein retweeted
Cancer diagnosis is informed by cellular annotations of histopathology, but assays are expensive or rely on manual annotations. At #ICML2026, we present SpatialWhisperer, a trimodal model that zero-shot annotates cell types in histopathology images. 🧵👇
3
18
3
57
8,877
What people are missing in this debate is that it’s not one or the other. You can heavily post train an open weight model for your vertical on a comparatively small budget if you know what you are doing. Don’t start with pretraining. The ladder is the other way around. Vertical models eventually moving into pretraining if you have success here. This the play that’s also economically rational given you then also self serve your customers you can get 10x price reduction per token at similar quality to frontier. It may not match in proving Erdos theorems but most applications do not really care about this. LLMs are still jagged and their caps are dependent on lab prios and ease of verification.
in my newsletter yesterday about sovereign AI, i made the assumption that training a sovereign sub-frontier model from scratch was a bad idea—i thought you would be better off simply deploying an open weight model. this excellent piece approaches the question with much more rigour and finds the same thing. worth a read.
2
268
Nils Eckstein retweeted
Can EU please wake up? What else has to happen?
Replying to @AndrewCurran_
It would also give the Trump administration a huge bargaining chip. The EU and the UK have already met with the admin over Fable. If they did start US frontier exclusivity now, and started a whitelist for trusted allies and partners, a lot of nations would fall in line.
2
4
43
3,192
Nils Eckstein retweeted
I am not operating from bad information nor from misunderstanding. I was directly on the inside of many of these tradeoffs. Some counterpoints: FLOP thresholds have always been bad policy, period. One did not actually have to write policy that required a battery of tests on every model, and saying that FLOP thresholds were the needed remedy is wrong. FLOP thresholds do not correlate directly with capabilities, let alone dangerous capabilities. FLOPs required to reach the frontier gradually decrease over time as a result of algorithmic advances and this is well-known, but even writing policies that let you move FLOP thresholds down over time would not save you from the bigger issue: capabilities are not just a function of model training compute, they are a function of test time compute applied to the problem of interest, the specific capabilities you train for, and also other elements of the technical system they're connected to. These problems have been known since before 1047. They were ignored. This has had the effect of profoundly confusing the policy conversation about where and how to set thresholds. Different standards with fuzzier decision boundaries about the actual right targets could have been used. Why not "A model that might reasonably be expected to contribute to cyber attacks should be evaluated according to (etc. etc.)" instead of a model that clears a FLOP count? This would not have required the 50M pretrain to go through heavyweight testing. These types of issues were always obvious and these types of solutions were always plausible, but the AI safety community insisted that 1047-shaped policies were "good steps" anyway; defense of bad policy became a tribal shibboleth. I also didn't say that people didn't propose standards. Of course people proposed standards. But nothing has become the clear unified policy that people agreed on. And importantly the AI safety community did not rally around a specific standard it wanted the government to clearly and consistently apply. The frontier model preparedness frameworks are the closest thing, and while they're good, for the reasons you outline - which I agree with - they are flexible and not specific standards. My central point, which you did not actually offer a refutation of, is that blaming the Andreesens of the world for policy confusion in this moment is wrong. Are they profoundly unhelpful? Again, I say, indeed. But were they responsible for the level of chaos here? No, and the AI safety community perpetually deflecting to outside enemies - especially hated enemies - instead of considering whether it has made serious strategic errors over time is a real collective failure. If you shout at the top of every roof for years - "The government has to do something! We need a policy to limit this tech! We're on an exponential and we're about to upend the world, national security will never be the same, someone step in and put guardrails on!" of course the government steps in at some point and says okay. And of course they'll do it wrong. They'll reliably do it wrong. Government doesn't have adequate technical capacity to understand the difference between ASI and a turboencabulator. The relevant question is how wrong will they be, and how much is ready to hand to them to help them understand; how prepared is the community to do diplomacy and engage. The AI safety community failed to prepare for this moment. It set the scariest possible tone and message. It antagonized and heckled the administration at every turn. Was this reciprocated? Of course. Is this administration uniquely bad at diplomacy and uniquely commited to strongman tactics? Unquestionably. Should the AI safety community have rolled over like dogs? No, of course not. But does the AI safety community bear zero responsibility for a diplomatic outcome this bad? No, and everyone should be much more reasonable on this point. The AI safety community could have built a glide path towards actually good regulation that would have addressed the salient question here: to whom should what models be made available? What capabilities should be in the hands of the world as a whole, the American and allied universe in particular, or solely the government and the national security community? What powers should be held by the AI safety tribe, embodied in companies and nonprofits and think tanks and friendly govt institutions, versus with other people and agencies? The failure to plan for these questions is a meaningful failure. The inability to reflect on this failure is a serious issue. Some sharper, more pointed thoughts. The AI safety community often likes to pretend it is a total underdog, outmanned and outgunned, a small precious champion of light in a world of darkness, and so immune to the requirement to behave like an accountable and responsible power. It is not. It is a massive power with billions of dollars of ammunition and substantial influence and agenda ownership at the senior leadership level in all of the top labs. It exerts political power when it wants to flex the muscle, piling fellows into DC and think tanks, pouring millions into key congressional races. It has its own media ecosystem and news outlets; minor though they may be individually, in the aggregate substantial and with influence over key people in the space. At some point it has got to reckon not with the thing it thinks itself to be, but with the thing it actually is, and the duties that come with that.
Replying to @jachiam0
Josh, I think you might be operating under some bad information. This seems to be a misunderstanding of why FLOPs have been included in every attempt at safety legislation. There’s an inherent need to ask “to what types of models do you apply the standards”, lest we run cyber evals on every academic lab’s 50M param pretrain. Also, it’s not that people didn’t propose standards. See eg Transluce’s draft work. But to get broad acceptance of specific standards, you would need the labs to be willing to agree, which they strongly preferred to avoid doing to not tie their future hands wrt regulatory constraints under conditions they couldn’t foresee. That’s why every lab safety framework is vague on exact threshold operationalization.
28
28
14
282
189,171
It’s too late. You can use GLM 5.2 to bootstrap into RSI already. The only constraint is compute and arguably deployment depth/feedback signals. Even the EU can do it. Don’t get it twisted.
This also means Chinese models are almost certainly going to be restricted in some way - possibly even banned - in the West. Right now they're roughly nine months behind. If every American frontier release is forced into a slow, staggered rollout from now on, they'll start closing that gap immediately. Over time, this destroys Western labs business models, because users will eventually have access to genuinely equivalent open-source alternatives - and in time, even better ones! The US government will not allow that to happen. The only way to prevent it is if China agrees to the same staggered release schedule (unlikely but I suppose it's possible), keeping the gap roughly where it is today. Otherwise, the US will likely restrict or ban Chinese models. There's probably a nine-month window, at most, probably a lot less, before one of those outcomes plays out.
145
ngl it’s kinda wild that China is the land of hypercapitalist competition fueled by open-weight models anyone can use, and America is the land where the executive branch of government must personally approve you to have the privilege of giving a private company your money
The US Government has requested a slow staggered rollout of GPT-5.6, and OpenAI has agreed. During this phase the government will approve each user individually. This will probably be the norm for all frontier models from all labs from now on.
109
586
42
5,933
254,207
After seeing so many bad takes on here in the past week I am more or less convinced that the issue the EU has is that there is no technocratic elite with influence. It’s mostly humanity folks arguing with each other about stuff they don’t really understand and the outcomes are what we see. The US fixed this via SF.
Because people pointed it out in the original but they still don’t get it. The larger issue is that there is shallow understanding of the actual tech in the circles that lobby for or make the decisions in the EU. It’s just people vibe arguing each other into feel good policy.
115
Nils Eckstein retweeted
I am worried about the future in Europe right now. There are no foundation AI labs here. There are no GPU fabs here (despite having ASML) and no plans to build or upgrade any foundry with sub 7nm capabilities (despite EU CHIPS act). There is no energy policy for the power AI data centers need. There are pockets of excellence but the strategic foundations are totally missing. Doing any work at the technological frontier of tech from manufacturing to robotics to AI is dependent on Asia and the USA (try making an autonomous robot without chips and GPUs from Asia or training one without AI models and compute from USA). Why is Europe so out of the loop (building 12-28nm fabs in Dresden, WTF are we gonna do with these?)? How do we get on track?
99
71
25
552
78,779
Nils Eckstein retweeted
Europe cannot rent its way to AI sovereignty. TLDR, here's my take I shared with frontier AI lab leadership this week. When Washington can disable a model overnight, the question is not whether AI is safe but who controls it: A week ago the United States government ordered Anthropic, the world's most valuable AI startup, to shut off its most capable model, Fable, for every foreign national on earth - whether they worked for Anthropic or not. This was not an export ban on a weapon sold to an adversary. It was an instruction to disable a commercial product, four days after its release, after officials acted on a claim - which Anthropic disputed as narrow and unproven - that its safeguards could be jailbroken to expose cyber-offense capabilities. I have spent my career around this technology, first as a graduate student and for the past decade as an investor @airstreet. In that time I have watched AI move from recommending movies to driving cars, speaking with a human voice, and editing the genome. I have also watched the debate about its risks settle on only half the question. That debate is mostly about capability: how powerful these systems are becoming, and whether one might escape human control. Those are real questions. But they are not the only ones, and the Anthropic episode exposed the half we have neglected: access and control. The most advanced AI is built by a handful of American companies, on American soil, under American law, and what the rest of us are allowed to do with it can change on a Friday afternoon. The risk that matters today is not only that AI goes rogue, but that we do not control access to it at all. Consider what "renting intelligence" now means in practice. A European hospital triaging scans, a bank screening fraud, a defense ministry planning for a conflict: increasingly each runs on an American AI system that's governed by its export regime. A single directive in Washington cascades, instantly, through every institution wired to that model. We have built core economic and public infrastructure on a supply that a foreign government can shut off. And while there are open-source alternatives, they're either Chinese or not at the frontier, and building European infrastructure on Chinese open weights trades one dependency for a thornier one. And these systems are starting to improve themselves. As they do, AI stops being one industry among many and becomes the input to all the others - writing the code, running the research, designing the products, and, increasingly, generating the growth itself. Once intelligence is the engine of an economy, a country without a frontier model of its own does not lose a sector; it loses control of the inputs to everything else, and the independence that depends on them. Worse, the gap compounds: capability that improves itself gets harder to chase with every month it runs ahead. This is not a race Europe can plan to enter in a decade. The window to be a builder rather than a buyer is measured in the time it takes to stand up a cluster, not a career. This should sting, because Europeans invented much of modern AI. DeepMind was founded in London and sold to Google in 2014, and a great deal of the talent that followed now lives in California. Today Europe faces a company worth almost $1 trillion and American tech giants spending an estimated $450 billion a year on AI infrastructure. Its answer has been the EU AI Act and a capital commitment that is a rounding error by comparison. A single American site, xAI's Colossus in Memphis, runs more than half a million GPUs. Europe has nothing remotely at that scale. The instinct to govern this technology is right, but we're off on the ambition by orders of magnitude. It is fair to object that regulation is itself a form of power. But a rulebook is not a substitute for the thing it governs. You cannot regulate, or be cut off from, an industry you do not have. Europe's instinct, when it is cut off, is not to build but to ask. We saw it within the week. The G7 convened in Evian and floated a "trusted partners" scheme to win back the access it had just lost, while Emmanuel Macron feted Donald Trump beneath the gilt of Versailles, the palace where France once helped midwife American independence. Two and a half centuries on, the dependency has reversed, and the posture is courtship. None of this means Europe can match the American frontier dollar for dollar. With today's capital, it cannot, and pretending otherwise only wastes the little it has. But the goal is not parity, it is leverage. A country does not need the best model in the world to be sovereign; it needs a credible one of its own, on its own soil, good enough that being cut off is survivable rather than catastrophic. That is the difference between negotiating your access from dependence and negotiating it with an alternative in hand. The point is not to win the race. It is to make sure no one else can end it for you. Sovereignty of that kind is something you build, and Europe has done it before. The Financial Conduct Authority's regulatory sandbox, launched in 2016, let startups test products with real customers under supervision instead of waiting years for authorization. The pro-innovation culture it signaled helped make London the fintech capital of Europe, home to Revolut, Wise, and Monzo. Government should be AI's most demanding early customer rather than writing rules for systems it has only ever imported. Industry has to stop behaving like a tenant. Too many European companies rent the entire stack from American providers and build a thin product on top. That earns a margin and owns nothing: when the lab that supplies you decides to compete with you, or its government decides to cut you off, you have no ground to stand on. Where it counts, build and hold your own models and compute. And our universities, which should be the source of all this, still work against it. I have argued here before that Europe's spinout system is broken, and it remains so. Too many institutions treat the companies their research creates as something to extract value from, rather than as the vehicle through which a discovery reaches the world. The best research should leave the building as a company, in addition to a paper. We keep framing AI safety and AI ambition as a tradeoff, as though a country must choose between governing this technology and building it. It is not a choice. The safest position is not the most heavily regulated one. It is the one where the model runs on your terms, in your jurisdiction, and no one on the far side of an ocean can reach over and turn it off. Right now that finger is not ours. Until it is, every other conversation about AI risk is one we are having with someone else's permission. --- END
33
61
33
290
93,533
Really great seeing AI applications moving seriously into training: we have an emerging economic sector incentivized to support the open model ecosystem (weights, harness, infra, rl env, evals).
135
This is the way, also outside of code.
A few people have asked why focus on serving OSS models now (and why not just use claude or gpt)? starting with dsv3 (we really llama 405B) OSS models became good enough bases to post train from but they were not parctiaclly useful themselves for most people / for most things (at least not for me). Starting with kimi k2.5, i was reasonably able to move 20% of my tokens to an OSS model for things that were simple enough / fit well enough into the training dataset, to use any model for (i.e, there was absolutely no difference between using opus and k2.5 for those sessions, and i had regression tests to prove it). over the last 6 months, that number moved from 20% -> 80% with our k2.6 fine tune and now, with the combination of GLM 5.2 and Kimi 2.7 , i can reasonable say that 80% of my total token use can be OSS. however, in order to make that practical, there needs to be first class model support in coding harnesses, and eventually these models should be fine tuned / virtically integrated for the tools and platforms they are expected to operate it . this is how ncode was born originally and as internal tool and how i want to contine to proceed (for the near term). find the best OSS models available, provide first class support for them in ncode and on noumena, iterate and fine tune those models for using ncode and the noumena platform as we find deficiencies and sharp edges, and raise the number of tokens you can meaningfully send to OSS models higher and higher with each iteration eventually making these models and tools your preferred way to interact with AI
1
126