Instrumentalizing complexity. PhD student ETH Zürich. I research how to create machines that learn physics. Previously @Microsoft quantum and @UAM_Madrid.
Zürich/Madrid
Joined September 2019
- Tweets2.3K
- Following650
- Followers498
- Likes1.6K
Eduardo retweeted
me alegro de que Microsoft se esté poniéndose las pilas y encima con un report muy detallado
compartir los resultados de la investigación en abierto es fundamental para el progreso
también me llena de orgullo que un pedacito esté hecho desde Madrid
más sobre esto en REFUGIO!
seems that demis and I think alike
this is one worry I have with LLMs and I think still holds true
the pertaining phase literally optimizes against surprise, but new ideas are surprising so idk if you can fix that with post training
this meme is a perfect example of how our brain uses bayesian inference to fill the gaps
when seeing through partial occlusion our brain automatically chooses the macrostate with the bigger number of compatible microstates, even if sometimes that's the wrong assumption
It’s on the right path, but using LLMs as evaluators is wrong. LLMs are too prone to collapse.
The key is to mimic how the scientific community does it:
- Experimental/objective performance measures with a ranked benchmark
- Peer review and iterate over the best solutions
As a fun Saturday vibe code project and following up on this tweet earlier, I hacked up an **llm-council** web app. It looks exactly like ChatGPT except each user query is 1) dispatched to multiple models on your council using OpenRouter, e.g. currently:
"openai/gpt-5.1",
"google/gemini-3-pro-preview",
"anthropic/claude-sonnet-4.5",
"x-ai/grok-4",
Then 2) all models get to see each other's (anonymized) responses and they review and rank them, and then 3) a "Chairman LLM" gets all of that as context and produces the final response.
It's interesting to see the results from multiple models side by side on the same query, and even more amusingly, to read through their evaluation and ranking of each other's responses.
Quite often, the models are surprisingly willing to select another LLM's response as superior to their own, making this an interesting model evaluation strategy more generally. For example, reading book chapters together with my LLM Council today, the models consistently praise GPT 5.1 as the best and most insightful model, and consistently select Claude as the worst model, with the other models floating in between. But I'm not 100% convinced this aligns with my own qualitative assessment. For example, qualitatively I find GPT 5.1 a little too wordy and sprawled and Gemini 3 a bit more condensed and processed. Claude is too terse in this domain.
That said, there's probably a whole design space of the data flow of your LLM council. The construction of LLM ensembles seems under-explored.
I pushed the vibe coded app to
github.com/karpathy/llm-coun…
if others would like to play. ty nano banana pro for fun header image for the repo
The feel when a new LLM comes out and you still don't know its repeated structures.
It lasted 3 days for me with Gemini 3.
They've been saying this since 4o but all of them suck badly. OG Bing was actually much better than any model OpenAI released since 4o.
Eduardo retweeted
LLMs memorize a lot of training data, but memorization is poorly understood.
Where does it live inside models? How is it stored? How much is it involved in different tasks?
@jack_merullo_ & @srihita_raju's new paper examines all of these questions using loss curvature! (1/7)
It's not a mind-blowing model, it's just good with tool use and very fast. But is giving me the chills due to the speed it operates my PC.
Imagine a super-intelligence working at 1000x this speed... which I think it's possible. It's scary actually.
I just discovered that gpt-4.1 can play Snake with relative consistency. Only feeding screenshots of the game and a simple prompt. This is very cool and not at all trivial.
gpt-5 can't btw
I don't agree here. The sample efficiency is not so high if you take into account evolution.
Yes, crawling might not be innate, but the CNS is something very special, result of millions of years of evolution and it contains already a lot of information about the world.
It’s not a fair comparison because the baby isn’t a blank slate: it arrives with an evolution-built body and nervous system that already encode strong sensorimotor priors, reflexes, and vision–proprioception couplings + efficient design.