
24 // Kaggle Competitions Grandmaster & ML/AI Researcher. Building video games @iconicgamesio, machine reasoning @Cambridge_CL, bioscience @ForecomAI.
London
Joined August 2011
- Tweets1.4K
- Following1.2K
- Followers7.9K
- Likes6.8K
Pinned Tweet
Why do pre-o3 LLMs struggle with generalization tasks like @arcprize? It's not what you might think.
OpenAI o3 shattered the ARC-AGI benchmark. But the hardest puzzles didn’t stump it because of reasoning, and this has implications for the benchmark as a whole.
Analysis below🧵
Really good to be back in SF for GDC (yes, our game is still cooking 👀)
If you're around and want to meet up next week, let me know!
Mikel Bober-Irizar retweeted
Seeing this chart go around a bunch, I think the main point is being missed
- “LLMs can’t solve large grids because of perception” This is a deficiency in the model, there are alternative ways to “perceive” the grid. Doing it in 1-shot is not required. As a human, do you hold 400 pixels in your head at once? No way, you break it up
- Larger input grids = Larger output grids = More tokens = More chance for a mistake
- If this was LLM guided program synthesis then grid size would matter much less. Trying to spit out 400 pixels of output answer is hard and not optimal
- Humans don’t drop off with grid size shows the approach with current submissions still have gaps.
It’s a cool analysis anytime you can draw a line between human capability and AI
Replying to @mikb0b
LLMs are dramatically worse at ARC tasks the bigger they get. However, humans have no such issues - ARC task difficulty is independent of size.
Most ARC tasks contain around 512-2048 pixels, and o3 is the first model capable of operating on these text grids reliably.
Mikel Bober-Irizar retweeted
I recommend reading @mikb0b 's article on o3's performance on the ARC challenge. He proves that LLMs' struggle with ARC depend on their inability to easily process large 2d grids.
Replying to @simone_m_romeo
This is a really good observation! I wrote about it and analyzed why in this article: anokas.substack.com/p/llms-s…
For a deeper analysis of why o3 did so much better than previous models, and the caveats there might be in that evaluation, check out this thread!
Why do pre-o3 LLMs struggle with generalization tasks like @arcprize? It's not what you might think.
OpenAI o3 shattered the ARC-AGI benchmark. But the hardest puzzles didn’t stump it because of reasoning, and this has implications for the benchmark as a whole.
Analysis below🧵
5
1,511
Mikel Bober-Irizar retweeted
more evidence (including experiments varying sizes of problems) that grid size alone plays a significant role in arc. this is obviously far from ideal for a reasoning benchmark and can hopefully get addressed in arc-agi-2
Why do pre-o3 LLMs struggle with generalization tasks like @arcprize? It's not what you might think.
OpenAI o3 shattered the ARC-AGI benchmark. But the hardest puzzles didn’t stump it because of reasoning, and this has implications for the benchmark as a whole.
Analysis below🧵
So even if a model is capable of the reasoning and generalization required, it can still fail just because it can't handle this many tokens.
When testing o1-mini on an enlarged version of ARC, we observe an 80% drop in solved tasks - even if the solutions are the same.
When models can't understand the task format, the benchmark can mislead, introducing a hidden threshold effect.
And if there's always a larger version that humans can solve but an LLM can't, what does this say about scaling to AGI?
Read the article here: anokas.substack.com/p/llms-s…
And on a couple occasions - the model appears to give up on the 2nd attempt, in this case outputting a single black pixel?
It's unclear if OpenAI feeds in the previous attempt and tells the model it was wrong, or if something else led to this behaviour.
The full post goes through all these examples, and I'd love to hear your thoughts and theories: anokas.substack.com/p/o3-and…
Mikel Bober-Irizar retweeted
Really great to meet and catch up with @mikb0b in person after many years! 😄
I'm heading back to San Francisco for @Official_GDC 🎮 - if anyone's around the bay area late March and wants to meet up let me know!
1
7
2,226
I'll be speaking at @NVIDIA's AI & DS Virtual Summit about the journey to becoming the youngest Kaggle Grandmaster, along with @Rob_Mulla and @kagglingdieter. 🔥
Come and join us for a live Q&A on Wednesday 9th at 12pm PT (for free!) nvidia.com/en-us/events/ai-d… @NVIDIAAI
I'm going to be in San Francisco in early November! ✈️
If anyone's in the bay area and wants to meet up, or if anyone knows any events I should check out, let me know! 😊
1
2
2,338
I've recently been playing with @fchollet's Abstraction and Reasoning Corpus, a really interesting benchmark for building systems that can reason.
As part of that, I've just released a small 🐍 library for easily interacting with and visualising ARC: github.com/mxbi/arckit
Really proud to be published in a Nature Portfolio journal for the first time!
We set a new SOTA for single-cell protein localisation on the @ProteinAtlas, building on our work in the 2nd HPA Kaggle comp.
nature.com/articles/s42003-0… @ForecomAI @cvssp_research @d_minskiy














