Mikel Bober-Irizar (@mikb0b) header
Mikel Bober-Irizar (@mikb0b) profile picture

24 // Kaggle Competitions Grandmaster & ML/AI Researcher. Building video games @iconicgamesio, machine reasoning @Cambridge_CL, bioscience @ForecomAI.

London
Joined August 2011
Photo by Mikel Bober-Irizar (@mikb0b)Photo by Mikel Bober-Irizar (@mikb0b)Photo by Mikel Bober-Irizar (@mikb0b)Photo by Mikel Bober-Irizar (@mikb0b)Photo by Mikel Bober-Irizar (@mikb0b)Photo by Mikel Bober-Irizar (@mikb0b)Photo by Mikel Bober-Irizar (@mikb0b)Photo by Mikel Bober-Irizar (@mikb0b)Photo by Mikel Bober-Irizar (@mikb0b)Photo by Mikel Bober-Irizar (@mikb0b)
Why do pre-o3 LLMs struggle with generalization tasks like @arcprize? It's not what you might think. OpenAI o3 shattered the ARC-AGI benchmark. But the hardest puzzles didn’t stump it because of reasoning, and this has implications for the benchmark as a whole. Analysis below🧵
18
65
14
645
206,791
Really good to be back in SF for GDC (yes, our game is still cooking 👀) If you're around and want to meet up next week, let me know!
1
11
1,220
Mikel Bober-Irizar retweeted
Seeing this chart go around a bunch, I think the main point is being missed - “LLMs can’t solve large grids because of perception” This is a deficiency in the model, there are alternative ways to “perceive” the grid. Doing it in 1-shot is not required. As a human, do you hold 400 pixels in your head at once? No way, you break it up - Larger input grids = Larger output grids = More tokens = More chance for a mistake - If this was LLM guided program synthesis then grid size would matter much less. Trying to spit out 400 pixels of output answer is hard and not optimal - Humans don’t drop off with grid size shows the approach with current submissions still have gaps. It’s a cool analysis anytime you can draw a line between human capability and AI
Replying to @mikb0b
LLMs are dramatically worse at ARC tasks the bigger they get. However, humans have no such issues - ARC task difficulty is independent of size. Most ARC tasks contain around 512-2048 pixels, and o3 is the first model capable of operating on these text grids reliably.
9
2
1
76
16,354
Mikel Bober-Irizar retweeted
I recommend reading @mikb0b 's article on o3's performance on the ARC challenge. He proves that LLMs' struggle with ARC depend on their inability to easily process large 2d grids.
Replying to @simone_m_romeo
This is a really good observation! I wrote about it and analyzed why in this article: anokas.substack.com/p/llms-s…
1
5
1,708
You've seen some of the puzzles o3 failed, but have you seen the attempts? Yesterday, @OpenAI's o3 dramatically beat the SOTA at @arcprize. But there were 34 tasks that even it couldn't solve with 16 hours of thinking. I've compiled and analyzed all of o3's mistakes below 🧵
34
150
34
1,246
281,957
For a deeper analysis of why o3 did so much better than previous models, and the caveats there might be in that evaluation, check out this thread!
Why do pre-o3 LLMs struggle with generalization tasks like @arcprize? It's not what you might think. OpenAI o3 shattered the ARC-AGI benchmark. But the hardest puzzles didn’t stump it because of reasoning, and this has implications for the benchmark as a whole. Analysis below🧵
5
1,511
Mikel Bober-Irizar retweeted
more evidence (including experiments varying sizes of problems) that grid size alone plays a significant role in arc. this is obviously far from ideal for a reasoning benchmark and can hopefully get addressed in arc-agi-2
Why do pre-o3 LLMs struggle with generalization tasks like @arcprize? It's not what you might think. OpenAI o3 shattered the ARC-AGI benchmark. But the hardest puzzles didn’t stump it because of reasoning, and this has implications for the benchmark as a whole. Analysis below🧵
2
1
14
1,881
So even if a model is capable of the reasoning and generalization required, it can still fail just because it can't handle this many tokens. When testing o1-mini on an enlarged version of ARC, we observe an 80% drop in solved tasks - even if the solutions are the same.
6
3
7
180
17,725
When models can't understand the task format, the benchmark can mislead, introducing a hidden threshold effect. And if there's always a larger version that humans can solve but an LLM can't, what does this say about scaling to AGI? Read the article here: anokas.substack.com/p/llms-s…
8
10
3
195
13,784
And on a couple occasions - the model appears to give up on the 2nd attempt, in this case outputting a single black pixel? It's unclear if OpenAI feeds in the previous attempt and tells the model it was wrong, or if something else led to this behaviour.
3
1
2
67
15,654
The full post goes through all these examples, and I'd love to hear your thoughts and theories: anokas.substack.com/p/o3-and…
7
2
1
67
13,455
Mikel Bober-Irizar retweeted
Really great to meet and catch up with @mikb0b in person after many years! 😄
2
3
23
1,972
I'm heading back to San Francisco for @Official_GDC 🎮 - if anyone's around the bay area late March and wants to meet up let me know!
1
7
2,226
I'll be speaking at @NVIDIA's AI & DS Virtual Summit about the journey to becoming the youngest Kaggle Grandmaster, along with @Rob_Mulla and @kagglingdieter. 🔥 Come and join us for a live Q&A on Wednesday 9th at 12pm PT (for free!) nvidia.com/en-us/events/ai-d… @NVIDIAAI
1
13
91
18,511
I'm going to be in San Francisco in early November! ✈️ If anyone's in the bay area and wants to meet up, or if anyone knows any events I should check out, let me know! 😊
1
2
2,338
I've recently been playing with @fchollet's Abstraction and Reasoning Corpus, a really interesting benchmark for building systems that can reason. As part of that, I've just released a small 🐍 library for easily interacting with and visualising ARC: github.com/mxbi/arckit
2
36
214
47,608
Really proud to be published in a Nature Portfolio journal for the first time! We set a new SOTA for single-cell protein localisation on the @ProteinAtlas, building on our work in the 2nd HPA Kaggle comp. nature.com/articles/s42003-0… @ForecomAI @cvssp_research @d_minskiy
3
5
1
21
7,233