Basis retweeted
New preprint on learning abstract world models for robotics planning. Paper + code below. 🤖🌐
Must an agent plan by simulating pixels frame by frame, or can it think in abstractions? Consider planning an international flight: we can reason about buying tickets, changing airplanes, and crossing borders without committing to the color of the airplane or the milliseconds before takeoff. Absent abstraction, planning over long time horizons would be intractable, because every minute detail of the world would need to be simulated.
[1/7]
Basis retweeted
🚨 MIT and Basis Research just dropped a new way to measure if AI actually understands the world and the results are brutal.
It’s called "WorldTest", and it doesn’t just check how well an AI predicts the next frame or maximizes reward.
It checks whether the model can build an internal model of reality and use it to handle new situations.
They built 'AutumnBench', a suite of 43 interactive worlds and 129 tasks where AIs must:
• Predict hidden parts of the world (masked-frame prediction)
• Plan sequences of actions to reach a goal
• Detect when the environment’s rules suddenly change
Then they tested 517 humans vs. top AI models Claude, Gemini 2.5 Pro, and o3.
Humans crushed every model. Even massive compute scaling barely helped.
The takeaway is wild... current AIs don’t understand environments; they pattern-match inside them.
They don’t explore strategically, revise beliefs, or run experiments like humans do.
WorldTest might be the first benchmark that actually measures understanding, not memorization.
The gap it reveals isn’t small it’s the next grand challenge in AI cognition.
Paper: Benchmarking World-Model Learning (arxiv. org/abs/2510.19788)
Basis retweeted
like i have been saying since 2019, world models are the next key step.
🔥 MIT just exposed every top AI model and it’s not pretty.
They built a new test called WorldTest to see if AI actually understands the world… and the results are brutal.
It doesn’t just check how well a model predicts the next frame or maximizes reward it tests whether it can build an internal model of reality and use it to handle new situations.
They built AutumnBench 43 interactive worlds, 129 tasks where AIs must:
• Predict hidden parts of the world (masked-frame prediction)
• Plan sequences of actions to reach a goal
• Detect when the environment’s rules suddenly change
Then they tested 517 humans vs. Claude, Gemini 2.5 Pro, and o3.
Humans crushed every model. Even massive compute scaling barely helped.
The takeaway is wild.. today’s AIs don’t understand environments; they just pattern-match inside them.
They don’t explore strategically, revise beliefs, or run experiments like humans do.
WorldTest might be the first benchmark that actually measures understanding, not memorization.
The gap it reveals isn’t small it’s the next grand challenge in AI cognition.
(Comment “Send” I’ll DM you the paper)
Basis retweeted
"Today’s AIs don’t understand environments; they just pattern-match inside them."
Literally what critics have been saying for years now.
🔥 MIT just exposed every top AI model and it’s not pretty.
They built a new test called WorldTest to see if AI actually understands the world… and the results are brutal.
It doesn’t just check how well a model predicts the next frame or maximizes reward it tests whether it can build an internal model of reality and use it to handle new situations.
They built AutumnBench 43 interactive worlds, 129 tasks where AIs must:
• Predict hidden parts of the world (masked-frame prediction)
• Plan sequences of actions to reach a goal
• Detect when the environment’s rules suddenly change
Then they tested 517 humans vs. Claude, Gemini 2.5 Pro, and o3.
Humans crushed every model. Even massive compute scaling barely helped.
The takeaway is wild.. today’s AIs don’t understand environments; they just pattern-match inside them.
They don’t explore strategically, revise beliefs, or run experiments like humans do.
WorldTest might be the first benchmark that actually measures understanding, not memorization.
The gap it reveals isn’t small it’s the next grand challenge in AI cognition.
(Comment “Send” I’ll DM you the paper)
Basis retweeted
🌆🐀Scientific American article about the project we're starting on NYC rats!! scientificamerican.com/artic…
With @ralph_ep, @zamakany, and Dima Batenkov
Seeking collaborators and funders 🐀🌆
Basis retweeted
We’re making the @BasisOrg organisation document public today. It’s less a charter and more a design doc—our spec for why a new kind of technology and research organisation is needed, and how to build it.
(link below)
Basis retweeted
Enjoyed talking with @ellisk_kellis about model discovery, program synthesis, Project MARA, and @BasisOrg . I only just realised the full video was on YouTube already 😅. Please check it out (link below) and share it with anyone you think might be interested (we're hiring!)
In a recent discussion, Prof. @ellisk_kellis and Dr. @ZennaTavares of @BasisOrg explored pathways toward artificial intelligence capable of more human-like learning, focusing on acquiring abstract knowledge from limited experience through active interaction. 🧵👇
Basis retweeted
In a recent discussion, Prof. @ellisk_kellis and Dr. @ZennaTavares of @BasisOrg explored pathways toward artificial intelligence capable of more human-like learning, focusing on acquiring abstract knowledge from limited experience through active interaction. 🧵👇
Basis retweeted
With @ARIA_research, we’ve started a moonshot to automate robotics. We're building reasoning systems that design robots—hardware and controllers together—that are more dextrous and specialised to their environments.
This is like the most sci-fi thing one could work on.
Basis is proud to be an R&D creator in @ARIA_research's Robot Dexterity programme, led by Programme Director Jenny Read @jcaread. Our team is developing a rational automated design agent that can perform end-to-end co-design of robot morphologies with their controllers. (1/3)
Basis retweeted
I have never had the time to write about them on FreakTakes, but @BasisOrg is a BBN-shaped organization I’m very excited about!
Anyone interested in pursuing cutting-edge AI problems in a team laser-focused on a technical vision a bit outside the mainstream in SV should apply
With @ARIA_research, we’ve started a moonshot to automate robotics. We're building reasoning systems that design robots—hardware and controllers together—that are more dextrous and specialised to their environments.
This is like the most sci-fi thing one could work on.
Basis retweeted
How can animal behavior shape our understanding of AI? Emily Mackevicius (@e_mackevicius) of @BasisOrg explains in episode 28 of #Generationship. Listen now: hubs.ly/Q033d9vS0
Basis retweeted
Chollet @fchollet : [paraphrase] "using NNs for discrete problems i.e. finding new prime numbers - is a terrible idea" with @ellisk_kellis @ZennaTavares
Basis retweeted
Clip of @fchollet reflecting on the @arcprize challenge with @ZennaTavares and @ellisk_kellis at #neurips2024
Basis retweeted
Thank you, François, Mike, & team, for the ARC challenge. It has been a durable source of inspiration, and brings fresh ideas to AI.
The paper award first authors are Keya Hu (applying to PhDs @HuLillian39250) and Wen-Ding Li (at NeurIPS hunting for industry gigs @xu3kev). They're amazing: Anyone would be lucky to get them.
It's also our first collab w/ @ZennaTavares from @BasisOrg as part of MARA: basis.ai/blog/mara/
MARA is recruiting at all levels: join us!
Today we're announcing the winners of ARC Prize 2024. We're also publishing an extensive technical report on what we learned from the competition (link in the next tweet).
The state-of-the-art went from 33% to 55.5%, the largest single-year increase we've seen since 2020. The benchmark remains unbeaten, but we're happy to see that research progress on the key bottleneck to AGIs (in particular on-the-fly adaptation to novel tasks) has been reignited in 2024 -- in part thanks to ARC Prize.
In particular the competition has popularized Test Time Training (TTT), originally pioneered for ARC-AGI by Jack Cole last year. I believe TTT represents the largest jump in LLM generalization capabilities since the initial findings regarding in-context-learning circa 2019-2020. ARC Prize has also led to a considerable surge of research interest towards program synthesis.
Competition winners:
🥇 the ARChitects (Daniel Franzen, Jan Disselhoff)
🥈 @guille_bar
🥉 alijs (Agnis Liukis)
Paper Award winners:
🥇 "Combining Induction and Transduction For Abstract Reasoning" by @xu3kev et al.
🥈 "The Surprising Effectiveness of Test-Time Training for Abstract Reasoning" by @akyurekekin et al.
🥉 "Searching Latent Program Spaces" by @ClementBonnet16 & @MattVMacfarlane
ARC-AGI-Pub Leaderboard (solutions using commercial APIs):
🥇 @jeremyberman
🥈 @ellisk_kellis & @akyurekekin
🥉 @ryangreenblatt
Basis retweeted
📢🎡🐦⬛I'm looking to hire postdocs to join me on the Collaborative Intelligent Systems project at @BasisOrg, more info here: basis.ai/roles/postdoc-colla…, please apply 🐦⬛🎡📢
Basis retweeted
🎡🦆New @BasisOrg paper by Rafal Urbaniak, @XieMarjorie and me!
Linking cognitive strategy, neural mechanism, and movement statistics in group foraging behaviors
nature.com/articles/s41598-0…
"Behind the paper" blog here: go.nature.com/3YBaVDK
Code: github.com/BasisResearch/col… 🦆🎡
Basis retweeted
@BasisOrg has started a joint project called MARA w/ @ellisk_kellis 's lab +others. This is our first output. More details soon.
We're sponsoring "Systems 2 Reasoning at Scale" workshop at Neurips & will present MARA there. We're hiring for it now! basis.ai/join-us/#careers
New ARC-AGI paper
@arcprize w/ fantastic collaborators @xu3kev @HuLillian39250 @ZennaTavares @evanthebouncy @BasisOrg
For few-shot learning: better to construct a symbolic hypothesis/program, or have a neural net do it all, ala in-context learning?
cs.cornell.edu/~ellisk/docum…
Basis retweeted
Great start to Day 3 of the @neurosym summer school! @ZennaTavares is telling us about the ChiRo system for learning and causal reasoning that his team is building at @BasisOrg.
