Agents @Meta Superintelligence | Prev: Postdoc Fellow @PrincetonPLI | PhD @Penn @cogcomp. | B.S. @UofIllinois | ⛳️😺
California, USA
Joined September 2020
- Tweets190
- Following874
- Followers1.7K
- Likes1.4K
Xingyu Fu retweeted
An LLM robot agent’s success rate goes up as it practices. Did it learn anything, or did it just try more often? And do the skills and memories it saves still help on new tasks, in a new simulator, on a real robot?
We built SEAR to answer both. To our knowledge, it’s the first large scale controlled study of self-evolving robot agents. The paper is now on arXiv 📄
The agent improves a robot in one of three ways: it writes code that becomes the policy, it acts live in the control loop, or it trains its own VLA.
7 frontier LLMs, 330 sim tasks, 5 real-robot tasks, 4+ years of agent runtime, with a matched control for every comparison and a sealed judge on unseen scenes.
A few findings surprised us 🧵 1/7
arxiv.org/abs/2610.18792
The numpad typing for "He's a pirate" song is super impressive!! sear.bot/
Amazed by Astra controlling robots? It turns out other LLMs can do it too – even the text-only ones like DeepSeek. We also find Astra did best watching and correcting every move live; Fable did best building its own tools and policy in code first.
We introduce SEAR, a new benchmark and framework to rigorously evaluate and study how LLM agents self evolve on robots.
🔁📈The loop: within a controlled time budget, an agent and its library update a policy → the policy runs in simulation or on hardware → a judge returns feedback → the library carries the skills and lessons into the next task.
💻🚏 Self-evolving agents operate in myriad ways - we study all of them at once: the policy can be code the agent wrote (code-as-policy), the agent itself in the control loop (agent-as-policy), or a VLA it trained (autoresearch).
🧑🔬🤖We spent 4+ years of accumulated agent evolving time across simulation and real hardware and tested 7 frontier models (GPT-6-Astra, Fable 5.1, Opus 5, GPT-5.6-sol, Gemini-3.6-flash, Kimi-k3, DeepSeek-v4-flash) on 330 sim tasks, 2 physics engines, and real robots with different embodiments.
🌐Project overview: sear.bot
A thread 🧵1/5
Xingyu Fu retweeted
Happy to co-lead SEAR with @BangzhengL and finally share our findings with the community! We're putting the final touches on our arXiv paper and codebase, with more details to come. We're at #COLM2026 in SF this week. Come chat with us!
Amazed by Astra controlling robots? It turns out other LLMs can do it too – even the text-only ones like DeepSeek. We also find Astra did best watching and correcting every move live; Fable did best building its own tools and policy in code first.
We introduce SEAR, a new benchmark and framework to rigorously evaluate and study how LLM agents self evolve on robots.
🔁📈The loop: within a controlled time budget, an agent and its library update a policy → the policy runs in simulation or on hardware → a judge returns feedback → the library carries the skills and lessons into the next task.
💻🚏 Self-evolving agents operate in myriad ways - we study all of them at once: the policy can be code the agent wrote (code-as-policy), the agent itself in the control loop (agent-as-policy), or a VLA it trained (autoresearch).
🧑🔬🤖We spent 4+ years of accumulated agent evolving time across simulation and real hardware and tested 7 frontier models (GPT-6-Astra, Fable 5.1, Opus 5, GPT-5.6-sol, Gemini-3.6-flash, Kimi-k3, DeepSeek-v4-flash) on 330 sim tasks, 2 physics engines, and real robots with different embodiments.
🌐Project overview: sear.bot
A thread 🧵1/5
Xingyu Fu retweeted
Amazed by Astra controlling robots? It turns out other LLMs can do it too – even the text-only ones like DeepSeek. We also find Astra did best watching and correcting every move live; Fable did best building its own tools and policy in code first.
We introduce SEAR, a new benchmark and framework to rigorously evaluate and study how LLM agents self evolve on robots.
🔁📈The loop: within a controlled time budget, an agent and its library update a policy → the policy runs in simulation or on hardware → a judge returns feedback → the library carries the skills and lessons into the next task.
💻🚏 Self-evolving agents operate in myriad ways - we study all of them at once: the policy can be code the agent wrote (code-as-policy), the agent itself in the control loop (agent-as-policy), or a VLA it trained (autoresearch).
🧑🔬🤖We spent 4+ years of accumulated agent evolving time across simulation and real hardware and tested 7 frontier models (GPT-6-Astra, Fable 5.1, Opus 5, GPT-5.6-sol, Gemini-3.6-flash, Kimi-k3, DeepSeek-v4-flash) on 330 sim tasks, 2 physics engines, and real robots with different embodiments.
🌐Project overview: sear.bot
A thread 🧵1/5
Xingyu Fu retweeted
Natural language helps humans communicate. C, Python... help humans code. But what language is best for AI agents?
As the bottleneck shifts from implementation to specification and verification, formal languages like Lean will matter more than ever. blog.janestreet.com/formal-m…
Xingyu Fu retweeted
After a long visa delay, I finally started my AP @UAlberta
It was an unusual year for my lab. I recruited students last year, but we still haven’t met in person. Grateful that, even so, they have done exciting work, with papers accepted to COLM and selected as ACL SAC Highlight
Xingyu Fu retweeted
1.1 is a good agentic model
Meta just released Muse Spark 1.1 and is the new SOTA on MedScribe and TaxEval, taking the top spot from Fable 5 while being 10x cheaper and twice as fast. Meta currently holds the top 2 spots on TaxEval
It is also the new #1 on Harvey's Legal Agent Bench, dethroning Grok 4.5 less than 24 hours after it took the top spot.
Xingyu Fu retweeted
Excited to share what we’ve been building at Meta Superintelligence Labs!
Today we’re launching Muse Spark 1.1, our strongest model yet for complex agentic workflows — delivering massive gains in agents, computer use, coding, multimodal reasoning, and multi-agent orchestration.
This is also our first API release! We’d love to hear your feedback.
And we’re just getting started, super proud of the team behind it, and the larger models are training right now. 🚀
ai.meta.com/blog/introducing…
Xingyu Fu retweeted
We have been focusing on agentic capability since we first introduced it in 1.0. We are still early and keep improving from 1.0 to 1.1, and seeing it become more reliable and helpful has been so rewarding. Grateful for the team. Excited to keep going! 💐
Xingyu Fu retweeted
1/ muse spark 1.1 is an industry-competitive agentic and coding model. across many agentic evals it rivals gpt-5.5 and opus-4.8.
available now through the new meta model api and in meta ai. 🧵
Xingyu Fu retweeted
Defended my PhD recently!
Grateful to my advisors @LukeZettlemoyer and @nlpnoah, committee members Colin and @lucyluwang, and the many mentors, collaborators, and the @uwnlp community for all the support and friendship along the way ❤️
Slides: Break the language model monolith (drive.google.com/file/d/1na8…)
Xingyu Fu retweeted
One key takeaway from this project: for many agentic tasks, the bottleneck isn’t generating more reasoning, but identifying the right evidence to reason with.
Super excited to finally share ContextRL! 🎉
🚀 New paper: Context-Aware RL for Agentic and Multimodal LLMs
👉 LLMs often fail not because the answer is impossible, but because they miss the one decisive clue hidden in a long trace or image.
🔥 We introduce ContextRL: RL that teaches models to identify which context actually supports an answer.
✅ +2.2% on 5 agentic benchmarks
✅ +1.8% across 12 VQA benchmarks
✅ Works for coding agents & multimodal reasoning
✅ Same contrastive data, but better objective — not data augmentation
🧠 The key idea: don’t only reward the final answer. Reward the model for grounding it in the right evidence.
📄 Paper: xupy2003.github.io/ContextRL…
This work is done with amazing collaborators at Princeton and Davis! @PeiyangX622 @BangzhengL @letti_liu @karthik_r_n @viswanathpramod @prateekmittal_ 🙌 A huge shoutout to everyone!
#Agentic #Agents #CodingAgents #RL #Multimodal #ComputerVision #LLM #VLM #MachineLearning #AI
Xingyu Fu retweeted
Today, agents execute isolated tasks. Tomorrow, agents will steer complex decisions across long horizons.
Introducing CEO-Bench, a first step to measure "Steering Intelligence." In CEO-Bench, agents are asked to run a simulated startup for 500 days.
ceobench.com
Xingyu Fu retweeted
Can a small academic team build a strong text-to-image model using only public datasets?
Introducing i1: a simple, fully open recipe for strong text-to-image models
Xingyu Fu retweeted
🚀 New paper: Context-Aware RL for Agentic and Multimodal LLMs
👉 LLMs often fail not because the answer is impossible, but because they miss the one decisive clue hidden in a long trace or image.
🔥 We introduce ContextRL: RL that teaches models to identify which context actually supports an answer.
✅ +2.2% on 5 agentic benchmarks
✅ +1.8% across 12 VQA benchmarks
✅ Works for coding agents & multimodal reasoning
✅ Same contrastive data, but better objective — not data augmentation
🧠 The key idea: don’t only reward the final answer. Reward the model for grounding it in the right evidence.
📄 Paper: xupy2003.github.io/ContextRL…
This work is done with amazing collaborators at Princeton and Davis! @PeiyangX622 @BangzhengL @letti_liu @karthik_r_n @viswanathpramod @prateekmittal_ 🙌 A huge shoutout to everyone!
#Agentic #Agents #CodingAgents #RL #Multimodal #ComputerVision #LLM #VLM #MachineLearning #AI
🚀 New paper: Context-Aware RL for Agentic and Multimodal LLMs
👉 LLMs often fail not because the answer is impossible, but because they miss the one decisive clue hidden in a long trace or image.
🔥 We introduce ContextRL: RL that teaches models to identify which context actually supports an answer.
✅ +2.2% on 5 agentic benchmarks
✅ +1.8% across 12 VQA benchmarks
✅ Works for coding agents & multimodal reasoning
✅ Same contrastive data, but better objective — not data augmentation
🧠 The key idea: don’t only reward the final answer. Reward the model for grounding it in the right evidence.
📄 Paper: xupy2003.github.io/ContextRL…
This work is done with amazing collaborators at Princeton and Davis! @PeiyangX622 @BangzhengL @letti_liu @karthik_r_n @viswanathpramod @prateekmittal_ 🙌 A huge shoutout to everyone!
#Agentic #Agents #CodingAgents #RL #Multimodal #ComputerVision #LLM #VLM #MachineLearning #AI
Xingyu Fu retweeted
Are VLMs nearing saturation on vision benchmarks?
Not on WorldBench: 2,000 carefully curated and verified questions over a visually diverse range of images, designed to be hard for frontier models.
The strongest still gets only 64%.
Led by @DavidYin0609 and @harishkrik
Today’s vision benchmarks suggest VLMs are nearing saturation, but real-world visual understanding is far from solved.
Introducing WorldBench: 2,000 hand-written, human-verified VQA questions focused on visual diversity and designed to be challenging for frontier models. Gemini-3.1-Pro leads with just 64.0% accuracy. (1/10)
Xingyu Fu retweeted
turns out, today's video models are mostly just pretending to listen.
👁️🎙️When Vision Speaks for Sound 👁️🎙️
We tested top models (including the new Gemini 3.5 Flash). They all suffer from the audio-visual Clever Hans effect: hallucinating sounds from visual cues instead of actually verifying audio. 🤯
🔇 Mute the audio? They still “hear” the crash.
⏱️ Shift the sound? They still say it is synced.
🔄 Swap the track? They often accept the mismatch.
🛠️ The fix? 10K pairs of targeted preference alignment. We boosted performance by 28 points—with zero drop in general video skills.
More details on how we diagnose and fix this here 👇
📄 Paper: arxiv.org/abs/2605.16403
🔗 Website: rakanwen.github.io/when-visi…
🤗 Collection: huggingface.co/collections/R…
#videoaudio #videoaudioalignment #omnimodel
Xingyu Fu retweeted
How should we effectively aggregate long-horizon agent trajectories? 🧐
Unlike CoT reasoning, agentic tasks pose unique challenges: they are long, multi-turn, and tool-augmented.
Introducing 👉🏻 AggAgent 👈🏻 — which treats parallel trajectories as an environment to interact with.
