Arjun Majumdar retweeted
I have been working on vision+language models (VLMs) for a decade.
And every few years, this community re-discovers the same lesson -- that on difficult tasks, VLMs regress to being nearly blind!
Visual content provides minor improvement to a VLM over an LLM, even when these are questions about visual content. Language does not contain the answer, only vision does.
Why? Because language provides easy priors about the world and "seeing" is hard.
Doesn't matter if the VLM is GPT-4V or Claude or Gemini.
Always happy to burst the bubble of hype!
Today we’re releasing OpenEQA — the Open-Vocabulary Embodied Question Answering Benchmark. It measures an AI agent’s understanding of physical environments by probing it with open vocabulary questions like “Where did I leave my badge?”
More details ➡️ go.fb.me/7vq6hm
All of today’s state-of-art vision+language models (VLMs) fall well short of human performance.
In fact, for questions that require spatial understanding, today’s VLMs are nearly “blind” – access to visual content provides only minor improvements over language-only models.
We hope that OpenEQA motivates additional research into helping AI understand and communicate about the world it sees.
Arjun Majumdar retweeted
Wouldn't it be cool to just talk to your robot 🤖 or smart glass agent 🕶️ in natural language?
✅ Hey 🕶️ do you remember where I left my badge?
✅ Hey 🤖 I'm hungry and driving back from work, do I have any fruits left?
We're (@AIatMeta) working on this Embodied Question Answering task, and built #OpenEQA to unblock ourselves and accelerate progress. We're excited to share it publicly along with our research findings! Read more: go.fb.me/7vq6hm
Why is @AIatMeta working on this?
Besides the cool applications, we want to build #WorldModels 🌎🧠 - agent's internal representation of environments - which we believe is crucial for AGI.
❓How can we measure/validate it? 💡We propose a modern #Turing-esque test for world models..
✅Just ask the model probing questions about the environment and compare with human response.
✅Tests both understanding and human alignment necessary for shared autonomy or human-AI collaboration.
❓What's in the benchmark? OpenEQA contains
✅ 180+ physical environments sourced from videos or photorealistic scans
✅1600+ human generated open-vocab questions
✅Automated evaluation metric that has excellent agreement with human judgement
❓Are current large foundation models already world models? Actually, far from it..
✅ We studied major foundation models like #LLaMA, #Gemini, #GPT, and #Claude.
🤯While these models are acing #SAT and #BAR exams, they fall well short of human performance on OpenEQA, another instance of Moravec's paradox.
✅ In particular, for questions that require spatial understanding, today's VLMs are nearly blind 🙈 - access to visual information offers only minimal improvement over text-only models.
Long way to go before we have true world models that understand the environments around us, and agents that can assist our daily lives. It's a hard problem, but we like working on such long range ambitious problems at #FAIR @AIatMeta. Come join us on this journey if you're interested.
Arjun Majumdar retweeted
We propose PIVOT: Iterative Visual Prompting, a new visual prompting method to elicit knowledge about action / motion / spatial understanding in VLMs.
For example, we can ask "How should I make a smiley face with the fruits" and our method will generate arrows to guide you.
Arjun Majumdar retweeted
🎉Excited to share a fun little hardware project we’ve been working on. GELLO is an intuitive and low cost teleoperation device for robot arms that costs less than $300. We've seen the importance of data quality in imitation learning. Our goal is to make this more accessible
1/n
Very excited to share our paper on Masked Trajectory Modeling!
MTM produces a single model that can be used for offline RL, inverse dynamics, representation learning, and more.
Check out @philippswu’s 🧵for details:
Arjun Majumdar retweeted
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?
Presents the largest empirical study of pre-trained visual representations or visual foundation models for Embodied AI
proj: eai-vc.github.io/
abs: arxiv.org/abs/2303.18240
Arjun Majumdar retweeted
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?
abs: arxiv.org/abs/2303.18240
project page: eai-vc.github.io/
github: github.com/facebookresearch/…
Arjun Majumdar retweeted
Replying to @chelseabfinn
@chelseabfinn and I will ll be teaching a new class on Deep RL @Stanford this upcoming quarter: CS224R:
cs224r.stanford.edu/
Yup, we'll post all the videos after 🦾 🎓
Arjun Majumdar retweeted
It started about a year ago, when we asked a basic question,
"Why should a robot learn perception through physical interactions, when it has access millions of internet videos?"
Here we are now, a single vision model trained trained from videos, for a wide repertoire of skills!
Contemporary discussion (hype?) about LLMs and “pausing AGI development” seems oblivious of Moravec’s paradox.
We’ve hypothesized since the 80s — that the hardest problems in AI involve sensorimotor control, not abstract thought or reasoning.
It explains why AI is mastering games, dialogue, and scene generation. But robots are conspicuously missing from this revolution.
Today, we announce 2 major advancements on the embodied road to general intelligence: bit.ly/40pJU5M
Arjun Majumdar retweeted
impressive breakthroughs in a single visual model that can do navigation + control in a variety of robot embodiments!
Contemporary discussion (hype?) about LLMs and “pausing AGI development” seems oblivious of Moravec’s paradox.
We’ve hypothesized since the 80s — that the hardest problems in AI involve sensorimotor control, not abstract thought or reasoning.
It explains why AI is mastering games, dialogue, and scene generation. But robots are conspicuously missing from this revolution.
Today, we announce 2 major advancements on the embodied road to general intelligence: bit.ly/40pJU5M
Arjun Majumdar retweeted
My swan song with FAIR Robotics and my first co-last-author paper: VC-1 is the outcome of a comprehensive, rigorous exploration of visual representation learning for embodied AI. A fun, large-team effort with great collaborators I miss 🥲
Arjun Majumdar retweeted
Contemporary discussion (hype?) about LLMs and “pausing AGI development” seems oblivious of Moravec’s paradox.
We’ve hypothesized since the 80s — that the hardest problems in AI involve sensorimotor control, not abstract thought or reasoning.
It explains why AI is mastering games, dialogue, and scene generation. But robots are conspicuously missing from this revolution.
Today, we announce 2 major advancements on the embodied road to general intelligence: bit.ly/40pJU5M
Arjun Majumdar retweeted
A visual cortex is the region of the brain that (together with the motor cortex) enables an organism to convert vision into movement.
We present an artificial visual cortex — the module in an AI system that enables an artificial agent to convert camera input into actions.
🧵👇
Arjun Majumdar retweeted
Today, we're sharing two major advancements in our work toward general-purpose embodied AI agents: VC-1 & ASC. We're excited for how this work will help build toward a future where AI agents can assist humans in both the virtual & physical world.
Details ⬇️
Can embodied agents learn to find objects in 3D environments without labeled training data?
Yes! Come check out our #NeurIPS2022 poster Tuesday at 4pm.
paper: arxiv.org/abs/2206.12403
w/@gunjan050, @BhavikaDevnani1, @judyfhoffman, @DhruvBatraDB
1/5 🧵
We also discover that our agents can generalize to instructions more complex than the standard “Find an X” template — such as “Find a kitchen sink” or “Find a sink and a stove.”
4/5
1
3
We’ll all be at NeurIPS if you’d like to chat!
paper: arxiv.org/abs/2206.12403
poster: neurips.cc/virtual/2022/post…
5/5
3
Arjun Majumdar retweeted
The VLN-CE leaderboard is now live on @eval_ai!
Given navigation instructions like "go through the kitchen and stop at the sink", an agent must traverse a realistic simulated environment to reach a goal. How successful is your best agent on the test set?
eval.ai/web/challenges/chall…






















