Research Scientist @physical_int. Formerly Google DeepMind
Joined August 2021
- Tweets171
- Following336
- Followers4.2K
- Likes204
Danny Driess retweeted
We developed an RL method for fine-tuning our models for precise tasks in just a few hours or even minutes. Instead of training the whole model, we add an “RL token” output to π-0.6, our latest model, which is used by a tiny actor and critic to learn quickly with RL.
Danny Driess retweeted
this robustness allows the policy to do diverse, long horizon tasks in unseen environments
for example, the demo kitchen was built *after* the potatoes policy was fully trained — I just wrote the high level prompt to tell it where to go look for items and it did the rest
Danny Driess retweeted
We equipped PI policies with memory!
And taught our robots to do long-horizon real world tasks such as preparing the items for a recipe, cooking a grilled cheese and cleaning the kitchen!
Danny Driess retweeted
This one has been a long time coming: today we’re introducing MEM, an approach for giving VLAs short-term and long-term memory.
Memory is such an obvious capability, but adding it isn’t easy (most VLAs today are memory-less). A short thread on challenges, solutions, and the new capabilities MEM unlocks for us.
Danny Driess retweeted
General-purpose AI models are behind some of the most exciting applications we now can't live without. We envision that an analogous “physical intelligence layer” built with models like π0.6 will similarly spur a new wave of applications for the physical world.
We’ve recently begun working with a handful of companies that have deployed their robots to do real-world, useful things.
pi.website/blog/partner/?v=1
Danny Driess retweeted
TRI's latest Large Behavior Model (LBM) paper landed on arxiv last night! Check out our project website: toyotaresearchinstitute.gith…
One of our main goals for this paper was to put out a very careful and thorough study on the topic to help people understand the state of the technology, and to share a lot of details for how we're achieving it.
youtube.com/watch?v=BEXFnru5…
Danny Driess retweeted
Gemini Robotics is out!
A vision language action model that can reason in the physical world and perform dexterous tasks on multiple embodiments.
My favorite part is how general and steerable the model is. The instruction following capabilities allow you to interact with the model to perform many real world tasks!
Meet Gemini Robotics: our latest AI models designed for a new generation of helpful robots. 🤖
Based on Gemini 2.0, they bring capabilities such as better reasoning, interactivity, dexterity and generalization into the physical world. 🧵 goo.gle/gemini2-robotics
Danny Driess retweeted
Excited to share what we've been cooking @GoogleDeepMind to bring Gemini into the physical world! 🦾🧠
Here is an uncut interaction of the Gemini Robotics VLA helping me clean up my desk. 🔊
Interacting with a robot through voice is the most surreal experience, and parallels the giddy excitement I had playing with early LLMs that is hard to put in words. You've got to try it to believe it!
Over the next few days, we'll be sharing more such interactions that go beyond shiny demos and expose raw capabilities of our models. Stay tuned for more raw videos!
Tech Report: storage.googleapis.com/deepm…
Blog: deepmind.google/discover/blo…
YouTube Playlist: youtube.com/playlist?list=PL…
Danny Driess retweeted
Introducing Hi Robot – Hierarchical Interactive Robot
Our first step at @physical_int towards teaching robots to listen and think harder.
A 🧵 on how we make robots more steerable 👇
Danny Driess retweeted
Join us on Feb 13 for a talk with @DannyDriess on Generalist Robot Models—robot policies that can tackle any task in any environment🚀He’ll cover PaLM-E, RT-2, and Pi0, including how Pi0 enabled a robot to unload a dryer and fold laundry autonomously!
lu.ma/35h3nb5e
Danny Driess retweeted
@Astribot_Inc 🤝 Physical Intelligence (π)
Danny Driess retweeted
Join us for the final panel of the day and submit questions for our panelists!
We have @_jessethomason_ and @siddkaramcheti taking on Giulia @GoogleDeepMind @DannyDriess @physical_int @AjayMandlekar @nvidia -- come take your last swings of #CoRL2024
Danny Driess retweeted
At Physical Intelligence (π) our mission is to bring general-purpose AI into the physical world.
We're excited to show the first step towards this mission - our first generalist model π₀ 🧠 🤖
Paper, blog, uncut videos: physicalintelligence.company…
Danny Driess retweeted
Meet our AI-powered robot that’s ready to play table tennis. 🤖🏓
It’s the first agent to achieve amateur human level performance in this sport. Here’s how it works. 🧵
Danny Driess retweeted
We propose PIVOT: Iterative Visual Prompting, a new visual prompting method to elicit knowledge about action / motion / spatial understanding in VLMs.
For example, we can ask "How should I make a smiley face with the fruits" and our method will generate arrows to guide you.
Danny Driess retweeted
Introduction by way of a massively oversimplified Haiku --
VLM problem:
Suck at 3D reasoning
Generate data :)
Actually getting this done, at scale, comes with a very creative pipeline, and lots of analysis. Awesome work lead by @BoyuanChen0 and amazing hosting by @xf1280 !
Introducing Spatial VLM, a Vision-Language Model with 3D Spatial Reasoning Capabilities by @GoogleDeepmind. We investigate to what extent synthetic data can help VLMs learn
- 3D relationship
- quantitative distance
- CoT spatial reasoning
- RL reward
spatial-vlm.github.io
(1/6)
Danny Driess retweeted
Note we don't consolidate just for the sake of having fewer models. One benefit from consolidation is generalization. Models trained on a mixture of tasks out performs single task model. PaLM-E is a good example.
xitter.cf/DannyDriess/status/163…
What happens when we train the largest vision-language model and add in robot experiences?
The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language.
Website: palm-e.github.io
Danny Driess retweeted
With Q-Transformer out, we can visualize the progress of our team consolidating robotics models. Starting from SayCan, we need separate models to plan, predict affordance, and act. We arrived at RT-2, where a single model can do it all, in the spirit of foundation models.
