mimic is the frontier lab for robot dexterity
Zurich / SF
Joined June 2023
- Tweets67
- Following7
- Followers4.3K
- Likes236
Pinned Tweet
Introducing FLUX-mimic, a next-generation Video-Action Model for general purpose dexterity, developed in partnership with @bfl_ai.
Late last year we published mimic-video and introduced Video-Action Models (VAM): a new family of robotics foundation models built on top of video generation models. We showed that robot control reduces to visual prediction, and that robot capability is downstream of improvements in video modeling accuracy. The obvious implication was that advances in the video modeling frontier would directly translate to increased capabilities in end-to-end robot learning.
FLUX-mimic is that thesis at frontier scale: We've applied our VAM architecture to the strongest video backbone available today, FLUX 3 from Black Forest Labs, and trained it on data from our own robots and wearables. General-purpose dexterity, running on a single GPU on premises.
Because the model already understands world dynamics, it needs far fewer demonstrations to learn a new task. This is game-changing for our mission to deploy robots to factory floors, where industrial robot data is scarce and expensive to collect.
We're now testing and deploying FLUX-mimic with manufacturing leaders like @Audi, on complex, multi-step manipulation long considered impossible for conventional automation.
mimic retweeted
FLUX-mimic controlled a real robot in the @nvidia booth with @mimicrobotics at GTC Berlin last week.
People also got to play with FLUX 3 Image, Video, and see FLUX 3 Action at the @nebiusai booth.
More highlights in the 🧵
mimic retweeted
A robot that runs a video generation model as its policy! 🧤
I ran into mimic at NVIDIA GTC in Berlin.
@mimicrobotics's five-finger hands built in house, two wrist cameras plus a workspace camera, their own whole-body stack doing impedance control on the arms.
The policy is FLUX-mimic, developed with Black Forest Labs on the FLUX 3 backbone.
The action decoder is lightweight and reads intermediate features from the backbone’s video prediction path. Over 95% of training compute went into predicting video. Actions are a thin readout on top.
And it works with the backbone frozen. Black Forest Labs reports the decoder beating prior VLAs with no backbone finetuning at all, a setting where VLAs reportedly fail outright.
The manipulation-relevant structure is already sitting in the video representation. Nobody taught it contact or cause and effect as a separate objective. It fell out of predicting tens of millions of hours of video.
One detail from the demo is the drop-in teleop. The operator takes over mid-episode to correct a wrong move, then hands back and the policy resumes.
That’s the safety story, and every correction is a labelled intervention at precisely the moment the policy was wrong.
Running at Audi on kitting, ECU insertion, and soft parts like seals and cables.
Thanks for explaining @elvisnavah! 🦾
It was awesome during @NVIDIARobotics GTC, thanks for inviting me!
~~
♻️ Join the weekly robotics newsletter, and never miss any news → ziegler.substack.com
mimic retweeted
We've been running a fully autonomous demo of FLUX-mimic nonstop at GTC for two days. We're even golden hour -proof. Lfg @mimicrobotics
mimic retweeted
The team Just got set up at GTC Berlin. You should come over to the @mimicrobotics x @bfl_ai booth, it's really good!
mimic retweeted
GTC Berlin, next week.
The team will be on the ground, come find us.
FLUX 3 Action at the @Nebius booth. FLUX 3 Image and Video at the NVIDIA booth, and a @mimicrobotics robot running FLUX-mimic.
@robrombach will be speaking on stage on Oct 22, 9AM for a panel and 4PM for a keynote.
We are hosting Afterglow, on Oct 21 with @FactoryAI, @modal and @NVIDIAAI, link in thread.
From day one, mimic has been focused on a single goal: general-purpose dexterous manipulation. Today we're proud to announce the mimic hand M1 and the mimic wearable U1.
We believe the only way to solve dexterous manipulation at scale is by going full-stack at the frontier of physical AI, building every layer ourselves around one fixed point, the human hand.
The M1 is a highly backdrivable, tendon-driven hand that covers the full range of human capability, from heavy payloads to fine manipulation.
The U1 is an exoskeleton that records human demonstrations directly matched to the M1's kinematics, no robot required.
Dexterity is one of the last barriers between AI and the physical world. Solving it unlocks entire categories of physical work that automation has never been able to reach.
This is the foundation on top of which our frontier AI work is done, and we’re just getting started. Full thesis linked here.
mimicrobotics.com/blog/solvi…
mimic retweeted
A few months ago, I travelled to Zurich together with @NVIDIArobotics to spend time inside one of Europe's most exciting robotics startups...
@mimicrobotics!
Their mission? To solve one of the hardest problems in robotics: human-level dexterity.
Because before robots can replace physical work, they first need to master the human hand.
In this episode, we go behind the scenes with the team building AI-powered robotic manipulation, discuss why dexterity is still one of the biggest bottlenecks in automation, and explore what it will take to bring truly capable robots into factories.
Mimic Robotics is part of the NVIDIA Inception, a program for startups and VCs, and is leveraging Cosmos to advance AI-powered robot learning.
If you're interested in Physical AI, robotic manipulation, and where this industry is actually heading, I think you'll enjoy this one.
Timestamps:
0:00 Inside Mimic Robotics
0:24 Building the future of robot manipulation
1:19 The mission behind Mimic Robotics
1:35 The AI powering autonomous robots
2:14 Why Mimic chose NVIDIA Cosmos
2:34 Video foundation models vs. traditional robotics
2:46 Building a multidisciplinary robotics team
3:24 Bridging customers and cutting-edge AI
4:40 Why Zurich is the perfect robotics hub
5:20 Europe vs. America: Scaling a robotics startup
6:30 Bringing generative AI to factory robots
6:49 The race to build Europe's robotics hyperscaler
♻️ Know someone working in robotics? They'll probably want to see this.
Check out the latest episode of @RoboPapers featuring @elvisnavah, mimic-video and Video Action Models!
Robotics fundamentally involves understanding the dynamics of how things change in the world in response to action and force. This is impossible to learn from static images; instead, it’s far more effective and more data-efficient to learn from video.
@elvisnavah joins us to talk about @mimicrobotic. One of the key findings from mimic-video is that pretraining on webscale video allows robots to learn physics priors; as a result, policies train faster, generalize better, and are capable of more impressive dexterity, versus training on static images or image-language pairs as per a VLM.
Watch Episode #81 of RoboPapers with @micoolcho and @chris_j_paxton to learn more!
mimic retweeted
I love @RoboPapers so it was cool to be invited and chat with @chris_j_paxton and @micoolcho and talk about VAMs and mimic-video!
Robotics fundamentally involves understanding the dynamics of how things change in the world in response to action and force. This is impossible to learn from static images; instead, it’s far more effective and more data-efficient to learn from video.
@elvisnavah joins us to talk about @mimicrobotic. One of the key findings from mimic-video is that pretraining on webscale video allows robots to learn physics priors; as a result, policies train faster, generalize better, and are capable of more impressive dexterity, versus training on static images or image-language pairs as per a VLM.
Watch Episode #81 of RoboPapers with @micoolcho and @chris_j_paxton to learn more!
