Flex-π is wonderful work
interesting result: test time “full joint” modality generation produces much better actions than action gen alone
“full joint” generates auxiliary outputs and then attends to them for action generation
I found this result surprising.
looking at the configs, there’s two possible explanations:
- it’s undertrained in action only regime (only hits 1/8th of the time)
- it’s able to leverage significantly more computation at test time
I suspect a combination of the two are at play, but I think it merits more investigation!
Flex-π is now open source.
💻 github.com/geyan21/flex-pi
🤗 huggingface.co/flex-pi
Code, checkpoints, and all our real-robot data from the five bimanual YAM tasks. One checkpoint runs every mode, action-only to full joint.
Large-scale pre-training checkpoints coming next. Give it a try!
one ablation to test this would be the same number of FLOPs to train and test the model, but let the tokens be scratch pad tokens rather than structured latents
it’s possible we just need to give models register tokens
arxiv.org/abs/2309.16588
what did i learn?
agents are a huge lever for high level intuition
if you can break a problem down well, these tools can take many shots on goal for subproblems
you get results faster, AND you intuition gets better bc you get more reps (GRPO on your brain)
(5/7)
if you make your living doing low level work well, you’ll probably find them only marginally useful — Go players got better watching AlphaGO, but not enormously
this matches SWE today — agents are a superpower for seniors and turn juniors into slop cannons
(6/7)
i recommend checking out my longer writeup and the paper to learn more (7/7)
vedder.io/misc/math_research…
what did i prove?
for a specific type of graph, i answered an open problem about the minimum edit needed to disrupt connectivity to more than one node
the topic is niche, but what’s exciting is I used agents to get to a creative full proof from only a high level idea
(4/7)
I proved novel math results using frontier models
my takeaway:
*regardless of domain*
if you have the ability to take big problems and break them down well, these models are a huge lever
if you're a low level execution monkey, its truly over
a thread (1/7)
first, why?
your codebase has novelty in it, so it’s easy to tell when a coding model is recalling patterns vs working with the nuances of your code
you can feel their creativity when doing new software / system design atop
(2/7)
it’s incredible what robot learning has already taught us about the universe
throw 50 hours of teleoperation at a flow matching model and arms suddenly know how to fold laundry
truly a marvelous fact
my father’s paternal line descends from the first Vedder, who changed his name when he moved to what’s now New York in 1654 without a penny to his name. he lived in a wooden cabin he built himself, where he, his wife, and all but one of his sons were murdered by natives; the surviving son spawned a long line of successful doctors, lawyers, and technologists.
my father’s maternal line immigrated in 1946, after surviving 4 years of horror in a Japanese run concentration camp in Indonesia. they started by cleaning hotels and apartments with their hands in NYC and ended with a successful financial services business
my mother’s father saw his family for the last time at 14 when he left to walk down the road until he found work; the great depression left his family unable to feed him. he found work as a farm hand, ultimately becoming a successful dairy farm owner / operator and land dealer
the gift the universe gave them was the inner strength to be the masters of their own fate, no matter the cards they were dealt
the gift this country gave them was opportunity to do this, unhindered by extractive institutions, rigid social casts, and entrenched interests
this is what makes the United States of America so special. the combination of opportunity and human capital, selected for people willing to put everything on the line for a better life, is found nowhere else on earth
it’s why we have one of the highest GDPs per capita on earth. it’s why we have led and continue to lead in every major technological and scientific development in the last century despite having a population smaller than europe or many countries in asia. it’s why the median American’s quality of life vastly exceeds the richest in all of antiquity.
and it’s why I’m proud to be an American
happy 250th to the greatest country on earth 🇺🇸🌎🦅 – at Panama City Beach, FL
I made this bug hunt loop into a codex skill
Replying to @KyleVedder
My pro tip
tell codex your runtime is riddled with bugs, and its job is to
1) make a commit that adds a failing test showcasing the bug
2) make a commit to fix that
on a branch in a loop
Let it run for 24 hours, kill it, point a new session at the giant wall of commits and ask it to assess what bugs are real vs fake and make a nice clean PR for the real bugs plus their fixes
About 75% of the bugs it finds will be real, and all it cost you were tokens and wall clock time, not your time
“why doesn’t the pretrained robot learning model work on my hardware?”
Check:
- state definition matches exactly what the model expects (e.g. action norm stats)
- cameras aren’t swapped
- arms aren’t swapped
- images arent bgr by accident (opencv 🤬)
- action space is correct (ee pose vs joint angles)
- hardware gripper units are correct (ARX is 0 to 4.5 rad, i2rt is 1 to 0, etc)
- record and playback on same hardware works
- BCing a simple pick up task with ~10 demos works
My pro tip
tell codex your runtime is riddled with bugs, and its job is to
1) make a commit that adds a failing test showcasing the bug
2) make a commit to fix that
on a branch in a loop
Let it run for 24 hours, kill it, point a new session at the giant wall of commits and ask it to assess what bugs are real vs fake and make a nice clean PR for the real bugs plus their fixes
About 75% of the bugs it finds will be real, and all it cost you were tokens and wall clock time, not your time
opinion piece: cargo cults, data flywheels, and novelty pumps
tl;dr: “deployment data flywheels” miss that scale should provide novelty, and deploying robots on a known task wrings most novelty out of its generated data
your ops team’s novelty pump is the only way out
🔗👇
full blog post motivates this further
and closes by reminding you to cherish your elite operations talent, bc that’s who’s making your policy better in the long run
vedder.io/misc/novelty_pump.…
unfortunately you need to be long prestige
I’m a public school kid (town hs, state ugrad)
I’ve had great life outcomes, but I had to be far better and work far harder than prestiged peers to even get a shot
as AGI compresses competency away, who you know will matter even more
Ok this is my final comment on this nonsense:
my qualifications are that I have spent all my time with people from anthropic, jane street, goldman, mckinsey, evercore, meta, google, citadel etc. everyone I know is clearing 1m TC at age 28 unless they're founders.
In the top 0.5% of society (which these institutions dominate), the average feeder school grad is *not* sufficiently intelligent
Did you go to phillips exeter? Wow congrats, you probably will still. bomb your jane street interview. Sorry, you should have been at a public school in the bay area doing the AIME.
The *only* place these networks still matter is high finance -- will be way easier for you to network into niche hedge funds, family offices, smaller PE firms etc. if you have this WASPy network.
In tech, quant and west coast VC, your prep schoo doesn't matter for shit. And as these roles started outpaying finance, it became more pronounced -- why make 400m at some sweaty 5 person PE firm as a spreadsheet monkey when you can leetcode your way to 1m at Meta doing more interesting work?
Tech/quant opened up a bunch of highpaying roles with a more meritocratic process and broke the prep-school WASP monopoly on elite, high paying jobs
That's why no one on the west coast cares about these prep-school lists. They're literally gatekeeping the path to mid-paying, relatively unprestigious jobs.
Look at all the top founders of the last 10 years -- tons of them were just bay area asians who were grinding science olympiad, quizbowl, and mathcounts -- thats the new meta a la scott wu for better or worse
















