Robot Learning Research | CS PhD from Penn

San Francisco
Joined August 2014
For the record: attention, scale, and a sufficiently hard problem is all you need
1
1
55
Flex-π is wonderful work interesting result: test time “full joint” modality generation produces much better actions than action gen alone “full joint” generates auxiliary outputs and then attends to them for action generation I found this result surprising. looking at the configs, there’s two possible explanations: - it’s undertrained in action only regime (only hits 1/8th of the time) - it’s able to leverage significantly more computation at test time I suspect a combination of the two are at play, but I think it merits more investigation!
Flex-π is now open source. 💻 github.com/geyan21/flex-pi 🤗 huggingface.co/flex-pi Code, checkpoints, and all our real-robot data from the five bimanual YAM tasks. One checkpoint runs every mode, action-only to full joint. Large-scale pre-training checkpoints coming next. Give it a try!
3
3
88
13,282
one ablation to test this would be the same number of FLOPs to train and test the model, but let the tokens be scratch pad tokens rather than structured latents it’s possible we just need to give models register tokens arxiv.org/abs/2309.16588
6
1,114
what did i learn? agents are a huge lever for high level intuition if you can break a problem down well, these tools can take many shots on goal for subproblems you get results faster, AND you intuition gets better bc you get more reps (GRPO on your brain) (5/7)
1
16
1,410
if you make your living doing low level work well, you’ll probably find them only marginally useful — Go players got better watching AlphaGO, but not enormously this matches SWE today — agents are a superpower for seniors and turn juniors into slop cannons (6/7)
1
13
1,183
what did i prove? for a specific type of graph, i answered an open problem about the minimum edit needed to disrupt connectivity to more than one node the topic is niche, but what’s exciting is I used agents to get to a creative full proof from only a high level idea (4/7)
1
9
977
I proved novel math results using frontier models my takeaway: *regardless of domain* if you have the ability to take big problems and break them down well, these models are a huge lever if you're a low level execution monkey, its truly over a thread (1/7)
3
3
1
79
8,186
it’s hard to do this for math research ability - computation (“solve this integral”) is different than proving a theorem - (almost) all math shares the same premises, so unless you’re doing truly new math, the model could just be recalling a proof online (3/7)
1
1
7
1,178
first, why? your codebase has novelty in it, so it’s easy to tell when a coding model is recalling patterns vs working with the nuances of your code you can feel their creativity when doing new software / system design atop (2/7)
1
8
1,683
it’s incredible what robot learning has already taught us about the universe throw 50 hours of teleoperation at a flow matching model and arms suddenly know how to fold laundry truly a marvelous fact
4
1
141
9,467
my father’s paternal line descends from the first Vedder, who changed his name when he moved to what’s now New York in 1654 without a penny to his name. he lived in a wooden cabin he built himself, where he, his wife, and all but one of his sons were murdered by natives; the surviving son spawned a long line of successful doctors, lawyers, and technologists. my father’s maternal line immigrated in 1946, after surviving 4 years of horror in a Japanese run concentration camp in Indonesia. they started by cleaning hotels and apartments with their hands in NYC and ended with a successful financial services business my mother’s father saw his family for the last time at 14 when he left to walk down the road until he found work; the great depression left his family unable to feed him. he found work as a farm hand, ultimately becoming a successful dairy farm owner / operator and land dealer the gift the universe gave them was the inner strength to be the masters of their own fate, no matter the cards they were dealt the gift this country gave them was opportunity to do this, unhindered by extractive institutions, rigid social casts, and entrenched interests this is what makes the United States of America so special. the combination of opportunity and human capital, selected for people willing to put everything on the line for a better life, is found nowhere else on earth it’s why we have one of the highest GDPs per capita on earth. it’s why we have led and continue to lead in every major technological and scientific development in the last century despite having a population smaller than europe or many countries in asia. it’s why the median American’s quality of life vastly exceeds the richest in all of antiquity. and it’s why I’m proud to be an American happy 250th to the greatest country on earth 🇺🇸🌎🦅 – at Panama City Beach, FL
3
1
88
5,091
i think this is best summed up in the words of one of the greatest statesmen to ever live, Lee Kuan Yew “the American culture… is that we start from scratch and beat you”
12
1,019
I made this bug hunt loop into a codex skill
Replying to @KyleVedder
My pro tip tell codex your runtime is riddled with bugs, and its job is to 1) make a commit that adds a failing test showcasing the bug 2) make a commit to fix that on a branch in a loop Let it run for 24 hours, kill it, point a new session at the giant wall of commits and ask it to assess what bugs are real vs fake and make a nice clean PR for the real bugs plus their fixes About 75% of the bugs it finds will be real, and all it cost you were tokens and wall clock time, not your time
1
22
4,915
“why doesn’t the pretrained robot learning model work on my hardware?” Check: - state definition matches exactly what the model expects (e.g. action norm stats) - cameras aren’t swapped - arms aren’t swapped - images arent bgr by accident (opencv 🤬) - action space is correct (ee pose vs joint angles) - hardware gripper units are correct (ARX is 0 to 4.5 rad, i2rt is 1 to 0, etc) - record and playback on same hardware works - BCing a simple pick up task with ~10 demos works
5
1
118
8,184
My pro tip tell codex your runtime is riddled with bugs, and its job is to 1) make a commit that adds a failing test showcasing the bug 2) make a commit to fix that on a branch in a loop Let it run for 24 hours, kill it, point a new session at the giant wall of commits and ask it to assess what bugs are real vs fake and make a nice clean PR for the real bugs plus their fixes About 75% of the bugs it finds will be real, and all it cost you were tokens and wall clock time, not your time
2
26
6,858
opinion piece: cargo cults, data flywheels, and novelty pumps tl;dr: “deployment data flywheels” miss that scale should provide novelty, and deploying robots on a known task wrings most novelty out of its generated data your ops team’s novelty pump is the only way out 🔗👇
11
9
9
115
30,690
full blog post motivates this further and closes by reminding you to cherish your elite operations talent, bc that’s who’s making your policy better in the long run vedder.io/misc/novelty_pump.…
2
33
3,838
unfortunately you need to be long prestige I’m a public school kid (town hs, state ugrad) I’ve had great life outcomes, but I had to be far better and work far harder than prestiged peers to even get a shot as AGI compresses competency away, who you know will matter even more
Ok this is my final comment on this nonsense: my qualifications are that I have spent all my time with people from anthropic, jane street, goldman, mckinsey, evercore, meta, google, citadel etc. everyone I know is clearing 1m TC at age 28 unless they're founders. In the top 0.5% of society (which these institutions dominate), the average feeder school grad is *not* sufficiently intelligent Did you go to phillips exeter? Wow congrats, you probably will still. bomb your jane street interview. Sorry, you should have been at a public school in the bay area doing the AIME. The *only* place these networks still matter is high finance -- will be way easier for you to network into niche hedge funds, family offices, smaller PE firms etc. if you have this WASPy network. In tech, quant and west coast VC, your prep schoo doesn't matter for shit. And as these roles started outpaying finance, it became more pronounced -- why make 400m at some sweaty 5 person PE firm as a spreadsheet monkey when you can leetcode your way to 1m at Meta doing more interesting work? Tech/quant opened up a bunch of highpaying roles with a more meritocratic process and broke the prep-school WASP monopoly on elite, high paying jobs That's why no one on the west coast cares about these prep-school lists. They're literally gatekeeping the path to mid-paying, relatively unprestigious jobs. Look at all the top founders of the last 10 years -- tons of them were just bay area asians who were grinding science olympiad, quizbowl, and mathcounts -- thats the new meta a la scott wu for better or worse
8
5
181
27,839
u need to pick the right prestige ladders tho OP is left side screeching at right side, but access to capital is already commoditized; unclear if left side can turn money into real influence rn a frmr trump 1 toilet paper boy has more political power than an OAI decamillionare
1
19
1,924