apaz (@apaz_cli) header
apaz (@apaz_cli) profile picture

https://nitter.gq/t.co/EYtS07NoX4 Making GPUs go brrr

Hiding in your wifi
Joined July 2019
Photo by apaz (@apaz_cli)Photo by apaz (@apaz_cli)Photo by apaz (@apaz_cli)Photo by apaz (@apaz_cli)Photo by apaz (@apaz_cli)Photo by apaz (@apaz_cli)Photo by apaz (@apaz_cli)Photo by apaz (@apaz_cli)Photo by apaz (@apaz_cli)Photo by apaz (@apaz_cli)
Pinned Tweet
Releasing mlsweep, a sweep scheduler and visualizer for distributed ML training. It aims to make launching runs across groups GPUs frictionless and achieve near feature-parity with wandb. But you can use it with whatever frameworks or loggers you like, wandb included.
4
8
1
54
8,475
Eva is a student at @PrincetonNeuro working with me, @brody_lab and Ben Eysenbach. She was abducted by ICE while trying to board a flight from SFN. Some thoughts below how we can help her.
Our colleague Eva Yi Xie, a Princeton neuroscience PhD student, was detained by ICE at Reagan National Airport on her way home from SfN, where she’d just presented her poster. She’s being held in Chantilly, VA. Please help us bring her home. 📷
277
297
21
1,792
250,833
Do any labs actually still use wandb? They don't, right?
3
44
5,958
I really need to turn mlsweep into an RL environment sometime.
7
532
I've mostly automated my experiments. Made a lot of progress on zeroth order pretraining, and I'm sitting on a bunch of findings. The bottleneck now is understanding and communicating them. I'll get around to open sourcing the tools I'm using also. It's RSI in an obsidian vault.
6
6
145
5,581
My GPUs are a space heater and I cuddle up next to them on chilly days while my experiments run. Tell me I'm not weird, I can't be the only one who does this.
1
11
421
I'm actually really confused as to why Anthropic would release the memory wiki findings. Seeing as they were so concerned in the METR report to censor even the name of their agent scaffold. Which undoubtedly uses some similar kind of memory system for grounding and coordination.
wtf this is unreasonably powerful had Opus write a wiki-style memory system for 1017, iterated a bit, and the uplift is honestly spooky. RSI in one Obsidian vault, Yud's nightmare. I cannot see yet how far it goes. I'll need to investigate more
4
5
1
166
12,593
Over the past week or so I've been carefully building a research wiki system too. For exactly the same reasons. I want to let agent swarms run rampant over my research projects, and they need a way to communicate with themselves, and with me. So, Anthropic's notes on wiki memory systems are ABSOLUTELY going to "accelerate other AI developers."
5
2
57
3,036
I did not expect them to just drop a research acceleration gold mine on me for no reason.
9
751
Sharon was one of the few people at AMD I interacted with who gave me hope for the company. This is a great loss for AMD, more than anyone realizes.
I've joined Anthropic! I'm excited to get back to (frontier) research and work on the models that I fell in love with a decade ago. I’m very grateful for my time at @AMD ♥️ Starting with the people: I'm honored to have had the opportunity to work with the one and only @LisaSu! She's an amazing leader, manager, and friend who allowed me so much agency in my work -- they say don't meet your heroes, but she definitely lives up to the pedestal that I've put her on. And thank you to my team: dedicated, resilient folks who've been brave enough to jump on a rocket ship together, in some cases several times to ever ambitious projects. For those who came with me from my startup Lamini - thank you also for bringing your passion, wonder, kindness, and (yes) silliness with me, as we found a home at AMD. And now the work: I built and ran an AI research org that led our frontier lab partnerships, where I served as executive sponsor. We were among the first in the AI research world to take RL kernel optimization really seriously last year, and it showed me that pushing a frontier capability forward takes far fewer people than you’d think (100x, and I'm now updating my priors to 1000x fewer, than just a year ago). That work has also had massive implications for the compute that labs and companies can now use out-of-the-box with models like Claude. That is to say, it’s always time to make some GPUs go brrr.
29
2,235
I think zettelkasten are a very bad abstraction for humans doing research. It takes so much time to wrangle and organize and maintain. Time you could spend on the thing you're actually working on. This logic does not apply to agent swarms.
7
453
Kinda insane that mlsweep's scope has crept from being a tool for making charts, to a fully fledged automated research system. I haven't been talking about it at all. I've just been using it to run my own experiments and build my own infra. But, more updates coming soon.
Releasing mlsweep, a sweep scheduler and visualizer for distributed ML training. It aims to make launching runs across groups GPUs frictionless and achieve near feature-parity with wandb. But you can use it with whatever frameworks or loggers you like, wandb included.
7
364
There have been a bunch of Zeroth Order Optimization research papers recently. I wrote an article digesting them and their mathematical implications. I went a bit overboard. It covers Dust and noise (qlabs), SOMA (@FrancoisChauba1), Async ZO (@BillyHoy1_), and future directions.
3
5
30
1,395
Dust is not just a small improvement over SPSA/ES/EGGROLL/MeZO. It's huge. I'll leave it to the article to discuss why. But the implied (and observed) improvements over weight perturbation are enormous. With that said, the theory still suggests room for improvement.
1
6
268
Reach out to me if you want to talk about zeroth order optimization, or any of the papers cited. It's too big a topic for any individual to tackle alone, and there is strength in community and openness. Blog post: apaz.dev/blog/ZO_Research_Up…
1
3
182
Up at 4am monitoring the research swarm, alas for my sleep schedule
1
5
280
it is honestly quite baffling to me that openai and anthropic still use slack. i expect this will change within the next six months. they could quite clearly build and maintain internal versions that are *more* secure, robust, and featureful with a team of ~10
110
19
16
1,435
504,527
There is very interesting overlap actually between the Dust paper and the YOCO paper. Dust is a biased estimator that prevents the model from being able to learn to generate kvcache beneficial to predicting upcoming tokens. But YOCO architecturally pre-removes this possibility.
1
13
860
This is perhaps half of the full story. Specifically, it splits it into a dust-optimal part (the cross decoder) and a dust-hostile part (the self decoder). Need to do experiments on what should be done here but I think it may lead to a better arch.
1
117
Why does this take two years to build what am I missing, I feel like I could do this in a couple weeks.
We're announcing the public beta of datavendor.ai We spent the last two years building the best solution for people buying and selling data, whether you're a researcher in a lab or a company trying to sell data. Data is the fundamental bottleneck for AI progress, and DataVendor will be the infra that solves it.
5
43
8,590