apaz (@apaz_cli) header
apaz (@apaz_cli) profile picture

https://nitter.gq/t.co/EYtS07NoX4 Making GPUs go brrr

Hiding in your wifi
Joined July 2019
I really need to turn mlsweep into an RL environment sometime.
7
534
I've mostly automated my experiments. Made a lot of progress on zeroth order pretraining, and I'm sitting on a bunch of findings. The bottleneck now is understanding and communicating them. I'll get around to open sourcing the tools I'm using also. It's RSI in an obsidian vault.
6
6
145
5,582
Over the past week or so I've been carefully building a research wiki system too. For exactly the same reasons. I want to let agent swarms run rampant over my research projects, and they need a way to communicate with themselves, and with me. So, Anthropic's notes on wiki memory systems are ABSOLUTELY going to "accelerate other AI developers."
5
2
57
3,036
I'm actually really confused as to why Anthropic would release the memory wiki findings. Seeing as they were so concerned in the METR report to censor even the name of their agent scaffold. Which undoubtedly uses some similar kind of memory system for grounding and coordination.
wtf this is unreasonably powerful had Opus write a wiki-style memory system for 1017, iterated a bit, and the uplift is honestly spooky. RSI in one Obsidian vault, Yud's nightmare. I cannot see yet how far it goes. I'll need to investigate more
4
5
1
166
12,593
Replying to @stochasticchasm
Yes you do
2
78
Replying to @dejavucoder
This is true.
2
109
Kinda insane that mlsweep's scope has crept from being a tool for making charts, to a fully fledged automated research system. I haven't been talking about it at all. I've just been using it to run my own experiments and build my own infra. But, more updates coming soon.
Releasing mlsweep, a sweep scheduler and visualizer for distributed ML training. It aims to make launching runs across groups GPUs frictionless and achieve near feature-parity with wandb. But you can use it with whatever frameworks or loggers you like, wandb included.
7
364
Dust is not just a small improvement over SPSA/ES/EGGROLL/MeZO. It's huge. I'll leave it to the article to discuss why. But the implied (and observed) improvements over weight perturbation are enormous. With that said, the theory still suggests room for improvement.
1
6
268
There have been a bunch of Zeroth Order Optimization research papers recently. I wrote an article digesting them and their mathematical implications. I went a bit overboard. It covers Dust and noise (qlabs), SOMA (@FrancoisChauba1), Async ZO (@BillyHoy1_), and future directions.
3
5
30
1,395
Replying to @MangoSweet78
For me a lot of it really is just sleeping well. After I turned 26 or so, it became no longer optional.
51
what if you just shot your infra engineers
12
613