Austin Veselka

New blog post on context parallelism, one of the large technical feats behind ART. furtherai.github.io/blog/pos… It's quite a long read but really cool if you're interested in the engineering. I think Astra has done quite a good job with my recent prompt in writing and revising this. I did work very closely with codex for the attn and GDN versions. This was maybe gpt 5.2 so it really isn't something where it can get very far itself. So, we planned and iterated on the whole system and details together. Only thing I didn't do is write the code, the Triton kernels. But these systems handle not only CP, but prefix tree dependencies as well. No one else does that. I think the models will be able to better take over in the future (e.g. the Mamba 2 version was unsupervised by 5.6 sol, not entirely sure how well this one works cause I didn't really stress test it's implementation past what it was designed for but the design is quite simple so I suspect it's ok). Luckily we have most CP regimes so new designs don't have to come from scratch.

To give an idea of the scale of performance for ART, I ran some benchmarks against SkyRL and e2e training is 3-19x faster. Qualifier is really just that we're lora only. Though it wouldn't be too hard to do full-weight. The gains are a result of 3 large technical advantages (prefix tree packing, CP, e2e optimization and autotuner) plus optimization on actual trainer speed. 19x comes when the prompt is 5-10k tokens and completions are short (matching some internal workloads) and 3x is for decode/inference bound cause we currently don't do inference replicas (requires a bit of work to route requests and handle potentially different replica states, I'll get to it). I'm fucking crushing it. Btw I did bench against slime (the popular guy) and we released a blog post about it earlier this year (would expect roughly the same speedups), but idk if they've made any of the important technical improvements for this kind of speed in the meantime. Maybe they have some advantages for coding agent style training

Astra does really well at coming up with code-based hypotheses for errors. It's always like, "We hit this mysterious issue, could be anything. First attempt to attach diagnostics failed so we really have no idea." Then you ask for a code based hypothesis from reading the relevant code path and it says, "Yeah here's the likely problem." 10 min later it confirms and fixes it. Haven't tried this much with other models really, previously they've been working on something where I can trust them to debug quickly. Current iteration loop is too slow for that

Very cool work, check out my autotuner in ART which solves a similar problem. Async RL requires setting a lot of parameters manually (rollout workers, min and max training batch size, queue size), and these can be a big lever on the e2e throughput of your training. The autotuner tries to optimize these automatically based on the concepts of trainer and inference load relative to their peaks. e.g. trainer and inference are underfed: increase rollout workers. The full logic is a bit more complex but reasonably robust and does a great job. Main caveat is that it may not be optimal in terms of real training to, say, increase batch size. This depends on needed sample efficiency of your run and the SNR. But the tuner doesn't know this precisely and generally (could be fun research). It still is usually way better to have it on than off.

furtherai.github.io/blog/ Made a blog for technical write-ups, will discuss my ART infra work (probably cool stuff to read if you're interested in RL infrastructure), math I'm reading (guides from codex on MuP/Complete(d)P-related math) as well as research projects in the future. I hate writing so this will be leaning heavily on AI (thank God for LMs to do this for me), but I'll read and edit and guide the writing, hopefully making it make sense. Would love to see questions, feel free to reach out. First post discusses prefix tree packing in ART, check it out if you're interested in RL and how to apply prefix caching to training.

I have been really optimizing ART for a while now, this is the first of some pretty interesting things I have built and we'll share. We'll talk about some really impressive context parallelism next time and then optimizing the end-to-end training speed of ART!