
Ph.D. @UofIllinois | Undergrad @UMich. Language Modeling.
Joined November 2024
- Tweets216
- Following573
- Followers381
- Likes566
Pinned Tweet
Transformers usually scale by making their blocks wider or stacking them deeper.
We show another path toward scaling LLMs: branching.
Introducing Multi-Branch Transformer (MBT): parallel composition of full Transformer blocks.
TL;DR:
• Better scaling: across a controlled pretraining scaling ladder, MBTs improve language-model quality at comparable training compute. Fitted curves indicate a standard Transformer needs up to 1.26× as much training compute to match our final MBTs' validation loss.
• Consistent pattern: controlled sweeps show the same behavior at every scale. With two branches, loss is lowest with ~1/3 of the blocks in parallel and only returns to the standard model's level near 80%.
• Faster decoding: with Joint-KV cache sharing and a branch-aware inference engine, up to 51.4% higher throughput on one H200 and up to 34% less KV-cache storage. More branches push the quality–speed Pareto frontier further out.
The architecture, the scaling results, and how we make it fast. 🧵
Gaotang Li @ NeurIPS2026 ATL retweeted
"Multi-Branch Transformer"
A potential way to scale Transformers might not be making them deeper or wider, but splitting them into multiple parallel branches.
This paper introduces Multi-Branch Transformers (MBTs), which let multiple independently trained Transformer blocks to process the same input in parallel before combining their outputs.
And by using their proposed Joint-KV, this lets these branches to share a KV cache instead of storing separate copies, reducing memory needs.
Across MoE models from 0.8B to 2.1B parameters, scaling estimates suggest conventional Transformers require up to 1.26x more training compute to achieve the same validation loss.
The final architecture also reduces KV-cache storage by up to 34% and improves decoding throughput by up to 51.4% on a single H200 GPU.
alphaxiv.org/abs/2611.19800
Gaotang Li @ NeurIPS2026 ATL retweeted
We decided to provide more transparency into pretraining progress at OpenAI for the sake of moving toward shared safety bars. In “Perplexity and Intelligence” I argue for common limits based on perplexity and other measurements, with progress beyond them gated on our ability to understand AI behavior.
openai.com/index/perplexity-…
Gaotang Li @ NeurIPS2026 ATL retweeted
What does it take to turn VLMs for cost-efficient zero-shot robot manipulation, without tools, without task-specific training, without external modules like VLA-as-tool or SAM3 or IK Solver ?
🤖 Introducing MotorMind !
💡 Zero-shot Manipulation: No Fine-tuning or Coding agent
💡 VLM-only: No external perception models, No tools, No VLA-as-tool, No IK-Solver
💡 Local Deployable: Qwen3.8-Next-Flash as backbone model for cost-efficient local deployment
💡 Strong Generalization: 83.3% success on LIBERO-PRO
💡 Real-Robot Transfer: 95% success on real-robot tasks without embodiment-specific training
Check out more:
📄 Paper: arxiv.org/pdf/2609.38078
🌐 Project: motor-mind.github.io/
💻 Code: github.com/Motor-Mind/MotorM…
Gaotang Li @ NeurIPS2026 ATL retweeted
A new way to arrange the blocks of a Transformer:
Usually models grow with wider blocks or more blocks in a row. They put some whole blocks side by side instead, as parallel branches with their own weights. The outputs are merged, with shared blocks before and after the branches. They hoped for better quality at the same training compute. Branches can also run at the same time, so generation could be faster.
With almost the same parameters and training compute, two branches gave lower validation loss than a standard Transformer.
Their final models share the KV cache across branches, which makes it 19% to 34% smaller than a standard Transformer's. All branches at the same depth use one cache, and each keeps its own attention queries. If each branch kept its own cache, a 1.6B model with 4 branches would need 1.2x the standard model's cache. Sharing brings it to 0.675x.
Decoding was also faster than in the standard model, because same-depth branches run as one step and read the shared cache once. On one H200 GPU, averaged over the four sizes, the 4-branch models decoded at 1.12x the speed of standard models with one 4K-token request. With 16 requests of 128K tokens at once, the average was 1.50x.
Transformers usually scale by making their blocks wider or stacking them deeper.
We show another path toward scaling LLMs: branching.
Introducing Multi-Branch Transformer (MBT): parallel composition of full Transformer blocks.
TL;DR:
• Better scaling: across a controlled pretraining scaling ladder, MBTs improve language-model quality at comparable training compute. Fitted curves indicate a standard Transformer needs up to 1.26× as much training compute to match our final MBTs' validation loss.
• Consistent pattern: controlled sweeps show the same behavior at every scale. With two branches, loss is lowest with ~1/3 of the blocks in parallel and only returns to the standard model's level near 80%.
• Faster decoding: with Joint-KV cache sharing and a branch-aware inference engine, up to 51.4% higher throughput on one H200 and up to 34% less KV-cache storage. More branches push the quality–speed Pareto frontier further out.
The architecture, the scaling results, and how we make it fast. 🧵
Gaotang Li @ NeurIPS2026 ATL retweeted
Inference compute is the largest untapped training resource in AI.
Can we use this ever-increasing resource to train models while they are being served? TL;DR: Yes!
We call it Population Serving: a new inference paradigm where, instead of serving everyone the exact same weights, we serve slightly perturbed copies of a model, score each on the traffic it gets, and merge them periodically.
No backprop and only one inference call per user query. Zeroth-order methods do their magic!
Work done at @berkeley_ai with Marquita Ellis, @GopalaSpeech and @LChoshen.
Blog: akshat57.github.io/blog/popu… 🧵
Gaotang Li @ NeurIPS2026 ATL retweeted
Excited to share our work on MBT! Through extensive ablations and a controlled scaling ladder, we explore branching as a promising new axis for scaling LLMs.
The architecture also opens up new opportunities to make inference faster through better parallel execution and KV-cache reuse. Lots more to explore, and really looking forward to seeing what others build on this!
Shout-out to all wonderful collaborators!
Transformers usually scale by making their blocks wider or stacking them deeper.
We show another path toward scaling LLMs: branching.
Introducing Multi-Branch Transformer (MBT): parallel composition of full Transformer blocks.
TL;DR:
• Better scaling: across a controlled pretraining scaling ladder, MBTs improve language-model quality at comparable training compute. Fitted curves indicate a standard Transformer needs up to 1.26× as much training compute to match our final MBTs' validation loss.
• Consistent pattern: controlled sweeps show the same behavior at every scale. With two branches, loss is lowest with ~1/3 of the blocks in parallel and only returns to the standard model's level near 80%.
• Faster decoding: with Joint-KV cache sharing and a branch-aware inference engine, up to 51.4% higher throughput on one H200 and up to 34% less KV-cache storage. More branches push the quality–speed Pareto frontier further out.
The architecture, the scaling results, and how we make it fast. 🧵
Check out "Multi-Branch Transformer" on alphaXiv alphaxiv.org/abs/2611.19800
Gaotang Li @ NeurIPS2026 ATL retweeted
Multi-Branch Transformer
Scales LLMs by branching, running full Transformer blocks in parallel instead of only making them wider or deeper, reaching lower validation loss at matched compute and up to 51.4% faster decoding.
Gaotang Li @ NeurIPS2026 ATL retweeted
1/ MBT composes full Transformer blocks in parallel branches.
We keep the shared initial and final layers and reorganize the computation in between. Each branch has its own parameters and remains a sequence of full Transformer blocks. Every token passes through every branch, and the branch outputs are fused before the shared final layers.
Relative to a standard Transformer, we explore three instantiations at comparable training compute:
• Wide–Shallow: keep block width and distribute the middle blocks across branches.
• Narrow–Deep: keep the sequential depth along each branch, with narrower blocks.
• Balanced: use intermediate branch width and depth.
Gaotang Li @ NeurIPS2026 ATL retweeted
Transformers usually scale by making their blocks wider or stacking them deeper.
We show another path toward scaling LLMs: branching.
Introducing Multi-Branch Transformer (MBT): parallel composition of full Transformer blocks.
TL;DR:
• Better scaling: across a controlled pretraining scaling ladder, MBTs improve language-model quality at comparable training compute. Fitted curves indicate a standard Transformer needs up to 1.26× as much training compute to match our final MBTs' validation loss.
• Consistent pattern: controlled sweeps show the same behavior at every scale. With two branches, loss is lowest with ~1/3 of the blocks in parallel and only returns to the standard model's level near 80%.
• Faster decoding: with Joint-KV cache sharing and a branch-aware inference engine, up to 51.4% higher throughput on one H200 and up to 34% less KV-cache storage. More branches push the quality–speed Pareto frontier further out.
The architecture, the scaling results, and how we make it fast. 🧵
This work would not have been possible without my amazing collaborators. Huge shout-out to: @LingjieChen127 @ChunyuanDeng @zhiweixux @HongzhengChen @chrismdesa @jingruih @yingheng_wang @hanghangtong
It has been a real pleasure working with all of you ❤️.
3
722
Gaotang Li @ NeurIPS2026 ATL retweeted
Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: openathena.ai/blog/marin-535…
Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities.
In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code).
Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.













