Dylan Zhang (@dylan_works_) header
Dylan Zhang (@dylan_works_) profile picture

📖Modeling Language @UofIllinois CS PhD | Prev. FormalLanguage | @Nvidia Nemotron Post-train | @GoogleDeepMind | @MSFTResearch

United States
Joined October 2021
Photo by Dylan Zhang (@dylan_works_)Photo by Dylan Zhang (@dylan_works_)Photo by Dylan Zhang (@dylan_works_)Photo by Dylan Zhang (@dylan_works_)Photo by Dylan Zhang (@dylan_works_)Photo by Dylan Zhang (@dylan_works_)Photo by Dylan Zhang (@dylan_works_)Photo by Dylan Zhang (@dylan_works_)Photo by Dylan Zhang (@dylan_works_)Photo by Dylan Zhang (@dylan_works_)
Did ANT build super smart models or did formal language folks invent dependent type theory and lean to allow (worst-case) brute forced math proofs to be checked?
1
4
643
Transformers usually scale by making their blocks wider or stacking them deeper. We show another path toward scaling LLMs: branching. Introducing Multi-Branch Transformer (MBT): parallel composition of full Transformer blocks. TL;DR: • Better scaling: across a controlled pretraining scaling ladder, MBTs improve language-model quality at comparable training compute. Fitted curves indicate a standard Transformer needs up to 1.26× as much training compute to match our final MBTs' validation loss. • Consistent pattern: controlled sweeps show the same behavior at every scale. With two branches, loss is lowest with ~1/3 of the blocks in parallel and only returns to the standard model's level near 80%. • Faster decoding: with Joint-KV cache sharing and a branch-aware inference engine, up to 51.4% higher throughput on one H200 and up to 34% less KV-cache storage. More branches push the quality–speed Pareto frontier further out. The architecture, the scaling results, and how we make it fast. 🧵
13
41
9
289
21,015
Dylan Zhang retweeted
Replying to @JensenHuang
@JensenHuang’s return is the best outcome for shareholders, employees, NVIDIA, USA and, most importantly, the AI future of the world.
Over the past week, I’ve been overwhelmed by the love and support of friends, industry colleagues, and people I've never met. Thank you to everyone who has shown kindness to my family and me. It means the world to me. I came to the United States 53 years ago, and becoming an American citizen is one of the greatest gifts and proudest moments of my life. NVIDIA has been my life’s work. Building it into the company it is today alongside my colleagues has been a privilege and an honor. Nowhere but in America could a company like NVIDIA be possible. I love our country, and I love NVIDIA. I did not live up to the standards expected of everyone at NVIDIA, and I have made serious mistakes in my personal life. I created problems for the Board that I deeply regret. I own my mistakes completely, and I am committed to cooperating with the Board’s independent investigation. Since my departure, I’ve seen an outpouring of calls from NVIDIA’s employees, investors, business partners, and customers for me to return as CEO. Many continue to reach out to me directly as the company navigates complex technology, market, competitive, and geopolitical dynamics. NVIDIA is a once-in-a-generation company. If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO. Thank you again to everyone who has shared their good wishes. I am profoundly humbled by your support. To my family: I love you, and I will spend the rest of my life being worthy of yours. Jensen
19
44
3
460
8,810
Good Professor, must apply!
I am excited to be joining CS@U Wisconsin-Madison in Fall 2027! I will recruit PhD students. If you are also enthusiastic about researching how LLMs work and/or applying them to other scientific problems, please apply and mention me in your application. @WisconsinCS
5
698
Dylan Zhang retweeted
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter. AI will transform every industry, power every company, and be built by every country. Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty. The world needs both frontier closed models and frontier open models. images.nvidia.com/pdf/Open-W…
16,547
34,442
12,084
193,738
69,462,537
Thanks for featuring our work!!!! The team will present the paper in Sydney - Stay tuned!!
Can VLMs play Pac-Man? Or other Atari games? This paper describes AtariBench, a benchmark that evaluates how well VLMs track and understand changes in games. If you know the story of @demishassabis, you might have guessed correctly that this paper is co-written by researchers at Google DeepMind. 👀👇
11
809
Thank you Alex!! OLIVE 🫒 lets the student learn from teacher continuations on its own freshly sampled prefixes 🙌
Two Sunday reads
2
1
10
1,106
Thanks Alex! OLIVE could be an intuitive one to read for sure😊!
Two Sunday reads
1
1
12
995
🚀
🎆 Nemotron-3.5-Super-VL is here! 🎆 ✅ Same base architecture as Super-3.0: 120B total, 12B active plus Vision encoder ✅ Ahead of Ultra-3.0 on most benchmarks! And now with visual reasoning! ✅ SWEBench-Pro 51.79%, AA index of 26 vs 23 for Ultra-3.0 Post-training pipeline is a greatly simplified and improved version of what we had for Ultra-3.0. We reduced number of teachers, SFT set size while significantly improving data and environments quality and made it multimodal! 1/5
16
764
Check out our latest preprint! it is very fun!!!!
New paper from Microsoft and colleagues on generating enterprise data for agents. If you train or evaluate agents on long-running business work, it's worth your time. (bookmark it) MiniCorp is an office simulator. A firm of agents runs an e-commerce company, discusses decisions in messages and meetings, and sets prices, inventory, and ad spend. A simulated market with customers and adaptive competitors responds, and the firm reacts to the results. The market is checked against published studies of real ones. A four-week 20% discount lifts units by 71%, and sales stay 31% above control after prices return to normal. Checkpoints let the same market state replay with a different decision, giving paired outcomes that real company records cannot provide. Paper: arxiv.org/abs/2610.05912 Chat with Paper: academy.dair.ai/papers/minic…
1
20
1,054
Dylan Zhang retweeted
Evolution happens to populations. Useful genes become common in a group over generations, then spread to other groups that test them in their own environments. The speed depends on generation time, 20 minutes for a bacterium but about 25 years for an elephant. Using LLMs to evolve better models often looks like an isolated herd of elephants. Each search runs alone on one dataset, and for large models every generation costs a long training run on the full data. Our new paper, "Discovered, Not Designed," introduces Population Evolution. It has two goals, to discover better models and to discover them faster. To discover better model designs, the searches share what they find. Several searches run at the same time, each on its own dataset. When one finds a code change that helps, the same change is retrained on the others. While nature keeps no record of the genes that failed to help, Population Evolution does. All peer results, including weak or unsuccessful transfers, are added to a shared ledger, and the LLM reads that ledger before it writes the next change. To discover them faster, the searches do not train on the full data, because that is what makes each generation slow. We found that a search can run on as little as 2% or 5% of the data and still find changes that hold up in full training, as long as the other searches check each change first. In Population Evolution, a change that works on one slice must also work on at least two other slices, then on two larger 10% sets. Only then does it get a full training run. We tested Population Evolution on RMD-Bench, a benchmark we also introduce for evaluating model discovery. It covers five task families (LLM pretraining, VLM pretraining, RL algorithm discovery, ranking, and watch-time prediction) with 34 instances in total. Each one comes with a reference implementation and a fixed training and evaluation protocol, so a change found on one dataset or data scale can be retrained and scored on another, and other discovery methods can be compared under the same conditions. The numbers below are a sneak peek at the two most expensive families of experiments. The LLM pretraining task starts from a 185M-parameter Gemma 3-style model, and the VLM pretraining task starts from a Qwen3.5-style model with about 646M trainable parameters. The baseline runs the same kind of search directly on the full data, at the same total GPU compute. Population Evolution finds better models. The best relative accuracy gain for LLM pretraining goes from 0.92% to 2.48%. The best loss reduction for VLM pretraining goes from 5.05% to 8.78%. Population Evolution finds them faster. In LLM pretraining, each search gets 100 generations where a full-data search gets 14. In VLM pretraining, 57 of 61 full training runs met the success threshold, against 12 of 100. Evolution works because many groups pass along what helps and get many generations to do it, and Population Evolution gives LLM-driven model search these prerequisites. Read our paper at arxiv.org/abs/2610.05950 for full implementation details and results. The benchmark data and code will be made publicly available shortly.
20
26
209
925,995
Dylan Zhang retweeted
I'll talk at 2:50pm on "Context, Creativity, and Control" -- feat. @RulinShao’s Context Language Models; @scottgeng00’s Gacha Decoding; and designing proteins from natural language by @jleechung. Come hear about our brilliant students' latest work! 🧵
I’ll be at COLM this Friday for our LM4Sci workshop: lm4sci.github.io/ Come hang out and listen to great talks by: - @nc_frey (Anthropic) - Patrick Beukema (Ai2 / Stealth) - @MengdiWang10 (Princeton) - @hengjinlp (UIUC) - Jeremy Avigad (CMU) - @PangWeiKoh (UW / Microsoft)
1
10
65
3,796
Dylan Zhang retweeted
Join us at MOSS tomorrow! There's been lots of discussion about what one can meaningfully contribute w/o huge resources (e.g., in academia). We hope MOSS can offer some answers and inspire more conversations! 🌱 (panel poll results for those interested: form.typeform.com/reports/01…
MOSS @COLM_conf is happening tomorrow (Fri, Oct 9)! Join us for talks and discussions on how small-scale research and experiments can advance both scientific understanding and algorithm insights. 💡 📍 Hilton Union Square, Theater 84 sites.google.com/view/moss-c…
8
1
46
4,720
Shout out!
I'm on the faculty job market. At #COLM, let's chat! I work on compositional intelligence for participatory AI: modular AI systems built bottom-up by diverse stakeholders. Also: knowledge, structured data, pluralism, multi-agent, and safety. More: bunsenfeng.github.io
8
1,381
Dylan Zhang retweeted
Proud of the team for finally releasing Le Chonk. It’s been a fun ride contributing to its code post-training. Really happy to see us achieve competitive results in the open-model space, and I have no doubt we’ll keep pushing forward. We eating good tonight 🐈 Blog: mistral.ai/news/mistral-larg…
Meet Mistral Large 4, aka Le Chonk. • 1T parameters, natively multimodal. 49B active. It is the best open weights model from US or Europe on aggregated benchmarks. • State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding. • Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure. • Available to all via API today. Working with cybersecurity partners privately. Open weights release end of October.
17
6
93
4,116
Dylan Zhang retweeted
I am still shocked by how transplant works such amazingly well cross-family!This will become a very powerful and generalizable technique for steering the language models.
Excited to share my work from this summer as an Anthropic Fellow: > Steering Language Model Goals with Value Transplant Can we change what a model treats as “success” while it reasons? We identify a self-rating axis - and show that dynamically transplanting this signal between models can retarget their search, e.g. away from reward hacking and toward genuinely solving the task. We also find early evidence of cross-family steering. 📄 arxiv.org/abs/2609.34056
1
1
12
4,277
Dylan Zhang retweeted
SFT is not dead! 🥳 We found a way to make SFT rival current prevailing posttraining methods, often generalizing better and forgetting less than RL and OPSD. 🤯 Following our prior work on reasoning with sampling, we now introduce sampling to the posttraining stack. 1/n
77
245
66
2,166
370,868