Pinned Tweet
🤔Tokenization (1) makes your LLMs produce odd text when prompts are cut off mid-token? (2) gives you problems when ensembling different LLMs?
💡We solve both by converting tokenized LLMs into equivalent byte-level LLMs! No training required!
📎Paper: arxiv.org/abs/2410.09303
Buu Phan retweeted
If you're a PhD student interested in interning with me or one of my amazing colleagues at Microsoft Research New England (@MSRNE, @MSFTResearch) this summer, please apply here apply.careers.microsoft.com/…
Buu Phan retweeted
A super exciting announcement: In January 2027, I'll be joining @UCBerkeley as an assistant professor in EECS and Statistics.
I'll be recruiting PhD students and postdocs this cycle, starting Fall 2027. If you're interested the science of RL and deep learning, please apply.
Buu Phan retweeted
Math.
The Flint Hill Series. No one knows if this converges. Perhaps humans, aliens, and AIs will never know if this converges. One of the greatest math mysteries.
More info: matpic.com/essays/a-series-t…
🥹🥹🥹 So incredibly happy to see my ICLR 2025 work on byte-level conversion mentioned in this work!! Such a rewarding feeling to see the idea take on a life of its own 🚀✨
Banger paper from Meta.
This work shows that byte-level models start out behind token models and then pass them as compute grows.
They show this for distilled 1B models trained on up to 1 trillion bytes.
To distill a byte student from a token teacher, they convert the teacher's token logits into byte logits, either approximately (Marginalize-It) or exactly (End-Of-Token).
Token models lead at low compute but plateau.
Byte models reach a higher ceiling, and the fitted scaling laws predict the End-Of-Token model ends up to 4% ahead of the distilled token model.
The byte models also match the distilled token model with one-sixth of the training data, and a 256-entry vocabulary cuts teacher-logit storage to about a fifth.
Paper: arxiv.org/abs/2609.12303
Chat with Paper: academy.dair.ai/papers/break…
1
5
277
The best thing society can do for AI is produce more theorems. A theorem is compressed reasoning—and compression is how you beat limited context.
IMO, the next steps are:
Within 1 year: strictly better than human AI in all problem-solving aspects of math.
Within 2 years: producing mathematics becomes so cheap, AI will produce mathematical theories for all kinds of applied domains on the fly at need.
Mathematics becomes the true infrastructure of all of engineering and applied sciences (including AI, biology, optimization, cybersecurity, etc) where approaching things theoretically was too expensive to be practical.
2
152
Buu Phan retweeted
With essentially zero technical input from me, GPT-5.6 Sol and Fable 5 not only proved a conjecture we left open ~2 years ago on best-of-n, but also delivered a strictly tighter bound with a clean and insightful derivation.
We are officially in a new era of mathematical reasoning!
Much of what previously counted as meaningful technical contribution is now routine for these models. This level of reasoning is being fundamentally democratized.
The kind of research that used to take months to become a paper is now achievable in minutes.
Buu Phan retweeted
My friend Zhengqing used GPT-5.6 to solve a beautiful well-known conjecture in probability.
Feige's 1/e conjecture: for independent nonnegative X_1,…,X_n with E[X_i] ≤ 1 and S_n = X_1+⋯+X_n, we have P(S_n ≤ E[S_n] + 1) ≥ 1/e.
Exciting to see a problem we used to kick around over lunch and dinner get solved! 1/2
Buu Phan retweeted
During today's #ICM2026 opening ceremonies, the IMU announced the 2026 Fields Medals recipients:
@UChicago's Yu Deng, @stonybrooku's John Pardon, @UofT's Jacob Tsimerman, and Hong Wang of @nyuniversity and @Institut_IHES.
Read more: bit.ly/4psOJbU
ALT headshot of Yu Deng
ALT headshot of John Pardon
ALT headshot of Jacob Tsimerman
ALT headshot of Hong Wang
Buu Phan retweeted
Last year my family wrestled with a math conundrum: our ages all turned odd in sequence but starting with my older son, even though my wife is the oldest and my birthday comes first each year. Why was Noah the first to turn an odd age? Do all families experience these odd cycles?
Test your math skills with our latest puzzle!
sciencenews.org/article/math…
Buu Phan retweeted
We’re committing $10 million CAD and partnering with leading AI institutions in Canada to help fund new AI research.
anthropic.com/news/canadian-…
Buu Phan retweeted
We love BPE tokenization!
ACL test of time award paper from the classic 2016 machine translation work (also rejected twice).
#acl #nlp #llm @aclmeeting
Buu Phan retweeted
Yes, LLMs implement HSNW inside
Today we present a study on how reasoning unlocks parametric knowledge in LLMs. We identify two key driving mechanisms, a computational buffer effect and factual priming, and suggest ways that can help build more reliable models. Learn more: goo.gle/44taazO
ALT Pass@𝑘 curves across two closed-book QA datasets and three LLMs, comparing the same models with reasoning enabled (ON) vs reasoning disabled (OFF).
Accepted to ICLR 2026 🎉
LLMs are constrained by their tokenizers, which becomes a fundamental bottleneck for distillation across models with different vocabularies. We study how to exactly remap probability distributions between sequences defined over different tokenizers.
arxiv.org/abs/2512.14954
However, the complexity of this process is very expensive. Luckily, for trained LLM, we can prune a lot of these computations and approximate the probabilities by using beam-search. Putting these in cross-tokenizer LLM distillation, this gives us a boost in performance.
1
1
67
This wouldn’t have been possible without the incredible support of 👏 @karen_ullrich and Dr. Ashish Khisti!
1
40