Pinned Tweet
We present Compute Optimal Tokenization! 🔡
Common in LLM scaling works stick to one tokenizer, sweeping data/model size.
But what happens when we control the tokenizer’s compression rate (bytes/token)?
Here we sweep tokenizers, params, and data across compute budgets: [1/N]
Amazing results! A great case for byte-based and latent-tokenized models like H-Net!
Still, most LLM research and all known frontier LLMs use subwords.
Here is my list of crucial research directions pushing us toward non-subword frontier models:
1/5
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
RL is not my area, but there is some potential here for byte-based models (limiting the action space) and latent tokenization (adapting compute to prediction difficulty 👀 “Beyond the 80/20 Rule”).
4/5
Tomasz Limisiewicz retweeted
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
RL is not my area, but there is some potential here for byte-based models (limiting the action space) and latent tokenization (adapting compute to prediction difficulty 👀 “Beyond the 80/20 Rule”).
4/5
Tomasz Limisiewicz retweeted
Looking for multiple emergency reviewers for #ICLR2027 for papers related to medical AI, LLM-judges, and audio LLMs. Please DM me if you are eligible and interested!
Tomasz Limisiewicz retweeted
If you're at #EMNLP2026 you should stop by our ✨tokenization tutorial✨ happening in <2 hours, starting at 9AM CET!
Fun fact: I'm actually not at EMNLP... I'll be presenting remotely. But I'll be up from 1AM to 4AM PDT because that's how much I love tokenization 🫶😄
Headed to #EMNLP2026? Want to learn about the latest advances in tokenization research in LLMs?
Come check out our tutorial on October 28. We will cover everything from fundamentals and emerging approaches with plenty of hands on demos and exercises.
See you in Budapest!
Szia 👋🇭🇺
Happy to be in Budapest for EMNLP.
I'll be sharing few lessons learned on byte language models, latent tokenization at tomorrow’s Tokenization Tutorial (in P2 from 9 -12:30).
tokenization-tutorial.github…
Tomasz Limisiewicz retweeted
We will present a tutorial on tokenization on Wednesday: nitter.gq/shocheen/status/210241…
Headed to #EMNLP2026? Want to learn about the latest advances in tokenization research in LLMs?
Come check out our tutorial on October 28. We will cover everything from fundamentals and emerging approaches with plenty of hands on demos and exercises.
See you in Budapest!
Tomasz Limisiewicz retweeted
Today we published a technical report on Muse Glimmer, our 30B open-weight model built for agentic tasks on consumer hardware.
It runs at practical speeds on a Mac or PC with a single consumer GPU at 4-bit precision, and leads comparably-sized open models on tool use and multi-step planning.
Read about the model architecture, distillation recipe, quantization and more: go.meta.me/4Bcvxol
Tomasz Limisiewicz retweeted
They say you can’t be at two conferences simultaneously. Well, our paper MATT won the Best Poster Award 🏆 at @MLinPL in Warsaw, while I presented it at @tokshopMeeting (@COLM_conf) in San Francisco. Kudos to Piotr Ludynia for showing it to the Polish audience!
A year and plenty of work have passed since the original ICLR submission @vikhyatk posted about. Since then, MATT has been accepted to @NeurIPSConf, and now I'm heading to @COLM_conf to present it at the Tokenization Workshop.
Here's what Model-Aware Tokenizer Transfer does 🧵
Tomasz Limisiewicz retweeted
Extremely sad to miss this, but if you have a chance, go learn about a topic researchers conveniently tend to forget despite its impact
Headed to #EMNLP2026? Want to learn about the latest advances in tokenization research in LLMs?
Come check out our tutorial on October 28. We will cover everything from fundamentals and emerging approaches with plenty of hands on demos and exercises.
See you in Budapest!
There are bigger workshops out there, but @tokshopMeeting is the friendliest one!
Tomasz Limisiewicz retweeted
Next up, we have Alexandre Défossez @honualx, cofounder of Kyutai, who is talking about audio-text modeling and related tokenization challenges.
