Jiaxuan Zou

Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: openathena.ai/blog/marin-535… Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities. In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code). Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.

GPT-4 level intelligence has gotten roughly 1800x smaller

Google is on a roll and about to surpass OpenAI and Anthropic, let me explain. Google released bard, trash. Google went a few years with their pro models and they was decent but didnt match the competition. Roughly a month ago google released argon. Around the level of 6 Astra and below opus 5.5 A week later they announced carbon, carbon is slightly better than opus 5.5. 2 weeks later they have a internal model “krypton” it’s better than all public models will be slightly better than 6.2 Astra and fable 5.5. Google catched up and surpassed anthropic and open ai. and it doesn’t look like they’re slowing down.

why ema is widely used in generative models while language models (almost) all use learning rate decay in practice?

Nice work! The most interesting part to me: the extra loss from repeating data comes mostly from rare tokens. Their embedding rows see the same few contexts every epoch and end up memorizing them. This points to a tension. A big vocab is great while data is unique: more text per FLOP, and rare rows don't have time to memorize. The scaling laws with vocabulary paper finds the optimal vocab grows with compute but shrinks when data is the bottleneck, "a mechanism to prevent over-fitting" (arxiv.org/abs/2407.13623). Once data starts repeating, though, those same rare rows become the problem. So my hypothesis: the vocab should shrink as data starts repeating, within a single run. I can imagine two ways to get there ⬇️. Path 1: a shifting mix of vocab sizes Proxy Compression (arxiv.org/abs/2602.04289) trains one model on both BPE tokens and raw bytes of the same text, and finds that it learns to align the two views and transfer between them. The same idea may work across vocab sizes: tokenize each sequence at one of several nested vocabs (128K, 64K, ..., just fewer BPE merges), mostly 128K while data is unique, then shift the mix toward smaller vocabs as data repeats. What I like about it: a repeated document comes back as a different token sequence, so it's closer to a fresh view than a pure repeat. And since " memor" is literally " mem" + "or", with both pieces trained all along, what the big rows learned can flow into the smaller pieces before those rows go cold. Nothing switches abruptly; the model just sees more of views it already knows. Same spirit as unified training (nitter.gq/yus167/status/21161813…): one evolving mix instead of hard stages. Path 2: cutting the vocab gradually Opus 4.7 moved to a much smaller tokenizer (reconstructions suggest about a third of the old vocab), even at ~30% more tokens per text. That makes a small final vocab look worth paying for. Instead of switching in one shot, you could shrink as you train: each time the epoch count doubles, drop the latest BPE merges, which are roughly the rarest tokens. This one is simpler: one tokenizer at a time, and each cut removes exactly the rows that are starting to memorize. Keeping the first K merges is the same as stopping BPE at K, so every new tokenizer nests inside the previous one and the cuts stay clean. I think Path 1 should transfer more smoothly but spends extra FLOPs on smaller-vocab views. Path 2 is cheaper and simpler, but each cut is a small "surprise" to the model. Curious where this breaks. Anyone tried something like this? > All of the above is my unverified intuition based on public work, and isn't related to anything I'm working on

We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.

Carbon!

From Parameter Symmetry to Effective Learning Rates kexue.fm/archives/11985