
pretraining @openai | prev. @tilderesearch
San Francisco, CA
Joined February 2019
- Tweets83
- Following216
- Followers263
- Likes1.4K
Pinned Tweet
couple of new results that I think are pretty cool. One is that MLP under-utilization seems to occur in the dense layers of some strong open-weight MoE models.
I think the extent to which this happens is a function of things like init scale and effective learning rate for tall params, but I found row-uniform updates let you tune these settings arbitrarily. Aurora should generally give more robust updates with very little overhead
arxiv.org/abs/2606.27715
Alec Dewulf retweeted
(1/10) Today, we’re sharing new research from @OpenAI: How does CoT monitorability (no-CoT capability and CoT controllability) evolve over the model training lifecycle?
We emerged with unsurprising but sobering findings: as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT. 🧵
exponential time hypothesis disproved 🤯
We are announcing three new discoveries in mathematics, computer science, and physics about harmonic analysis, complexity theory, and quantum field theory.
openai.com/index/building-sc…
GPT-6.1 Terra: delivering fast and capable computer use at a lower price.
GPT-6.1 Terra understands what’s on screen, takes action across apps, writes naturally, and is a more capable model for your whole workday with the efficiency to take on more of it.
🤖 Made with AI
Alec Dewulf retweeted
People have escaped the TL and are asking me for resets on the street now. I’ll make little cards to hand out
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
Alec Dewulf retweeted
I used GPT-6 Astra to break an unsolved cipher to one of Napoleon's generals that had gone unread for 217 years.
What makes this impressive isn't actually the codebreaking, but that Astra completed the entire multi-modal workflow in ~6 hours from a single image and goal.
1/6🧵
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
Alec Dewulf retweeted
How our data shaped neural architecture discovery, and how automation can reshape the future
coreauto.com/blog/how-our-da…
Alec Dewulf retweeted
Newton-Schulz is what makes Muon work, but it's also a massive pain compared to Adam. *Dion3* makes it ~6x faster and more parallelizable too.
Combines our work on row-filtered gradients, Gram-NS alg, and GPU kernels in one package. Try pip install dion!
arxiv.org/abs/2608.11612
Alec Dewulf retweeted
New personal blog post: Attribution in the Optimizer's Geometry!
I explore using the natural metric induced by the optimizer for parameter-attribution. I find evidence that you should attribute with the geometry you train in. 🧵
Alec Dewulf retweeted
Can a model run deeper at test time than it was ever trained to? And if depth becomes a loop instead of a stack, do we need better optimizers to keep training stable?
@coreauto is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper"
We picked a cursed problem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases.
This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way.
The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank
There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation.
Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!
Alec Dewulf retweeted
Find ways to beat this benchmark that tries to understand the limits of deep learning architectures and optimization?
We are funding compute, there is a nice way to test it out. Its a single GPU problem.
Cracked team @tilderesearch x @coreauto collaboration.
onelayerdeeper.ai/
There were (and still are) kernel competitions.
Now we want to present something new: optimizer competition. Show your skills, and learn along the way optimizing through hard problems.
the best score so far is 0.06%... hoping to see some interesting looped model submissions :)
We wanted to create a competition for arch x opt co-design! The problem is *very hard*
Incredibly excited to collaborate on this with the amazing folks at @coreauto
Give it a try :)
7
1,531
Alec Dewulf retweeted
onelayerdeeper.ai/
There were (and still are) kernel competitions.
Now we want to present something new: optimizer competition. Show your skills, and learn along the way optimizing through hard problems.
Alec Dewulf retweeted
hybrid linear attn is scalable!!
Introducing Kimi K3: Open Frontier Intelligence
🔹 2.8 Trillion Parameters, 1 Million Context, Native Multimodal
🔹 Kimi Delta Attention enables up to 6.3x faster decoding in million-token contexts
🔹 Attention Residuals deliver ~25% higher training efficiency at <2% additional cost
🔹 Built for long-horizon agentic coding and self-evolving workflows
Kimi K3 is now live on on Kimi.com, Kimi Work, Kimi Code, and the Kimi API.
Open Weights by July 27, 2026.
🔗 API: platform.kimi.ai
🔗 Tech blog: kimi.com/blog/kimi-k3
big congrats to the Kimi team! really cool to see them scale kimi delta attention and attention residuals. This is very likely the largest hybrid-attention model ever trained, including closed-source
Introducing Kimi K3: Open Frontier Intelligence
🔹 2.8 Trillion Parameters, 1 Million Context, Native Multimodal
🔹 Kimi Delta Attention enables up to 6.3x faster decoding in million-token contexts
🔹 Attention Residuals deliver ~25% higher training efficiency at <2% additional cost
🔹 Built for long-horizon agentic coding and self-evolving workflows
Kimi K3 is now live on on Kimi.com, Kimi Work, Kimi Code, and the Kimi API.
Open Weights by July 27, 2026.
🔗 API: platform.kimi.ai
🔗 Tech blog: kimi.com/blog/kimi-k3





















