Alec Dewulf (@AlecDewulf) header
Alec Dewulf (@AlecDewulf) profile picture

pretraining @openai | prev. @tilderesearch

San Francisco, CA
Joined February 2019
Photo by Alec Dewulf (@AlecDewulf)Photo by Alec Dewulf (@AlecDewulf)
couple of new results that I think are pretty cool. One is that MLP under-utilization seems to occur in the dense layers of some strong open-weight MoE models. I think the extent to which this happens is a function of things like init scale and effective learning rate for tall params, but I found row-uniform updates let you tune these settings arbitrarily. Aurora should generally give more robust updates with very little overhead arxiv.org/abs/2606.27715
2
24
2,779
Alec Dewulf retweeted
Some of you have been asking very politely. Some of you have just been asking. GPT-6.2 Astra and Sol are now available. Super efficient and built to perform where it counts—the real world. openai.com/index/introducing…
2,297
3,614
3,733
36,052
13,484,435
Alec Dewulf retweeted
(1/10) Today, we’re sharing new research from @OpenAI: How does CoT monitorability (no-CoT capability and CoT controllability) evolve over the model training lifecycle? We emerged with unsurprising but sobering findings: as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT. 🧵
125
189
77
1,479
277,330
Alec Dewulf retweeted
GPT-6.1 Terra: delivering fast and capable computer use at a lower price. GPT-6.1 Terra understands what’s on screen, takes action across apps, writes naturally, and is a more capable model for your whole workday with the efficiency to take on more of it.
🤖 Made with AI
2,238
2,550
2,339
28,792
8,375,979
Alec Dewulf retweeted
People have escaped the TL and are asking me for resets on the street now. I’ll make little cards to hand out
1,190
226
167
11,495
1,270,075
Alec Dewulf retweeted
We’re releasing a broad range of new mathematical results produced by an internal frontier model. We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results. github.com/openai/math
1,895
6,272
4,764
40,539
22,849,472
Alec Dewulf retweeted
I used GPT-6 Astra to break an unsolved cipher to one of Napoleon's generals that had gone unread for 217 years. What makes this impressive isn't actually the codebreaking, but that Astra completed the entire multi-modal workflow in ~6 hours from a single image and goal. 1/6🧵
315
1,661
387
11,941
5,422,323
Alec Dewulf retweeted
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics. The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra. The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
5,635
19,967
14,813
119,632
75,477,197
Alec Dewulf retweeted
This is GPT-6 Astra. Anything you can do on a computer, Astra can do for you. Fast.
9,132
35,056
26,632
337,619
139,550,852
Alec Dewulf retweeted
Newton-Schulz is what makes Muon work, but it's also a massive pain compared to Adam. *Dion3* makes it ~6x faster and more parallelizable too. Combines our work on row-filtered gradients, Gram-NS alg, and GPU kernels in one package. Try pip install dion! arxiv.org/abs/2608.11612
11
39
5
362
34,758
🍿🍿
We ran 1,373,020 benchmarks across 222 open ML kernel implementations. The result is Popcorn: the world’s largest collection of verified ML kernels. Today, we’re releasing it to everyone.
7
1,237
Alec Dewulf retweeted
New personal blog post: Attribution in the Optimizer's Geometry! I explore using the natural metric induced by the optimizer for parameter-attribution. I find evidence that you should attribute with the geometry you train in. 🧵
2
6
68
4,473
Alec Dewulf retweeted
Can a model run deeper at test time than it was ever trained to? And if depth becomes a loop instead of a stack, do we need better optimizers to keep training stable? @coreauto is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper" We picked a cursed problem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases. This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way. The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation. Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!
14
57
10
492
78,588
Alec Dewulf retweeted
Find ways to beat this benchmark that tries to understand the limits of deep learning architectures and optimization? We are funding compute, there is a nice way to test it out. Its a single GPU problem. Cracked team @tilderesearch x @coreauto collaboration.
onelayerdeeper.ai/ There were (and still are) kernel competitions. Now we want to present something new: optimizer competition. Show your skills, and learn along the way optimizing through hard problems.
7
10
1
112
15,440
the best score so far is 0.06%... hoping to see some interesting looped model submissions :)
We wanted to create a competition for arch x opt co-design! The problem is *very hard* Incredibly excited to collaborate on this with the amazing folks at @coreauto Give it a try :)
7
1,531
Alec Dewulf retweeted
onelayerdeeper.ai/ There were (and still are) kernel competitions. Now we want to present something new: optimizer competition. Show your skills, and learn along the way optimizing through hard problems.
3
12
3
173
43,097
Alec Dewulf retweeted
hybrid linear attn is scalable!!
Introducing Kimi K3: Open Frontier Intelligence 🔹 2.8 Trillion Parameters, 1 Million Context, Native Multimodal 🔹 Kimi Delta Attention enables up to 6.3x faster decoding in million-token contexts 🔹 Attention Residuals deliver ~25% higher training efficiency at <2% additional cost 🔹 Built for long-horizon agentic coding and self-evolving workflows Kimi K3 is now live on on Kimi.com, Kimi Work, Kimi Code, and the Kimi API. Open Weights by July 27, 2026. 🔗 API: platform.kimi.ai 🔗 Tech blog: kimi.com/blog/kimi-k3
9
28
1
431
25,128
big congrats to the Kimi team! really cool to see them scale kimi delta attention and attention residuals. This is very likely the largest hybrid-attention model ever trained, including closed-source
Introducing Kimi K3: Open Frontier Intelligence 🔹 2.8 Trillion Parameters, 1 Million Context, Native Multimodal 🔹 Kimi Delta Attention enables up to 6.3x faster decoding in million-token contexts 🔹 Attention Residuals deliver ~25% higher training efficiency at <2% additional cost 🔹 Built for long-horizon agentic coding and self-evolving workflows Kimi K3 is now live on on Kimi.com, Kimi Work, Kimi Code, and the Kimi API. Open Weights by July 27, 2026. 🔗 API: platform.kimi.ai 🔗 Tech blog: kimi.com/blog/kimi-k3
1
2
1
30
30,238