@FSchaipp

training llms @ cohere. previously @ tum and inria.

 Paris, France
fabian-sp.github.io/
Joined July 2020
  • Tweets564
  • Following779
  • Followers1.4K
106Media
Learning rate schedules seem mysterious? Turns out that their behaviour can be described with a bound from *convex, nonsmooth* optimization. Short thread on our latest paper 🚇 arxiv.org/abs/2501.18965

The Surprising Agreement Between Convex Optimization Theory and...

We show that learning-rate schedules for large model training behave surprisingly similar to a performance bound from non-smooth convex optimization theory. We provide a bound for the constant...

arxiv.org
Aaron Defazio
@aaron_defazio3 Feb 2025
The sudden loss drop when annealing the learning rate at the end of a WSD (warmup-stable-decay) schedule can be explained without relying on non-convexity or even smoothness, a new paper shows that it can be precisely predicted by theory in the convex, non-smooth setting! 1/2
5
27
2
142
33,468
new paper (and last one from my postdoc): arxiv.org/abs/2610.16518 we analyze WSD schedules for least-squares problems. this is excellent work from Noé, who lead this paper during his internship at Inria, together with @BachFrancis .
2
11
103
5,252
this is a follow-up to our paper from 2025 using convex nonsmooth theory. the least-squares setting allows to get exact (!) risks, and to model other factors such as noise/batch sizes. (caveat: the maths become much harder)
1
1
3
411
overall, many results also hold in the least-squares case: - linear-decay is best among WSD - LR cooldown is always beneficial (by a polynomial factor here) - the base LR can be reliably transfered across cooldown lengths
1
3
339
here we go again
2
2
24
4,145
interesting read: What's the matter with France? paulkrugman.substack.com/p/w…
2
1
15
5,888
what is your p( ai researchers soon will be the successors of post-2008 bankers )?
1
7
1,609
TIL that Higgs and Hinton have the same PhD advisor
1
8
797
Personal update: I am excited to join @cohere where I will be working on pretraining from the Paris office. This also means that my postdoc at Inria comes to an end. I had a great time in the lab, with awesome colleagues, and lots of learning.
15
3
163
8,602
Particular thanks to @BachFrancis, @umutsimsekli, @TaylorAdrien for giving me the freedom and space to do research, and for supporting whenever needed.
6
586
Fabian Schaipp retweeted
Fabian Schaipp@FSchaippJul 6
Why would you want to do this? 1) With our law, you get the correct scaling for the optimal batch size with two/three batch sizes per sweep 2) You can also derive scaling laws for *suboptimal* batch size intervals (e.g. which batch sizes waste at most 5% of compute)
1
2
5
629
never seen a place that takes coffee culture as seriously as Seoul
15
604
very nice paper finding that Muon & SOAP improve training of models for molecular simulations #ICML2026
2
27
2,123
Poster today at 5pm! #ICML2026
Fabian Schaipp@FSchaippMay 27
"It's easier to tune the LR for method A than for B." We tried to formalize this for model-based stochastic optimization methods. We find a key quantity, called stability index, that describes how stable a (weakly) convex bound is as a function of LR. 📚arxiv.org/abs/2602.09842
3
1
24
2,932
Fabian Schaipp retweeted
Josephine Parquet
@josephinePqt1Jul 8
🎬 Field Notes: is more data always better? Physical AI is booming, and so are the data providers feeding it. But what actually makes a robot smarter? Awesome to chat with @fabinsch1, researcher at @inria who just presented his thesis in policy learning for robotics. 🧵
4
4
1
28
3,346
How to scale the batch size in LLM pretraining? New paper on a scaling law that splits token budget D into training steps and batch size. 💺 arxiv.org/abs/2607.01487
4
14
102
7,861
more replies
Why would you want to do this? 1) With our law, you get the correct scaling for the optimal batch size with two/three batch sizes per sweep 2) You can also derive scaling laws for *suboptimal* batch size intervals (e.g. which batch sizes waste at most 5% of compute)
1
2
5
629
I will be at ICML from Wednesday, happy to chat on this! 🇰🇷
1
6
235
awesome blog post! the Apertus project keeps on pushing optimization research for open LLMs.
Alex Hägele@haeggeeJun 15
New research from our MLO Lab @EPFL: Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors. Magnitude-Direction Decoupling (MD): a simple optimizer tweak, and what we (currently) think is the right way to train efficiently at scale. 🧵
5
26
3,969
Fabian Schaipp retweeted
Antonio Orvieto
@orvieto_antonioJun 17
Judging optimizer gaps by looking only at language modeling with a fixed batch size is dangerous: one gets only 1/2 of the story. @orientino_ and @ruuustem_10 went beyond. Turns out that for every model, task, and data, there is always a setup where Adam > SGD. 🧵
2
15
3
74
10,626
Load more