MTS @ Datology | @ICMEStanford PhD | @dartmouth
San Francisco, CA
Joined April 2021
- Tweets66
- Following388
- Followers172
- Likes953
A backtest-able model with real modern synthetic data is huge alpha
Kaleigh Mentzer retweeted
When a model rephrases documents the knowledge cutoff guarantees are lost. It can introduce information its sources never had. This is catastrophic for backtesting. So we built a way to generate synthetic pretraining data with measurable, controllable knowledge cutoff guarantees
Kaleigh Mentzer retweeted
Tomorrow I'll be on Datology's Curation Studio webinar discussing what went into measuring our 6x compute multiplier.
Come ask about scaling ladders, baselines, and quantifying data quality!
10am PT w/ @arimorcos @hurrycane @josh_wills: events.datologyai.com/curati…
guaranteed to be a good watch
Data quality is the ultimate compute multiplier.
Curated data matched our largest baseline with 6x less compute. A 12B-A1.4B model beat 30B-A3B on a fifth of the pretraining compute.
Live Tuesday 10am PT with @hurrycane @lukemerrick_ @josh_wills: events.datologyai.com/curati…
As a researcher, I cannot understate how much easier this determinism makes my life. No more mystery "maybe it's just distributed computing?? 🤷♀️" results from my dataloader
Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count.
In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run.
Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus.
The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count.
✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai)
Read more: pytorch.org/blog/bringing-el…
Kaleigh Mentzer retweeted
Applications are open for the January cohort of the Datology Labs Fellows Program.
Funding, compute, and mentorship to work on the science of data.
Paid · One semester · San Mateo/NYC
Apply by Nov 15: labs.datologyai.com/fellowsh…
Fresh ideas, mentorship, compute, engineering support - what more could a researcher want!
Kaleigh Mentzer retweeted
"Not all data is equally valuable" - often yapped about, but Datology have done the work to quantify that and put it into in practice
Kaleigh Mentzer retweeted
Tbh I just found out that Thomson-1 beats GPT 5.6 Sol
That is insane!
That’s our curation is allowing a 400B model to beat the best model in the world at the time on it’s specialized domain: legal
Owning your own intelligence is more critical than ever, but harnessing your data to build a model that can beat the frontier has required deep data curation expertise.
We built @datologyai Curation Studio to change that. Curation Studio enables teams of all sophistication levels to customize curation to their use cases, delivering a massive compute multiplier that fundamentally changes the economics of model ownership.
We applied Curation Studio to the best publicly curated datasets from Nvidia, HuggingFace and others, and were able to deliver a 6x compute multiplier out of the box, all through better use of existing data.
This has allowed us to deliver huge wins for our customers. For example, we worked with @thomsonreuters to help them build Thomson-1 for only $450K in compute, which outperformed GPT 5.6 Sol in head-to-head comparisons and did so for 100x lower cost per task.
Curation Studio brings those capabilities to everyone.
This is a big moment for us at Datology, but it's just the beginning. We'll be expanding Curation Studio support to more modalities and verticals. And stay tuned for more research releases coming very soon!
Kaleigh Mentzer retweeted
We believe intelligence should be democratized, not just controlled by one or two labs on the planet, and we're getting there through data.
Kaleigh Mentzer retweeted
Wow this is awesome! If you train & own your own models you guarantee information security and IP priority - while cutting token spend
You get to own your alpha!
@thomsonreuters got frontier legal performance - for ~$500k in compute!?!
Which pharma is first? @EliLillyandCo?
Kaleigh Mentzer retweeted
A data loader is only as useful as the data it loads. This is why this week is so exciting: we're sharing our Curation Studio and how we use it to generate the best tokens (or raw strings to be tokenized and packed on-the-fly) possible. Check out our release announcement!
This result is super impressive ESPECIALLY considering the "baseline" here is crazy sophisticated
Kaleigh Mentzer retweeted
Look at him, that’s my 1000x Engineer
Your what?
My 1000x Engineer! My throughput whisperer. Look at him, you notice anything different about him? Look at his face
I’ll give you a hint, his name is Josh. He put H100 in his name before it was cool. Yeah, I’m sure Zephon scales.
Data Loader (n.): The part of your training stack that is too ML-y for the systems folks and too systems-y for the ML folks.
It's the work I love best, and I had a ton of fun building a new one with the team here at @DatologyAI. My three favorite things about Zephon:
Kaleigh Mentzer retweeted
Data Loader (n.): The part of your training stack that is too ML-y for the systems folks and too systems-y for the ML folks.
It's the work I love best, and I had a ton of fun building a new one with the team here at @DatologyAI. My three favorite things about Zephon:
At @datologyai we run thousands of data experiments. Two things keep biting us:
- Same run on 16 vs 8 GPUs → different evals
- Every pipeline change → re-materialize the dataset
Introducing Zephon, the data loader we built to process data on the fly, deterministically. 🧵
Can confirm this works really well - amazing work @MaxiBoether and team!
At @datologyai we run thousands of data experiments. Two things keep biting us:
- Same run on 16 vs 8 GPUs → different evals
- Every pipeline change → re-materialize the dataset
Introducing Zephon, the data loader we built to process data on the fly, deterministically. 🧵
Kaleigh Mentzer retweeted
We sat down with the research teams from Thomson Reuters and DatologyAI to break down how they built the Thomson-1 model. We get into the data, the mid-training, and what it takes to build a frontier model on open weights.
If you're building your own model, this one's worth your time. With @schwarzjn_, @yegin_bang, @szcjerry, @arimorcos, @amrokamal1997, @KaleighMentzer, and @josh_wills
Full podcast in the link below.
Hmm looks like data is pretty important
Pretraining progress seems to be coming mostly from data improvements.
@who_is_jerbear and I pretrained combinations of year-representative open model recipes and data corpuses across 2019 to 2025 at various small scales.
Data improvements contributed 3.24x as many compute multipliers as model improvements did (12.0x vs 3.7x).
And the gains stack independently - a better dataset helps every architecture about equally, and vice versa.
Here are full results, plus what we think this means for the future of AI progress:
dwarkesh.com/p/pretraining-p…
