⬢ chaos dancing star @paradigmainc

Icecrown Citadel
Joined February 2022
just dropped the technical report for our first model
We are releasing the technical report for Limite 1B -Violetto. Violetto is our first model, designed to push mathematical capabilities per parameter to its limits, trained from scratch.
1
4
65
2,616
tensorqt in SF retweeted
Next couple days will be exciting for flywheel, we are cooking
We're releasing a number of updates to Flywheel in the coming days. First up, new ways to work on Flywheel graphs together: • Share a whole graph in one save • Ask for access from any link • Link straight to a graph or a node
1
18
552
bunch of updates coming to flywheel. we’ll post one per day, stay tuned
We're releasing a number of updates to Flywheel in the coming days. First up, new ways to work on Flywheel graphs together: • Share a whole graph in one save • Ask for access from any link • Link straight to a graph or a node
1
29
1,002
tensorqt in SF retweeted
We're releasing a number of updates to Flywheel in the coming days. First up, new ways to work on Flywheel graphs together: • Share a whole graph in one save • Ask for access from any link • Link straight to a graph or a node
2
9
2
43
2,937
we are seeing today the last days of Europe’s age of exploitation. we must enter a new age of exploration
2
29
994
tensorqt in SF retweeted
new pfp who dis
1
2
2
31
2,193
tbc, this was recorded before any millennium problem was solved using AI ;)
Can AI make the next great scientific discovery? Francesco Pappone (@tensorqt), co-founder of @paradigmainc , joins Europe on Edge to discuss AI scientists, the SPRIND Frontier AI Challenge, and Europe’s AI ambitions. 🎥 Trailer below 🎧 Full episode in the first reply
1
1
31
1,386
tensorqt in SF retweeted
Can AI make the next great scientific discovery? Francesco Pappone (@tensorqt), co-founder of @paradigmainc , joins Europe on Edge to discuss AI scientists, the SPRIND Frontier AI Challenge, and Europe’s AI ambitions. 🎥 Trailer below 🎧 Full episode in the first reply
1
4
1
11
1,617
tensorqt in SF retweeted
update: training without a backward pass is now faster than backprop. what I did: a data-free start (block 1’s heads begin as lag-1 RoPE heads), a tail average of the weights, skipping 4 blocks’ attention credit late in the run, then overlapped collectives and a compiled forward.
forward-only pretraining now matches backprop on the Noise benchmark. Relay: val 3.704 in 2.5 min on 8×H100. backprop with the same model, data and optimizer: 3.712. the current record, Boost→Cut: 3.945 in 7.6 min. no autograd anywhere. 🧵
12
32
4
285
15,075
we did this (mainly for lr) while training violetto (both during pre-training and post-training)
Excited to introduce reactive optimizers: We let an agent reactively adapt the optimizer code during the training run! The agent acts like a team of scientists constantly choosing the best algorithm, learning rate, momentum, etc, based on its own analysis of step-to-step training dynamics! The results are really exciting. Besides out-performing standard "best-practice" recipes across everything we threw at it, the agent gains a few "superpowers" that no human could have designed into the optimizer... (1/n)
4
1
47
2,235
adopting a sparsity mindset
4
28
1,070
science is big
5
5
66
2,220
tensorqt in SF retweeted
This next paper is a bit quirky, but here goes: Introspection is hard to study in both humans and LLMs. But, we do have internal senses that are verifiable, like proprioception (the sense of where your arm is right now). What would proprioception look like in language models? As a digital analogue to the sense of position, a model could verbalize which expert activated in a certain layer for any previous token. It turns out that LLMs can learn a range of such proprioception skills where they understand and report on their internals. We can calibrate and score these reports, and evaluate whether training for proprioception changes model behaviors.
14
69
10
468
22,313
tensorqt in SF retweeted
we got backprop's loss at ~4× its time, and 3× faster than the record. this was made possible by using @paradigmainc's Flywheel infra from the get go, preventing rabbit holes and encouraging branching and backtracking.
1
4
48
3,118
fun thing about Emanuele is that he will casually mention he did this in the middle of some unrelated conversation
forward-only pretraining now matches backprop on the Noise benchmark. Relay: val 3.704 in 2.5 min on 8×H100. backprop with the same model, data and optimizer: 3.712. the current record, Boost→Cut: 3.945 in 7.6 min. no autograd anywhere. 🧵
1
4
77
5,325
tensorqt in SF retweeted
forward-only pretraining now matches backprop on the Noise benchmark. Relay: val 3.704 in 2.5 min on 8×H100. backprop with the same model, data and optimizer: 3.712. the current record, Boost→Cut: 3.945 in 7.6 min. no autograd anywhere. 🧵
Inspired by Nick Cammarata’s interpretability essay “Neural Networks Are Already Clean”, I turned a couple of its ideas into speedups for pretraining without backpropagation. Still in the works. 2.3x speedup, cool xitter.cf/industriaalist/status/…
11
40
7
440
56,102
this has always been the case, the universe doesn’t owe you simplicity. everything looks simple because that’s all we could discover
If we want to continue progress in science, we must be prepared to acknowledge that the correct description of nature will eventually simply be too complicated for humans to understand. I do not think it is wise to make human understanding a requirement. AI is the way forward.
5
4
57
2,595