writing as a rehearsal of language. language as a rehearsal of logic. pretraining research
Joined October 2022
- Tweets2.4K
- Following1.1K
- Followers2.2K
- Likes10.4K
Pinned Tweet
I spent the last six months trying to deconstruct Taalas's patents. We think it can be better: 100x fewer memfetches than their bitROM, better software by better quantization than their hardware team could.
So we wrote a compiler that:
> takes a huggingface checkpoint,
> quantizes the model
> descends the weights down metal layers to RTL and GDS, do your DRC, Yosys, PEX, make the electrical waveform execute a mat-vec from your huggingface checkpoint (thanks Cambricon tech papers)
in ~7000 lines of human-readable code.
we're looking for someone with contacts with a foundry/access to 7nm PDKs or contacts at a cheap EuroPTW shuttle? I'm broke and unemployed.
@itsclivetime this is what i think is the future of perplexity per picojoule. @zerohedge @zephyr_z9 you guys wanna see 40,000 tokens/sec?
Progyan retweeted
totally fraudulent. this is retarded
today, we introduce a little mischief to the world.
for four years, we’ve been brewing real brain cells on electrodes to understand how they compute. now, we’ve turned those discoveries into a video model anyone can use.
from 0.5¢ a second, mischief 1 is at least 5x faster than its base and the cheapest image-to-video model on @fal and @OpenRouter. soon to be on @awscloud marketplace.
it’s not a delusion anymore. biocompute is real, commercial and this is just the beginning.
happy halloween.
Progyan retweeted
This one seems like an obviously helpful no-brainer.
Multi-Token Prediction (MTP) can be done much better.
🚀 Turn out,you can recover the gradient for KL self-distillation to next-token prediction (NTP) for free without ever computing the KL. You just need a one-line code change:
h_ntp += h_mtp.detach() - h_mtp
Soft Multi-Token Prediction is pretty much a free lunch for MTP.
📜 Paper: arxiv.org/abs/2610.38331
💻 Code: github.com/JaydenTeoh/Soft-M…
🤗 HuggingFace: huggingface.co/papers/2610.3…
ALT Soft Multi-Token Prediction Illustration
Progyan retweeted
Multi-Token Prediction (MTP) can be done much better.
🚀 Turn out,you can recover the gradient for KL self-distillation to next-token prediction (NTP) for free without ever computing the KL. You just need a one-line code change:
h_ntp += h_mtp.detach() - h_mtp
Soft Multi-Token Prediction is pretty much a free lunch for MTP.
📜 Paper: arxiv.org/abs/2610.38331
💻 Code: github.com/JaydenTeoh/Soft-M…
🤗 HuggingFace: huggingface.co/papers/2610.3…
ALT Soft Multi-Token Prediction Illustration
Progyan retweeted
Foundation models are trained in stages: pretraining, midtraining, and post-training. But is this separation fundamental?
We introduce unified training: a single algorithm for the full pipeline. This gives
⚙️ Simpler infrastructure
🔬 A framework for better science
(1/n)
Yet another lovely paper.
arxiv.org/abs/2610.38331
Improving MTP by only keeping self-distillation of next token prediction in the loss term.
Progyan retweeted
I improved NanoGPT Speedrun from 39.9s → 6.7s (-83.2%), more than every record since 2025 combined (-81.4%)!
The web is full of rare, repeated content that the LM can’t memorize in time. I solve this with retrieval-augmented n-gram hints🧵
Progyan retweeted
Congrats---This looks like significant refinements over the dynamic chunking paper. Presuming you kept the x2ish flops-to-wall inefficiency there, that's still an x5 reduction in wall clock time (and tokens) for training.
Blog:
Part 1 cartesia.ai/blog/from-tokens…
Part 2 cartesia.ai/blog/from-tokens…
(Voiceover created entirely by Cartesia's Sonic text-to-speech model!)
Progyan retweeted
Let me break this down because it is too good to be true:
Any subword tokenizer that uses token units smaller than BPE benefits from teacher forcing on any likelihood eval. You literally leak more. So 5x is realistic on any logprob eval.
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
One of the few papers in the last few years that's made me want to find who the author is and say thanks! The paper before this (Belief-State Transformer) and the paper that follows (Full Bandwidth Transformer) are also lovely reads!
alright my guys if you are interested in figuring out what happen when you bias a transformer towards predicting it’s own next hidden state I have a 2h interview with @jayden_teoh_ on the subject!
method is ridiculously simple and has ton of benefits like modelling new york
Progyan retweeted
alright my guys if you are interested in figuring out what happen when you bias a transformer towards predicting it’s own next hidden state I have a 2h interview with @jayden_teoh_ on the subject!
method is ridiculously simple and has ton of benefits like modelling new york
Progyan retweeted
Introducing Nestor, a new generation of scalable architectures that generalize MoE transformers and explicitly promote sparsity in both attentions and experts, hence learn structured deep representations more efficiently. Work with @TianzheC and @druv_pai: ma-lab-berkeley.github.io/Ne…
Progyan retweeted
I was going to have a nice quiet evening and then saw ETH is false
y’know there’s things you learn in your first CS class, and you just get used to them. SAT taking exponential time is one of those things… I never questioned it, smarter people have thought about this forever…
Progyan retweeted
Here's a direct link to OpenAI's PDF claiming a refutation of ETH. In particular, it's claimed that 3SAT on n variables is in about exp(n/(log n)^{1/2+o(1)}) time
github.com/openai/math/blob/…
Progyan retweeted
I converted Qwen3 1.7B to use my tokenizer.
Results: 1.5x prefill and generation, 1.5x the context (more data in same number of tokens)
Quality loss is minimal
Check it out huggingface.co/SwayStar123/Q…
Might try for much larger models now that this is proven out
Introducing Combinatorial BPE Tokenizer!
Standard BPE tokenizers have vocabulary for many variations of a core word, particularly; commonly words with a space infront of them (eg "Hello" and " Hello" as two tokens), as well as various capitalization variants.
Why not fold those all into just one core token?
Progyan retweeted
Gemini 4 Argon is underrated 😱.
When auditing the traces, we found that Pi Harness just ends the run whenever it hits a fallback (e.g. for security task) in 20% of the runs. We rerun Argon with Antigravity Harness, and now it beats Astra and sits right behind Opus 5.5. Congrats to the team @GoogleDeepMind and looking forward to the new ckpt.
Progyan retweeted
I did a deep dive into what would the interplay between RL policy and Inference frameworks look like if one were to do RL on heterogeneous systems. The post covers Blackwells for the trainer and TPUs for inference.
gauravjain.bearblog.dev/hete…
A lot of it is extrapolation from a ton of brilliant work that has been discussed in the industry and academia covering topics like Routing Replay, Score Centering and Masked Importance Sampling. Let me know what you think. Appreciate all feedback.