award loosing computer scientist looking for a chance to change the world
compyuter
Joined January 2016
- Tweets3.5K
- Following1.5K
- Followers234
- Likes48K
cabral retweeted
Ok guys so all the recent weight-space programming buzz inspired me to write a blog post on zeroth-order, weight-space methods (back to the basics), from the theory to a bit of update on our latest result.
Turns out plain weight perturbations don't just work for fine-tuning on math, they work for post-training agents too: vanilla ES takes Qwen3.5-4B from 12.1% → 21.1% on Terminal-Bench 2.1, past the released Tmax RL checkpoint (18.2%).
Better yet, no credit assignment, no critic, no masking, no messy hyperparameters, just good ol' perturb-and-update. Oh did I forget to mention that all with population size of ~30?
Here's why I think this could be huge for long-horizon agentic post-training 🧵+blog post👇
cabral retweeted
The #TauCeti project is using AI to build a library of formalized mathematics: Human-written roadmaps specify the intended mathematics, and the #Lean kernel checks every proof the AI contributors produce.
Adversarial AI reviewers then assess whether the results match what the roadmaps intend. Kim Morrison writes that this process “brings the material up to a standard we can keep building on.”
The aim: reusable formalizations of “all of basic mathematics.” The project is incubated by Lean FRO and the Mathlib Initiative, and open to contributions from everyone.
See the Tau Ceti use case for more: lean-lang.org/use-cases/tau-…
i've adapted @_lyraaaa_'s "neuron-explorer" for @liquidai's LFM3-1.7B, and its really fun, i'm having a blast chatting with different doses on different neurons.
i think i found an interesting one: L9N6126+1.0 apparently gives it a sense of "Unsureness".
cabral retweeted
Estou lancando hoje um novo benchmark de LLMs - comprar ingresso no ingressopalmeiras.com.br , lidando com todos os 999982343 problemas do site.
Score ate agora
GPT-6 Astra - zero
Claude Opus 5.5 - zero
Claude Fable - zero
Claude Mythos - zero
Claude Devil - zero
Deepseek - zero
cabral retweeted
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
cabral retweeted
1/4
Meet Qwenputer: Qwen3.5-4B with a full RISC-V CPU inside its weights, using the model's own Gated DeltaNet layers as RAM.
It runs C and MicroPython in its forward passes, no external runtime.
On DeepMind math: base 55–90%, base + Python tool 95–100%, Qwenputer 97.5–100%.
cabral retweeted
I saw the recent softmax bug which was found by Gemini, and tried to reproduce it with a property-based test in Hypothesis. Surprisingly, even the proposed fix fails this test. So, I wrote a more robust softmax which does indeed pass this test!
arxiv.org/abs/2610.36427
cabral retweeted
Flávio Bolsonaro é chefe de organização criminosa que desviava dinheiro, aponta Ministério Público glo.bo/2PHxkNf #G1
cabral retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
cabral retweeted
🚨 URGENTE!
Sem nenhum anúncio amplo ao público, o TSE alterou o local de 4.352 seções eleitorais faltando apenas UMA SEMANA pro 2° turno.
Com uma média de 300 eleitores por seção eleitoral no Brasil, isso pode prejudicar mais de 1 MILHÃO de eleitores.
Por isso, estou cobrando que tanto o TSE quanto o Tribunal Regional Eleitoral de São Paulo façam uma campanha informativa, com URGÊNCIA, na televisão, na internet e rádio, explicando às pessoas as razões da mudança, que elas podem ter sido afetadas e como conferir o novo local de votação.
O direito ao voto não pode ser prejudicado de última hora pela falha de comunicação do Tribunal responsável por proteger o direito ao voto.
Mas não podemos deixar que isso tire nosso ímpeto ou a capacidade de ir às urnas no dia 25.
Confira seu local de votação no aplicativo e-Título hoje e confira de novo, no máximo, na quinta-feira, pois, caso o aplicativo trave e pare de mostrar o local de votação, você precisa deletar seus dados do aplicativo, esperar 24 horas e fazer o login novamente.
cabral retweeted
My dear friends, I’d like to show you my latest toy project
Introducing Handwired: turn your regular JavaScript code into a neural net!
- no training
- exact outputs vs the JS program
- fully differentiable
All runnable in a browser, without GPU
cabral retweeted
UltraBERT is here (along with super interesting lessons learned from training it)!
152M + 380M encoders, 32K context, built for classification, NER, QA, reward modeling + retrieval.
Most interesting part is the training recipe:
-5% masking for classification
-50% masking for retrieval
-Separate LM head for each
-Distillation only on the 5% sequences
Distillation findings are also really insightful. They trained a 1.3B teacher, pruned it by width to 380M and 150M, and distilled both smaller models from the teacher. But they only applied logit distillation to the 5% masked sequences.
Some other interesting training lessons:
-Encoder performance saturated early, even as loss kept falling
-Multi-stage data mixtures had little effect after fine-tuning
-32K context added during only the final 100B of 2T training tokens
-Distillation not automatically useful across every capability
Strong size-to-performance tradeoff too:
-380M leads the authors’ comparison on GLUE + BEIR
-150M matches RoBERTa-large on classification at 2.3x smaller
-~8 point lead on code + math retrieval
-99%+ Passkey recall through 32K
Great open source release! Long live BERT ✌️
@huggingface weights: huggingface.co/collections/d…
Paper: arxiv.org/pdf/2610.15028v1
cabral retweeted
Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel.
- Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
- Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract.
- Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
- And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
The funnel is driven by a combination of factors:
1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel.
2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel.
3. Access. Free-tier, paid-tier, internal.
So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.
Judging by my tl there is a growing gap in understanding of AI capability.
The first issue I think is around recency and tier of use. I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on AI a little too much. This is a group of reactions laughing at various quirks of the models, hallucinations, etc. Yes I also saw the viral videos of OpenAI's Advanced Voice mode fumbling simple queries like "should I drive or walk to the carwash". The thing is that these free and old/deprecated models don't reflect the capability in the latest round of state of the art agentic models of this year, especially OpenAI Codex and Claude Code.
But that brings me to the second issue. Even if people paid $200/month to use the state of the art models, a lot of the capabilities are relatively "peaky" in highly technical areas. Typical queries around search, writing, advice, etc. are *not* the domain that has made the most noticeable and dramatic strides in capability. Partly, this is due to the technical details of reinforcement learning and its use of verifiable rewards. But partly, it's also because these use cases are not sufficiently prioritized by the companies in their hillclimbing because they don't lead to as much $$$ value. The goldmines are elsewhere, and the focus comes along.
So that brings me to the second group of people, who *both* 1) pay for and use the state of the art frontier agentic models (OpenAI Codex / Claude Code) and 2) do so professionally in technical domains like programming, math and research. This group of people is subject to the highest amount of "AI Psychosis" because the recent improvements in these domains as of this year have been nothing short of staggering. When you hand a computer terminal to one of these models, you can now watch them melt programming problems that you'd normally expect to take days/weeks of work. It's this second group of people that assigns a much greater gravity to the capabilities, their slope, and various cyber-related repercussions.
TLDR the people in these two groups are speaking past each other. It really is simultaneously the case that OpenAI's free and I think slightly orphaned (?) "Advanced Voice Mode" will fumble the dumbest questions in your Instagram's reels and *at the same time*, OpenAI's highest-tier and paid Codex model will go off for 1 hour to coherently restructure an entire code base, or find and exploit vulnerabilities in computer systems. This part really works and has made dramatic strides because 2 properties: 1) these domains offer explicit reward functions that are verifiable meaning they are easily amenable to reinforcement learning training (e.g. unit tests passed yes or no, in contrast to writing, which is much harder to explicitly judge), but also 2) they are a lot more valuable in b2b settings, meaning that the biggest fraction of the team is focused on improving them. So here we are.
cabral retweeted
Let me show you something crazy.
This is Qwen 3.8 27B, and every frame is 1 real equation from the model's actual math. The prompt is easy and thinking is off: we ask what the capital of France is, with a one-word reply.
If you think this video is doing math fast, it's actually skipping about 7.2 MILLION equations between every frame. After all the math, Qwen gives the final answer: "Paris".
That one-word answer took 25.8 BILLION equations, and my NVIDIA RTX PRO 6000 Blackwell did it in about 13 milliseconds. This video is 32 billion times slower than the actual GPU. Showing every single equation at one per frame at 60 fps would take over 13 years, for one word.
Now imagine scaling that from 27B parameters up to models with hundreds of billions or even trillions of parameters, and that's why AI needs so much compute.
Appreciate your GPUs, no matter which one you have. They're actually insane.
