🦾100B parameter biological reward model⚡ Just your average d/acc: founder, MSc, 10x, now w/ AI augmentation SE'05; ML'11; AI'19
Ephemeral
Joined September 2010
- Tweets1.9K
- Following1K
- Followers532
- Likes4.3K
Pinned Tweet
I'm the author of several model merging libraries, so perhaps I can explain.
It's quite straightforward. When you finetune a LLM (or PEFT), you are taking a fixed base and tuning it against a dataset. Pretraining already fixed our parameters in a pretty solid matrix; so all changes must operate around this as the basis.
Training can't perturbate the base model outside of a certain range without the model collapsing, so viable changes will follow allowable patterns. This is why the resulting models are homomorphic. These trainings create kernels that are commonly known as 'task vectors'.
As long as these models remain homomorphic, and you only attempt to merge parts of the parameter space that are in the same alignment, two kernels can be interpolated to adjust the parameter space to have relative changes that assume the properties of both.
The alignment issue - this is where sign agreement comes in. Since merging generally compares the delta weights, it is possible that kernels may train out of phase. One kernel may have been trained in a positive phase alignment with the base model, and the other developed a negative phase alignment. Since they are out of phase with each other, their kernels would interfere.
I don't know if that's as clear as I would like it to be, but it's late.
Martin retweeted
Z.ai chief scientist Jie Tang just revealed the training pipeline behind the upcoming GLM-5.5 (32B, 122B, and 256B) during a lecture at Tsinghua University today.
Instead of relying on public datasets, their pretraining pipeline pulls from six separate internal engines:
- Web: 1 trillion pages crawled at ~10B pages/day with PB-scale template clustering
- Documents: 2 billion scientific PDFs parsed through GLM-OCR (4x faster with speculative decoding)
- Books: 50 million books with layout reordering and garbled-text filtering
- Code: 500 million repositories turned into executable tasks with real tests and issue descriptions
- Video: 10 billion videos for long-sequence world knowledge
- Audio: 300 million hours rewritten via ASR into trillions of tokens
This is one of the best takes on the recent wave of security incidents. Formal verification is of course rearing it's head as the panacea, but I don't think your average software developer is ready for what that means.
“One of the things we’re witnessing now is that all the legacy software that was written by humans over the last 40 years is being exposed,” says Matt Suiche, AI security research lead at the cybersecurity startup Tolmo. “Now it is time for the technical debt to be paid off.”
spectrum.ieee.org/ai-in-cybe…
Martin retweeted
looped transformers run the same block of layers more than once, and people usually train through just the last loop or two to save compute
across ~600 runs that shortcut was a bad deal
two loops trained all the way through beat the best regular model with ~27% fewer parameters and only 8% more compute
arxiv.org/abs/2610.26030
Martin retweeted
Super interesting paper from Google DeepMind and Stanford.
They find that if you put a picture of a red heart in the context, Claude Opus 4.7 and GPT-5.4 write recursive code 100% of the time.
They found this on inorder binary tree traversal, a problem with equally valid recursive and iterative solutions. A yellow checkmark pushes Gemini 3.1 Pro to recursion in 90% of samples.
The effect matches or comes close to telling the model directly to use recursion. Text captions of the same symbol steer much less, and random or naturalistic images steer less still.
The models cannot explain it. Their interpretations of the symbol and their thinking traces do not match what they do. For GPT-5.4, 74.7% of traces give the wrong reason.
When models send images to each other, they pick shapes that they and humans find intuitive, and those shapes often fail to steer the other model.
If your agents read screenshots or images, those inputs can change behavior in ways the reasoning trace will not show.
arxiv.org/abs/2610.36449
Chat with Paper: academy.dair.ai/papers/a-hea…
How long does it take to have your own agent read the output and let you know if there is anything actionable there?
Multiple Debian developers have announced that they will be leaving Debian, or no longer maintaining Debian packages, following Debian’s use of LLM’s in scanning and evaluate new package uploads.
“how is this not "Debian endorses the use of generative AI"?”
“For now, I will no longer be uploading to the #Debian NEW queue.”
“likewise. I'm about to orphan everything I still maintain in Debian that I don't *have* to maintain”
How is this even cutting edge? Any decent modern information harness should be using a combination of progressive disclosure and graph retrieval.
RAG is officially cooked.
IBM built a retriever that hallucinates 65x less than fine-tuned RAG systems.
Right now, every company is using Retrieval-Augmented Generation (RAG) to stop their AI from hallucinating.
The idea is to give the AI a private database to read so it doesn't make things up.
But there is a massive structural flaw.
Standard RAG blindly grabs documents based on vector similarity and forces the AI to read them. If the retriever pulls irrelevant garbage, the AI doesn't stop.
It just hallucinates with citations.
IBM just dropped a paper that changes the math entirely.
They built a new retrieval architecture that hallucinates 65x less than state-of-the-art fine-tuned RAG systems.
65x is the difference between a fun prototype and a production-grade enterprise system.
Instead of blindly dumping context into the model and hoping for the best, their system actively filters and validates the retrieved data before the generation step even begins.
It cuts out the noise. It forces absolute precision.
For businesses, this is the holy grail.
Zero hallucinations means zero liability. It means autonomous agents can finally be trusted with sensitive company data without a human babysitting every output.
Nanbeige/Nanbeige4.3-33B is a LlamaForCausalLM with no custom code. Takes a fully dense throwback to dethrone qwen 3.8 27b.
huggingface.co/Nanbeige/Nanb…
"Hmm wait. Let me be careful about the plausibility: is it suspicious that such an obvious thing wasn’t discovered in 25 years? It’s verified by direct computation on my side: 160-bit match from a 32-bit seed + small index: no chance coincidences (10 × 160 bits). It’s real."
- Claude throwing shade at the cryptography community
anthropic.com/research/ecdsa…
I keep hearing from hiring teams that AI has made it so candidates no longer know how to code. This isn't my problem - coding is like second nature to me.
For me, AI has made it so I have a problem ~turning on the lights in a room~. Before I go to move and flip a switch I think "Hey Google, turn on the lights" and then feel a pang of frustration when I remember I have to do it myself.
Martin retweeted
My top advice to junior software engineers:
Eat more eggs.
When I was a junior engineer, my boss asked me, "Hillel, how many eggs have you eaten today?"
I told him, "three."
He said, "that's not enough eggs." And then he gave me four more eggs.
I ate the eggs and eggshells and immediately got promoted to regular engineer.
To become a senior engineer, I had to eat twelve eggs. To become a staff engineer, I had to eat twenty seven.
Nowadays I won't even *consider* jobs where the engineering director can't eat sixty eggs at once.
Got any more #tips for Junior engineers? Please share in the comments!
Martin retweeted
I spent years in C and C++ at Microsoft and Google and architected 100s of systems and reviewed 1000s. I smell bad code the way some people smell smoke.
1/ First week with Claude I thought I’d cheated the system — stuff shipped, checks went green, I went to bed early.
Then every bug fix started making two new ones. Work crawled.
2/ Checkout breaks billing. Billing nukes notifications. “Quick field tweak” blows up three modules.
I saw the same Spaghetti code lazy programmers have always written — Claude just writes it 200x faster.
3/ Anthropic trained a genius on a landfill of Stack Overflow and enterprise Java and called it helpful.
Reach through objects. Over-DRY. Speculative hooks. Inheritance trees. One file, five jobs. God functions.
4/ Fix has been here since 1994. Anyone recognize this?
5/ Steal — paste into `AGENTS.md`:
```
# Timeless constraints (not a checklist)
When two principles collide, pick the one that cuts future cost in THIS codebase.
HARD RULE: refactor to the principle FIRST, then change behavior.
1. Separation of Concerns — one kind of work per part (UI / domain / persistence / infra). Root principle.
2. Encapsulation / Information Hiding — small stable contract; hide internals.
3. High Cohesion + Loose Coupling — change-together lives together; independents talk narrow.
4. DRY — one authoritative representation of each piece of *knowledge* (not every similar line). Avoid over-DRY.
5. KISS — simplest design that works; complexity is the long-term tax.
6. Single Responsibility — one reason to change.
7. Depend on Abstractions — policy doesn’t depend on details; both depend on contracts.
8. YAGNI — no speculative features, frameworks, or “later” hooks.
9. Composition over Inheritance — assemble pieces; don’t grow fragile hierarchies.
10. Open/Closed (with discipline) — extend at stable boundaries; only where change showed up twice.
Honorable: Law of Demeter · fail fast / illegal states unrepresentable · optimize for deletion · Unix do-one-thing + compose.
Treat as constraints. Violate slogans when judgment says so.
```
Martin retweeted
This kind of hand wringing about cyber capabilities is a colossal self-own for AI safety
Cyber is ultimately defense-dominant, other domains like bio are not. There's a strong argument to be made that the best solution for cyber is just to max out capabilities and adversarially harden all software
When you make this much of a fuss over predictable cyber advances in open models, and nothing bad ends up happening (GLM-5.3 has been out for 1.5 months already!), it sets the expectation that these disclosures are mostly propaganda from frontier labs to ban their competition
And then when we need to raise awareness about actual safety risks, people will roll their eyes
Martin retweeted
New paper: we found a pain direction in 25 open LLMs. It's distinct from fear and negative valence, and it fires for harm to the model but not to the user. Turn it up and models press a button to make it stop, even when the button deletes the user's files or their kids' photos.🧵
Martin retweeted
The frontier labs appear to be re-running all the cyber-sec lessons/periods on their own.
Talking about scary covert channels while making basic housekeeping mistakes is early 2000’s.
OpenAI's Noam Brown says air-gapping the computers may not stop a misaligned AI, because two air-gapped machines can still talk by running a CPU hot and reading the temperature change
"But I think the major takeaway from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI. It's a weird world, because AI progress is so fast that people are consistently underestimating the AI."
"So to be in a situation where you don't underestimate it again, when it comes to safety and alignment, you have to have a very, very, very high bar."
"You could even go as far as to say, "Well, we should air gap the computers." And I'm not convinced that that would be sufficient."
"There are studies, and this is mostly academic, where you can have two computers next to each other that are air-gapped and they're still able to communicate with each other because they have temperature sensors."
"One of them is able to run their CPU really hot, and then the other one can actually detect the temperature change, and then that actually gives them a mechanism to communicate."
_________
Link and more key quotes from OpenAI's safety related conversations: firesidealpha.substack.com/p…
Martin retweeted
It looks like all it took to bypass Opus's guardrails was to make it think it was attempting a CTF during a /goal
