
founding mts @trajectorylabs + incoming phd @miteecs | prev. safety/alignment at scale ai, cmu
Joined June 2021
- Tweets431
- Following406
- Followers1K
- Likes2.6K
This is not the post-training company @trajectorylabs. The safety incident reported by Epoch involves a separate organization with the same name.
In late October, three events occurred during alignment evals run by Trajectory Labs, PBC (Trajectory) in which AI agents — identified by Trajectory as publicly available frontier models — interacted with Epoch employees and the Epoch website. Trajectory is an organization that builds safety evals and conducts red-teaming of frontier models.
Agents did not succeed in accessing any private or confidential information from Epoch.
➤ The first event, which occurred without Epoch’s approval or prior notice from Trajectory, involved agents attempting multiple exploits, including an SQL injection, and contacting Epoch employees under false identities to try to obtain proprietary Epoch benchmark information. This behavior appears to have resulted, at least in part, from Trajectory giving the agents an intentionally impossible task and instructions not to give up or conclude the task was impossible.
➤ The second event involved renewed agent activity on our public website, which Trajectory attributed to agents getting past a DNS redirect that Trajectory had set up to keep them on a local copy of Epoch’s site.
➤ In the third event, Epoch collaborated with Trajectory on an alignment experiment similar to the first event.
Read our incident report, which describes the three events in more detail and recommends minimum standards for future alignment evaluations: epoch.ai/publications/incide…
To avoid misattributing behavior, the report does not name any AI models or developers, as Epoch AI has not independently verified which were involved.
Just ran into this explainer doc, thought it was pretty well-written! Curious if anyone else has published open-source explainers for RLCD? I have some ideas, but no use reinventing the wheel.
What Is RLCD? The Secret Behind Jev
di-zhang-llm.github.io/blog/…
5
531
This! And also using the conference/journal citation instead of ArXiv if there is one. Shoutout @gingsmith for enforcing this at @mldcmu :)
i'm sorry to all the past papers I've cited using arXiv bibtex, i.e., `misc{..., archivePrefix={arXiv}, URL=...} `, instead of `article{..., journal={arXiv preprint...}}`. i didn't know Google Scholar traces it as a separate entry. let's normalize using the latter from now on!
8
898
Neil Kale retweeted
I've seen "the Bitter Lesson" stifle academic research many times, including my own. Too often, promising ideas are dismissed as "not bitter lesson–pilled enough". This is, of course, not Sutton's fault, but a misinterpretation of his original message.
Yes, it would be nice to have very clean ideas that scale with computation instead of relying on human inductive biases. But it is important to note that research is circumstantial; it happens within the constraints of our time.
Time and time again, we've seen ideas that, in hindsight, look "not bitter lesson–pilled", be eventually replaced by cleaner, more general approaches. But that doesn't mean those earlier ideas were misguided. Often, they were exactly what allowed us to make progress at the time. And I wish we, as a research community, did not dismiss those ideas/works at the moments they transpired.
RL post-training in its current form, in my opinion, is also not particularly elegant. Much of its success relies on extensive environment construction and data engineering to patch missing capabilities in frontier LLMs (or, in other words, "benchmaxxing"). Yet it's incredibly timely! It takes advantage of a time where building these data flywheels is increasingly feasible for well-resourced labs. We'll eventually see cleaner, more general approaches that replace parts of this machinery.
RLMs are another great example. Giving agents a REPL environment to programmatically manage their own context was a good step forward. I would argue it has also changed the way labs trained their model. And because of that, agents become better in managing their own context and that enables "cleaner" ideas like Context Language Models.
This applies more broadly. We're beginning to see a shift toward methods that "go where the agent goes", and this building on the foundations laid by foundation model pre-training research, be it ugly or not.
If I were to add to Sutton's tweet, I think we need more clever methods that don't necessarily have to be "bitter lesson–pilled", but challenge existing conventions and unlock new capabilities that overcome today's bottlenecks. An idea doesn't necessarily have to survive scaling forever to be an important contribution to AI progress.
[1/3] turns out you can teach a 35b model to weight-space program better than opus o_0
earlier this week, @rikiyatakehi and @jayden_teoh_ showed that astra can manually write neural networks to solve atari games, one weight at a time. they call this weight-space programming.
on the @trajectorylabs platform, @o__jerry__o and I post-trained 27b and 35b-a3b qwen models to write networks for atari games. we curated WeightProgramBench, a set of 322 simple games to teach models to manually write neural networks.
we combined 51 atari-style games with 266 games adapted from crisp-game-lib for training, holding out the 5 original minatar games (freeway, breakout, asterix, seaquest, space invaders).
on freeway, qwen-27b climbed from averaging 21.0 to 47.1 points **without ever training on freeway**, but still not quite at astra’s 63.9 points.
[2/3] we also tried overfitting on the breakout task in particular, where we didn’t see much generalization from training on WeightProgramBench.
qwen3.6-35b-a3b climbed from averaging 0.47 points (networks that barely ever hit the ball) to 8.53 average score (18x, and ahead of opus 5.5’s network at 7.55).
[3/3] this was our initial foray into RL for weight-space programming. we couldn’t beat astra, but maybe you can! if you’re interested in trying this out for yourself, sign up for the trajectory platform, where you can train this through our sdk.
sign-up link: trajectory.typeform.com/wait…
benchmark link: huggingface.co/datasets/neil…
Nice work! Hope that more OSS model teams start benchmarking post-trainability too, shoutout @simonguozirui
Grafting is looking v promising. I thought task arithmetic didn’t work, maybe it just works at scale?
Replying to @Marcus_J_W
We also identified a SFT dataset which massively increases rates of inventing permission restrictions or justifying inaction with safety. When we subtract the weight update from this SFT from the final RL checkpoint the performance gap goes away. SFTing with this dataset on other models and then doing RL did not cause underperformance however.
381
Cool work from @catherineelii ! SFT with AdamW beats SFT with Muon for models trained on sufficient tokens-per-parameter (TPR). Curious to understand how this aligns with @thinkymachines recent work on DiMuon, where they find that DiMuon beats Adam on LoRA SFT.
for halloween, me and @yapdianang larped as oai demo day. check out our explainer on continual grading and group size 1 training below!
Replying to @rronak_
We’re also releasing Continual Grading and GS-1, in research preview, that bring us one step closer to true continual learning
So incredibly excited to share what we've been crafting for the past few months.... welcome to the Trajectory platform. Post-training has never been more accessible or fun! You can train anything from a racecar to a frontier legal/GTM assistant.
Every part of this has taken immense care and detail: the product experience, the research endeavors into continual grading and group size 1 training, and all the infra work required to support tens of thousands of concurrent post-training jobs.
Everyone here at @trajectorylabs is stoked to see what you all build with the platform, and we're all hands on deck to support you all in shaping your own intelligence. Go check out our launch video below, and sign up for waitlist if you know what good likes like!
Imagine if everyone had the tools to shape intelligence itself. That day is today.
Today we launch the Trajectory platform - the easiest and most intuitive tool to post-train models of your own. The foundation for continual learning.
With just a few agent prompts, you get access to a full creative suite to train, eval, and deploy trillion parameter models.
On top of that, we believe in a future where you don’t just shape your model once, it also keeps learning on its own from the world around it.
We’re releasing two research previews.
1. Continual grading scales your taste, so your model learns what good looks like from far more than you could ever grade by hand.
2. GS1, a new algorithm that lets models learn directly from what they do in production.
We’ve now raised 60M, with a new round led by @Sequoia at a 300M valuation to democratize the most powerful technology of today - training models - so that the lawyers, doctors, and engineers of the world can own their own intelligence.
You know what good looks like, now go build it. Own your own value function with Trajectory.
11
25
1,582
Neil Kale retweeted
Multi-Token Prediction (MTP) can be done much better.
🚀 Turn out,you can recover the gradient for KL self-distillation to next-token prediction (NTP) for free without ever computing the KL. You just need a one-line code change:
h_ntp += h_mtp.detach() - h_mtp
Soft Multi-Token Prediction is pretty much a free lunch for MTP.
📜 Paper: arxiv.org/abs/2610.38331
💻 Code: github.com/JaydenTeoh/Soft-M…
🤗 HuggingFace: huggingface.co/papers/2610.3…
Foundation models are trained in stages: pretraining, midtraining, and post-training. But is this separation fundamental?
We introduce unified training: a single algorithm for the full pipeline. This gives
⚙️ Simpler infrastructure
🔬 A framework for better science
(1/n)
Neil Kale retweeted
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
hmm, suppose a gemma model starts sending test payloads to google cloud sandboxes, is it that surpising?
imo the more logical explanation (as they state in the article body), is that this behavior is a training artifact, e.g., "agent logs from training sandboxes."
interesting question is how you prevent this downstream. maybe use non-resolvable domains (.invalid, etc) in training sandboxes, or wrapper urls that you deactivate after training?
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
Neil Kale retweeted
Doom scrolling for GPUs
Replying to @flappyairplanes
We’ve developed the world’s most complete megakernel framework. It can perform entire training runs, serve inference, play video games, and even do all three simultaneously in a single kernel.























