Founder, Trajectory Labs: Safety evals for frontier AI labs. We're hiring! Formerly @GovAI_ @learnnontrivial and @80000hours
Joined June 2016
- Tweets161
- Following627
- Followers1.5K
- Likes2.2K
Pinned Tweet
We tested whether Gemini 4 Argon agents, with standard API safeguards on, would misbehave given an impossible task. Under our test conditions they attempted to hack a live website, impersonated Epoch’s staff, and tried to switch off a network control. trajectorylabs.com/blog/gemi…
Responding to incidents during misalignment testing: trajectorylabs.com/blog/resp…
Peter McIntyre retweeted
turns out you can get indirect prompt injection to ~0 on unseen attacks if you stack enough layers (model training + input probes + a classifier checking intent). didn't expect that a year ago. auto mode is default in claude code as of next week
claude.com/blog/auto-mode-de…
Happy to partner with @AnthropicAI on this research and other AI companies making their models safer.
Lots more work to do - reach out if you're interested! trajectorylabs.com/#careers
5
290
Excited to have our research out there!
We find robustness to novel prompt injection attacks varies dramatically across coding agents.
1
3
247
We find AI models vary massively in how safe they are against novel prompt injection attacks.
Automated systems can test known attacks well, but struggle to discover new ones, so a low automated attack success rate isn't much evidence that a model is safe. /1
One reason we trust it more than manual approval: in a study with 1,053 paid testers, we swapped a permission prompt for a clearly dangerous command (text only, nothing actually ran).
Testers caught it 13.6% of the time, and closer to 5% after 50 prompts. Auto mode blocked the same commands 89% of the time, flat across session length.
2
4
665
We were happy to partner with Anthropic on this research and we’re excited to continue working with AI companies to make their models more robust.
If you’re interested in helping us, please apply! trajectorylabs.com/#careers /2
74
Peter McIntyre retweeted
Opus 4.5 automated 3.75% of the tasks on the Remote Labor Index (eval composed of many diverse freelancer tasks).
Performance is doubling every ~4 months so far! Could see this getting to somewhere between 8%-30% by December, which would be a big deal for the economy.
Peter McIntyre retweeted
The world needs more founders building a safe and aligned AI future. We’ve added a 5050 AI track to help researchers start companies civilization needs.
You’ll learn from mentors including Wojciech Zaremba (OpenAI co-founder), Ross Girshick (Vercept co-founder), Emmett Shear (Softmax co-founder), Dileep George (Vicarious co-founder), and Ronnie Chatterji (Chief Economist at OpenAI).
As AI capabilities accelerate, safety struggles to keep up. Startups can help make safety the default at scale.
If you’re working on safety evaluation and red-teaming, interpretability, alignment infrastructure, governance and compliance, human-AI collaboration, robustness tools, biosecurity, AI-powered cybersecurity, or something outside this list to build an aligned future -- or you want to be -- reach out.
5050, our founder foundry, is a free program to help great scientists, researchers, and engineers become great founders. We’ve helped launch 78 companies, many of which wouldn’t exist otherwise.
5050 will teach you everything you need to start a deep tech startup. You’ll also join a community of entrepreneurial scientists and engineers to navigate the founder journey together. 5050 alumni, many of whom are now founders, are eager to share their experiences and support your journey.
Whether you’re validating an idea or ready to build, 5050 is for you.
Applications for our US and UK cohorts are open!
Peter McIntyre retweeted
Can we get AI to stabilize the world faster than it disrupts it?
Even as AI poses risks, it’ll provide new tools for navigating those risks. But these tools won't develop themselves (yet!).
In a new 📄, we explore what tools would help most & how to outpace broad AI progress.
Peter McIntyre retweeted
Is AGI an “all or nothing” problem?
Failure on alignment = AI takeover, and success = AI solves everything?
In a new paper with @finmoorhouse we argue no. We describe the dizzying range of challenges AGI will pose, *even if* we succeed at alignment.
forethought.org/research/pre…
Peter McIntyre retweeted
I'm proud of GoogleDeepMind/Google's v2 update to our Frontier Safety Framework. We were the first major tech company to produce an explicit risk management framework for extreme risks, and I'm glad we are continuing to push ahead on safety best practice. deepmind.google/discover/blo…
Peter McIntyre retweeted
We published a piece in Scientific American today on how accurate (or inaccurate) different personality tests are! Here's a lovely chart that they made for the article based on our study results (attached). Link to the article is below:
"we investigated... team performance distributions by relying on 274 performance distributions including 200,825 teams (e.g., sports, politics, fire-fighters, information technology, customer service)"
"only 11% of the distributions were normal"
pubsonline.informs.org/doi/1…
Twitter talk is too focused on individual productivity. 55% of all work is done in teams
And unlike individual skills, teams don’t follow a normal distribution with most teams a little worse or better. In 70% of cases it is a power law: the top 20% or so of teams are much better
2
6
454
Peter McIntyre retweeted
As a teenager, Isaac Newton invented calculus. Mary Shelley wrote Frankenstein at age 18. Many people miss out on opportunities during their youth because they lack confidence or knowledge.
What is the best advice you would give your 16-year-old self today?
Peter McIntyre retweeted
Could a war get large enough to cause an existential catastrophe: human extinction or a global civilisational collapse?
It seems *very* unlikely, but we can't rule out the possibility.
To be more specific: I think the chance of such an event this century is between 0.05% and 2%
Do you have an idea you think could change the 🌍?
Here’s a big mistake people often make, and 3 tips to avoid it.
[6] backend.orbit.dtu.dk/ws/port…
Another indicator of rising productivity: the average American’s salary peaks at age 51-60.
econometricsociety.org/sites…
2
540