Ben Edelman (@EdelmanBen) profile picture

Agent Security lead @ U.S. Center for AI Standards and Innovation. Prev: science of deep learning PhD @ Harvard

Joined March 2014
Photo by Ben Edelman (@EdelmanBen)Photo by Ben Edelman (@EdelmanBen)Photo by Ben Edelman (@EdelmanBen)Photo by Ben Edelman (@EdelmanBen)Photo by Ben Edelman (@EdelmanBen)Photo by Ben Edelman (@EdelmanBen)Photo by Ben Edelman (@EdelmanBen)Photo by Ben Edelman (@EdelmanBen)Photo by Ben Edelman (@EdelmanBen)Photo by Ben Edelman (@EdelmanBen)
Ben Edelman retweeted
To prepare for and respond to SI control incidents, NIST CAISSI, NSA SISC, and CISA have authored voluntary practices for organizations training and evaluating highly capable agents.
1
31
4
94
8,504
Ben Edelman retweeted
Excited to share @NIST+CAISI’s initial public draft on how to run and report results of automated evals. If you have opinions on evals, we’d love your feedback — help us improve the AI evals ecosystem! Public comments accepted through March 31st via [email protected]. more in🧵
3
6
1
29
3,765
Ben Edelman retweeted
One of the best places if you have technical background and care about AI going well!
People sometimes ask me how to leverage a technical background to jump into U.S. AI policy. As of this week my answer is straightforward: apply to join us at CAISI! We're a startup within government, and we're doing a hiring surge.
3
23
4,795
Ben Edelman retweeted
Seems like a great opportunity for technical talent to come into government and help the USG make sound, technically informed decisions on AI
CAISI is hiring for a bunch of exciting new roles, from partnerships to technical experts in AI x bio / chem and more. They're serious about bringing in strong researchers & engineers and letting them do good work. Based in DC or SF: nist.gov/caisi/careers-caisi
9
14
2
140
49,925
CAISI is hiring for a bunch of exciting new roles, from partnerships to technical experts in AI x bio / chem and more. They're serious about bringing in strong researchers & engineers and letting them do good work. Based in DC or SF: nist.gov/caisi/careers-caisi
4
39
5
168
85,142
Ben Edelman retweeted
CAISI is recruiting an intern to support an agent security standards project. Position closes Jan. 15 for a February start. Please help spread the word. Details in thread:
2
16
1
52
14,531
Ben Edelman retweeted
This is cool work by CAISI to publish a case study on models cheating on evals in practice! nist.gov/caisi/cheating-ai-a…
1
12
65
6,316
Ben and team are extremely good—e.g. check out their cool work w OpenAI!
Replying to @EdelmanBen
Did not have "red team frontier AI systems as part of federal govt job" on my bingo card during my ML PhD. It's been a blast.
1
1
1
16
2,714
Ben Edelman retweeted
MOSS is happening this Saturday (7/19) at West Ballroom B, Vancouver Center! We are excited to have an amazing set of talks, posters, and panel discussions on the insights from and potential of small-scale analyses. Hope to see a lot of you there! 💡
6
3
22
14,594
Ben Edelman retweeted
We are extending the deadline to May 26th 4:59pm PDT (11:59pm UTC). Thank you everyone for your interest & inquiries; we look forward to learning about your results! 🪄
Announcing the 1st Workshop on Methods and Opportunities at Small Scale (MOSS) at @icmlconf 2025! 🔗Website: sites.google.com/view/moss20… 📝 We welcome submissions! 📅 Paper & jupyter notebook deadline: May 22, 2025 Topics: – Inductive biases & generalization – Training dynamics & inference behavior – Evaluation & stress tests – Failure mode diagnosis – Architectural & ecosystem innovations – Mechanistic interpretability – Societal & ethical considerations – Alignment & fine-tuning methods – Synthetic tasks & emergent phenomena – Algorithmic innovations See you all in Vancouver! #ICML2025 #MOSSWorkshop
9
13
2,022
Cool work by @EdelmanBen and the US AISI team! Happy to have helped out a bit along with @AlexandraSouly :)
1/ Excited to share a new blog post from the U.S. AI Safety Institute! AI agents are becoming increasingly capable. But they are vulnerable to prompt injections in external content – an agent may be given task A, but then be “hijacked” and perform malicious task B instead.
1
15
1,488
It's great to see that AgentDojo is being used to evaluate the robustness of models that seem secure when running simple, static attacks. Amazing work from the US AISI!
1/ Excited to share a new blog post from the U.S. AI Safety Institute! AI agents are becoming increasingly capable. But they are vulnerable to prompt injections in external content – an agent may be given task A, but then be “hijacked” and perform malicious task B instead.
2
5
20
2,747
Ben Edelman retweeted
New preprint (w/ Apple collaborators Aravind Gollakota, Parikshit Gopalan, Charlotte Peale, and Udi Wieder)! We define "higher-order calibration" and prove that it's a necessary and sufficient condition for accurate uncertainty decomposition. (1/N) arxiv.org/abs/2412.18808
1
2
9
770
The 4:30pm poster session today at #NeurIPS2024 will feature "The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains," by @ezra_edelman, Nikolaos Tsilivis, @SurbhiGoel_, @EdelmanBen and @EranMalach. #KempnerInstitute
1
7
11
1,809
Ben Edelman retweeted
Will be presenting this work at #NeurIPS2024, today 11am, poster #2311. Come visit us!
Modern generative models are trained to imitate human experts, but can they actually beat those experts? Our new paper uses imitative chess agents to explore when a model can "transcend" its training distribution and outperform every human it's trained on. arxiv.org/abs/2406.11741
1
5
11
1,885
Very excited to speak at the Theory of Interpretable AI seminar coming Thursday at 10am EST. Please join to hear about the recent work on how margin maximization can explain some intriguing observations in mechanistic interpretability.
📢Theory of Interpretable AI Seminar📢 On Thursday, @depen_morwani from Harvard will present work on how **margin maximization** can explain observed phenomena in mech. interpretability! ⏲️Thursday Dec 5th, 4pm CET / 10am EST 🌐tverven.github.io/tiai-semin… @ML_Theorist @tverven
3
19
4,039
I just started a Substack about AI - how it can be used for public benefit, and how to mitigate its downsides. First post is on SB 1047, a controversial California AI bill now at Gov. Newsom’s desk to approve or veto aiinterpreter.substack.com/p… 1/3
1
1
1
9
3,710
Here's a promising new Substack newsletter about AI by Jenny Kaufmann, a PhD student in math/machine learning at Harvard. First article is about the pros and cons of SB 1047. aiinterpreter.substack.com/p…
1
1
9
2,815
This paper seems very interesting: say you train an LLM to play chess using only transcripts of games of players up to 1000 elo. Is it possible that the model plays better than 1000 elo? (i.e. "transcends" the training data performance?). It seems you get something from nothing, and some information theory arguments that this should be impossible were discussed in conversations I had in the past. But this paper shows this can happen: training on 1000 elo game transcripts and getting an LLM that plays at 1500! Further the authors connect to a clean theoretical framework for why: it's ensembling weak learners, where you get "something from nothing" by averaging the independent mistakes of multiple models. The paper argued that you need enough data diversity and careful temperature sampling for the transcendence to occur. I had been thinking along the same lines but didn't think of using chess as a clean measurable way to scientifically measure this. Fantastic work that I'll read I'll more depth.
123
284
78
2,226
388,813
Ben Edelman retweeted
Generative models are trained to imitate the data distribution, but can sometimes outperform the human experts who created the data! 🚀 We study this "transcendence" phenomenon both theoretically and empirically. Fun work with an amazing group of authors! arxiv.org/abs/2406.11741
Modern generative models are trained to imitate human experts, but can they actually beat those experts? Our new paper uses imitative chess agents to explore when a model can "transcend" its training distribution and outperform every human it's trained on. arxiv.org/abs/2406.11741
1
2
14
2,177