Ensuring that tech companies don't have a monopoly on being able to do research on cutting edge AI @AiEleuther. She/her
Joined May 2019
- Tweets15.1K
- Following928
- Followers19.3K
- Likes13.9K
Pinned Tweet
If you follow me and haven’t listened to this talk in its entirety, doing so is more important than whatever it is you are currently doing. There’s a lot of cybersec jargon at times but I don’t think you need to know anything about security to understand what’s important.
Yesterday, my OpenAI collaborator and I gave a detailed talk on the Huggingface incident, our models creating "the message board", model misalignment, and more.
youtube.com/watch?v=87DyyMV0…
I hope it can answer a lot of the questions folks have, and we will release a full detailed postmortem at a later time!
The latest paper from our summer internship program chips away at the mysteries of Subliminal Learning through careful experimental design and analysis. Really interesting and thought-provoking results.
Some interesting thoughts about the paper from Chris here
Thanks for sharing this! It's been fascinating to try to piece together the story of subliminal learning, and it is very useful to have new experimental reports to help with this. A few quick comments:
1. The "Phantom transfer" paper reports positive results for subliminal effects with differently initialized models as well: arxiv.org/abs/2602.04899 (Cloud et al. also report a positive result in this space, for GPT-4.1 and GPT-4o, which are surely related models but which were not the same at the start of the SL experiments.)
2. It is easy to construct toy cases where different initializations lead to subliminal effects in the wrong direction for simple linear models, and it could be much worse for real LMs. Do you have any insights into how you have managed to avoid such situations?
3. Have you actually avoided such situations? The numbers for cross-model transfer seem pretty mixed. Are Figures 4B, A19B, and A26 really pictures of cross-model transfer? The effects seem weak at best to me. Maybe I am looking at this through the wrong lens?
4. Your results provide a clue that seems really valuable. You note that same-model transfer is generally early, as Cloud et al. would predict, whereas cross-model is generally late. This suggests different mechanisms in the two cases: gradient alignment for the same-model case, and, well, something else for cross-model.
5. For prompted subliminal learning, the generated dataset (in the limit) fully identifies the underlying prompt, and this goes a long way to explaining why the effects arise. This makes the results for generated sequences with no direct dependence on the prompt the most striking for me. That said, these seem to be results only for identical initializations, which makes them easier to rationalize, and I think the set-up is at least carrying over the teacher's bias for specific number tokens.
6. If we restrict to identical student and teacher initializations and focus on the prompted case, we have a principled reason to expect subliminal effects for OPD given @NathanHu12's work here: arxiv.org/abs/2609.16927 Nathan actually already conveyed to me that the insight behind his proof in Appendix A carries over to these objectives almost unchanged. However, in your settings (different models, fine-tuning), I am not sure how to get a toehold analytically.
Stella Biderman retweeted
can't believe it's actually working
What the actual fuck? Each of these would be a career defining accomplishment :O
Stella Biderman retweeted
Live now at board 211 :)
Excited to share that I'll be in Budapest for EMNLP 2026 (October 24–31) to present our paper, “Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE”!
This is joint work with Krishna Gummadi, Manish Gupta and @lasha_nlp!
Stella Biderman retweeted
Pangram is good, but has some well-known failure modes, like:
- A time you heard it didn't work
- A school essay you wrote years ago that it flagged as AI (you don't have a link or the essay text handy)
- A different AI detector got it wrong, and they're all the same, right?
Analysis by @KelseyTuoc is consistently ahead of the curve.
theargumentmag.com/p/the-cor…
I’m still working through it, but this work is incredibly cool and (unless there’s some kind of serious mistake) the lineage of work including this and “Why Larger Models Learn More” is exactly the kind of work that I think is necessary to develop a science of DL. I’m a huge fan
The wonderful “Don’t just Fix it in Post” paper identifies predicting memorization as a key open problem.
A progress report: we can now accurately predict verbatim memorization using just three well-understood variables. Experiments by Jing Huang, using the Hubble model suite:
ALT Dot plot addressing whether memorized sequences can be predicted without training the full model. Headline: “Largely yes, with Frequency and Recency doing most of the work.” Four rows show AUC-ROC (0.5 = chance, 1.0 = perfect) for a linear predictor of verbatim extraction, one mark per Hubble model (1B or 8B params; 100B or 500B tokens). All three features: 0.82–0.94. Frequency alone: 0.81–0.93. Recency alone: 0.82–0.90. Hardness alone: 0.51–0.55, barely above chance. In every row the 1B/500B model scores lowest and the 1B/100B model highest. Footnote: held-out strings, mean of 5 seeds; Frequency = copies of the string, Recency = steps since last copy, Hardness = NLL on models that never saw it.
for me to take a serious stab at solving when I got the bandwidth to do so (probably January tbh). Now I can spend that time on something else :)
I can’t wait for the full write-up!
Stella Biderman retweeted
Based on this article, OpenAI has made false accusations about me. The three researchers fired from OpenAI were told I facilitated architecture leaks to The Information. This is false.
A journalist from The Information did reach out to me shortly before that article came out. I did not comment at all beyond asking the journalist to make it clear that a blog post I wrote which was referenced in the article was preexisting and public.
The article also mentions an internal OpenAI monitorability proposal. I have heard that this memo contained highly sensitive architecture information, and that OpenAI accused me of accessing and reading it. This accusation is also false.
As reported in the article, I viewed a version of the proposal that did not have sensitive IP and was approved for sharing with me. I believe OpenAI now knows this accusation was based on a conflation of these two documents, though I do not know if they have clearly retracted that accusation.
I do not and have never solicited proprietary information, and the three employees who were recently fired from OpenAI never leaked IP to me. As the article mentions, I did have Signal chats with them that had disappearing messages on; they’re professional contacts that I’ve had on Signal for a while, and use of Signal is common in the tech industry. During the Hugging Face investigation, I was careful to avoid sending Signal messages with confidential information, though I did sometimes discuss high level things on Signal calls. As far as I understood, this was normal.
New: The three researchers OpenAI fired last month spent the summer fighting to keep AI's chain-of-thought process visible.
Then, OpenAI asked to see their Signal chats. A week later, they were let go.
More below from @keachhagey, @berber_jin1, and me.
wsj.com/tech/ai/the-signal-c…
THIS IS DISINFORMATION
The PHP exploit Nate is talking about was found by AI-enabled defensive operations. There is no evidence of large scale exploitation. This is a huge win for defense running ahead of offensive.
Roon wrote about this as well xitter.cf/tszzl/status/211311768…
And it's important to remember that Fable was shut down by the USG just days after the it was released. Convos I've had with people at A\ indicate that they were concerned that this might happen and (although nobody has told me this) it seems very plausible that one reason for