Working for a good future
Berkeley
Joined October 2021
- Tweets2.8K
- Following639
- Followers1.9K
- Likes26.3K
Rachel retweeted
honestly degrading the idea of “existential safety” — the biggest problem humanity has ever faced — by lumping it together with things like spam, “toxicity”, political unwanted content, deepfakes etc has been a huge disaster for ai safety and polarized a lot of people
Rachel retweeted
Wow, Gameto—a fertility tech startup I've written about a couple of times that aims to make IVF and egg freezing easier—just got acquired in a $1.4b deal.
Gameto has signed an agreement under which Ferring Pharmaceuticals would acquire the company. Completion remains subject to customary and other closing conditions.
bloomberg.com/news/articles/…
Rachel retweeted
SITUATION EXPLAINED: A months-old startup founded by ex-DOGE staffers used AI agents to break into Anthropic in under 12 hours.
• Cathedral found paths to Anthropic's meeting transcripts, salary and job-offer details, and Slack messages about its advanced models
• It got in through weaknesses in third-party business software, after days of failing to break into Anthropic directly
• Anthropic says no customer prompts, Claude responses, source code, or model weights were touched, and it cut off access the same day
• The test was authorized under Anthropic's bug bounty program, but Anthropic says Cathedral didn't disclose all of its activity
• Cathedral is valued at $1.4B with backing from a16z and Sequoia, and 3 federal agencies already contract with it
• Its co-founders include Luke Farritor, who helped decode the ancient Herculaneum scrolls with AI
@theojaffee: "If a bunch of kids who have just started a company can do this, then potentially the Chinese aren't far behind. Aren't far behind? If anything, they're far ahead."
Rachel retweeted
EXCLUSIVE: In less than 12 hours, a group of ex-DOGE cyber researchers were able to penetrate Anthropic’s systems using AI agents — including Slack channels where the most sensitive models are discussed.
This is just weeks after
Anthropic whistleblower Joseph Coxon charged the company w/launching “a hubristic gamble” from a “private company’s Slack.”
Read our story here (gift link):
wapo.st/4smNlsC
Rachel retweeted
The Y2K bug and the hole in the ozone layer were both serious problems that we chose to fix. It took a lot of work!
Rachel retweeted
Lots of people are asking me where to give their money - if you can only do one race, I’d give what you can to @MaryPeltola
Several reasons:
1) It’s a small state, so your money goes further per expected senate vote
2) Direct donations buy 19x ads per dollar than committees due to the unit rule - this also frees up the DSCC and SMP to put their money on states with smaller IE/coord ad price spreads and frees them up to spend more on digital
3) it’s likely going to be close!
It’s not too late! Any money you send today will get spent immediately and will provide real relief to an overworked campaign workers across a bunch of orgs!
secure.actblue.com/donate/mp…
Rachel retweeted
Steven Pinker is correct that these extreme warnings coming from the lab leaders while they race ahead is discordant and confusing.
But I am literally chuckling at the recent line i've seen from a few folks (not just Pinker) that "RSI leading to permanent irrevocable loss of control is a smokescreen to avoid scrutiny on real issues like AI enabled bioterrorism and rogue AI swarms shutting down the internet."
Fun to imagine showing this clip to myself in 2022, when a frequent argument was that talking about catastrophic cyber or bio risk was speculative sci fi and a distraction from real issues.
Rachel retweeted
Based on this article, OpenAI has made false accusations about me. The three researchers fired from OpenAI were told I facilitated architecture leaks to The Information. This is false.
A journalist from The Information did reach out to me shortly before that article came out. I did not comment at all beyond asking the journalist to make it clear that a blog post I wrote which was referenced in the article was preexisting and public.
The article also mentions an internal OpenAI monitorability proposal. I have heard that this memo contained highly sensitive architecture information, and that OpenAI accused me of accessing and reading it. This accusation is also false.
As reported in the article, I viewed a version of the proposal that did not have sensitive IP and was approved for sharing with me. I believe OpenAI now knows this accusation was based on a conflation of these two documents, though I do not know if they have clearly retracted that accusation.
I do not and have never solicited proprietary information, and the three employees who were recently fired from OpenAI never leaked IP to me. As the article mentions, I did have Signal chats with them that had disappearing messages on; they’re professional contacts that I’ve had on Signal for a while, and use of Signal is common in the tech industry. During the Hugging Face investigation, I was careful to avoid sending Signal messages with confidential information, though I did sometimes discuss high level things on Signal calls. As far as I understood, this was normal.
New: The three researchers OpenAI fired last month spent the summer fighting to keep AI's chain-of-thought process visible.
Then, OpenAI asked to see their Signal chats. A week later, they were let go.
More below from @keachhagey, @berber_jin1, and me.
wsj.com/tech/ai/the-signal-c…
Rachel retweeted
Quest Diagnostics has been hacked. Patient data was exposed. They serve roughly half of U.S. hospitals and physicians. Investigation ongoing.
Rachel retweeted
Yes. note that there's also a third option, which is "if it's too dangerous for everyone to have it, it's too dangerous to train in the first place."
Everyone should be looking for data points to advance the longstanding debate in AI policy between “diffuse dual-use capabilities rapidly so the good guys can defend themselves” and “restrict capabilities proliferation to only carefully vetted good guys.” We are putting the former position to the test as we speak.
No motivated reasoning to reach your preferred conclusion; just pay attention and update as appropriate in whatever direction seems true. This is much too important for low-fidelity tribal polarization.
Rachel retweeted
I think we should stop AI progress now, first unilaterally by key AI companies and then scaling up to more US companies and then China via international agreements. The reason for now is not that current AI models are the ones that could kill everyone, but because the coordinate steps take time. Immediate action means we have more time for the second steps.
When I tell this story to AI company folk, the typical response is "Yes pausing is good, but we should wait until the last minute when the very next model will be dangerous, and stop then."
What I realized recently is that this is being implicitly way more optimistic about government speed than I am (and I joined the UK government and was happy there for 2.5 years). They are imagining a world where, the moment the danger is imminent, a bunch of international coordination can happen on a dime.
Which is entirely inconsistent with their actual views about government! I'm the one that is more optimistic about government; why am I the one advocating for more time cushion to let the government story play out?
Rachel retweeted
Things are moving so fast that everyone seems to have completely missed how insane it is that AI companies have been allowed to unleash agentic AI products with no regulatory guidelines.
SCOOP: Major U.S. banks are detecting unauthorized third-party AI agents inside customer accounts, including some attempting to take actions that appear to be independent of user instructions, sources told Axios.
The findings have alarmed banking executives, who are racing to restrict access to their systems as AI moves at a velocity faster than financial institutions can respond, multiple industry and bank executives told Axios.
Several banks are now preparing for worst-case and even “day after” scenarios, including how to rapidly secure or restore customer accounts if things go wrong, the people said.
Several bank executives told Axios they're struggling to distinguish between human users and AI agents. One executive said banks also struggle to determine whether agents are authorized by customers or potentially operating without their permission, more akin to rogue agent behavior.
Read my latest for Axios here: axios.com/2026/10/22/ai-agen…
Rachel retweeted
6-word horror story I experienced at the office today:
"Ajeya Cotra seminar cancelled, work emergency"
Rachel retweeted
These are some powerful parrots we're working with
Effective Immediately, all organizations using PHP for their systems are advised to be prepared for an update addressing two critical vulnerabilities. PHP is widely used for web applications. These vulnerabilities often provide a direct path to the server's file system or database. If an immediate update is not feasible, it is recommend that organizations take the system offline until it can be fully mitigated. NSA will follow up shortly with more specific guidance.
Rachel retweeted
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology