Helping to keep the future on track. Views my own. RAND CAST / @IFP / @JoinFAI / @Harvard_Law
Somerville, MA
Joined February 2010
- Tweets889
- Following1.5K
- Followers869
- Likes2.9K
Ben Murphy retweeted
A further announcement today as part of the budget: the UK government has allocated £40m of funding to a new "National Agentic Security Unit" which will "develop new capabilities to detect, investigate and respond to cases of rogue agentic systems. This will integrate monitoring and incident response functions with new operational capabilities."
gov.uk/government/publicatio…
1/7 I have two simple tests for British AI, driven by our values. Is it safe? And is it ours to shape? 🇬🇧
Today, Chancellor @JohnHealey_MP is announcing a major, ambitious package of investment to strengthen Britain's defences against emerging threats, back British innovation, and build our technological independence.
Ben Murphy retweeted
I couldn't resist the call of a purple avatar. I've joined @IFP as director of emerging tech.
Ben Murphy retweeted
Protecting critical systems against advanced AI cyber capabilities is an enormous challenge, but fundamentally the knowledge and technology required is available already; the difficult part is getting relevant organisations to the frontier of cybersecurity best practice, and quickly.
@benjaminmmurphy and I wrote about this last year, and recent events have shown it to be more salient than ever. Comprehensive understanding of inventory and attack surfaces, memory-safe languages for handling untrusted data, and a robust patch deployment process remain the best ways that most organisations can improve their digital security.
arxiv.org/abs/2508.15808
AI models weaponizing unfixable hardware bugs is bad, but it's a far cry from making software security futile - they still need to get your server to run the code in the first place!
A world where every RCE or shared hardware turns into an escalation of privilege spells the end of cheap-and-cheerful hosting for sensitive workloads, but that should have been part of the threat model already and the fix is in software. The hyperscalers have offered single-tenant hosts, HSM-backed keys and ephemeral credentials for over a decade, so the boundaries between untrusted input and the core workload or control plane can be enforced by both hardware and software.
Patching the next generation of hardware against novel attacks still hardens our security posture, but the real solution is an architecture where nothing worth stealing or subverting lives on shared silicon, and operators can get started on that right now.
Ben Murphy retweeted
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
Ben Murphy retweeted
This section lays out exactly the challenge Dario had spelled out in an interview with @dwarkesh_sp. Dwarkesh asked Dario why Anthropic wouldn't just commit to way more compute if they are building "countries of geniuses".
Dario replied that the challenge was uncertainty with demand forecasts. Such is the scale and pace of exponential growth that if you are off by just a little bit on the timing or growth of demand, Anthropic could be exposed to 100s of billions of compute overcommitments (or under for that matter.
That translates into either not having enough compute hence way underserving burgeoning demand and ceding the market to competitors or a massive shortfall threatening bankruptcy.
This is also why having a liquid and robust market for compute derivatives is important if not essential. These are increasingly problems that American capital markets know how to solve, well.
ANTHROPIC JUST PUBLISHED ITS S-1 FOR A U.S. IPO
Here's everything you need to know:
The Listing:
• Nasdaq Global Select Market, ticker $ANT
• Leads: Goldman Sachs, Morgan Stanley, J.P. Morgan
• Price range and share count: not disclosed yet
The Growth:
• H1'26 revenue: $16.3B vs. $1.16B in H1'25
• 2025 revenue: $4.59B vs. $386M in 2024
• $100K+ customers: 11,900+ vs. 950+ a year ago
• $1M+ customers: 1,900+
• $10M+ customers: ~250
• Consumption-based: $13.1B, 80% of H1 revenue
• Subscriptions: $3.2B, 20% of H1 revenue
The Losses:
• H1'26 operating loss: $693M vs. $2.14B
• H1'26 net loss: $80.7B
• $72.4B fair-value charge on convertible notes
• That charge tracks its rising valuation
• 2025 included a $1.5B Bartz copyright settlement
The Compute Bill:
• H1'26 compute & infrastructure: $8.9B
• Google: at least $114.5B through July 2033
• Amazon: at least $110.0B through April 2036
• Microsoft: at least $44.9B through May 2033
• ~$265.6B in total minimum commitments
• Owed regardless of actual usage
The Cash:
• $131.7B in cash & short-term investments
• Up to $77B more in note facilities:
- Google: $20B
- Amazon: $15B
- Broadcom: $42B
Who Controls It:
• Public benefit corporation
• Class F (Founder LLC): 50.1% of the vote
• Members: Dario & Daniela Amodei + 5 co-founders
• Long-Term Benefit Trust picks board majority
• Class A: 1 vote per share
• Class N (strategic partners): no vote
• Ownership caps: Amazon 12%, Google 14.99%
• 5% of shares to Anthropic Public Benefit Fund
The Risks They Flagged:
• Warns advanced AI could pose
"catastrophic or existential risks to humanity"
• Training-data copyright suits could hurt it
The Team:
• ~5,300 employees as of Sept. 30
Use of proceeds: compute, hiring, general purposes
Ben Murphy retweeted
Anthropic Risk Factors via the S-1:
1. Our success depends in large part on our ability to continue to develop the most advanced and capable AI models, and any failure to do so could adversely affect our business.
2. If we are unable to secure adequate compute or if we fail to direct our compute effectively across our workloads, we may be unable to train models that remain competitive with those of our competitors or adequately serve our customers.
3. If our actual compute requirements differ materially from our projections, we may fail to secure sufficient compute to meet our needs or we may be committed to excess compute that remains underutilized.
4. Constraints on the critical inputs supporting compute could limit our ability to secure adequate compute.
5. We are highly dependent on the continued service of our co-founders, senior management team, and key technical employees, and the loss of any of these individuals may adversely affect our business, financial condition, and results of operations.
6. We may require additional capital to meet our substantial ongoing obligations and support our business growth, and our ability to generate sufficient revenue and secure additional capital on acceptable terms is critical to our continued operations and competitive position.
7. Our company culture has contributed to our success and, if we cannot maintain this culture as we grow, our business could be adversely affected.
8. We face intense competition, and if we are unable to compete effectively, our business, financial condition, and results of operations could be adversely affected.
9. If we fail to attract new customers, retain existing customers, or expand usage of our models, platform, and applications, our business, financial condition, and results of operations could be adversely affected.
10. If the market for our models, platform, and applications develops more slowly than we expect or declines, our growth may slow or stall, and our business, financial condition, and results of operations could be adversely affected.
11. The political environment and our engagement with policymakers and public positions on AI regulation and safety could subject us to heightened scrutiny, political backlash, or disproportionate regulatory action.
12. If the use of our AI models results in harm, whether through use by malicious actors or misuse by other third parties, or if our model guardrails or safeguards fail to prevent such misuse, our reputation and brand, business, financial condition, and results of operations could be adversely affected.
13. Our AI models may behave in unintended, misaligned, or autonomous ways, including by taking harmful actions that no user or third party directed, which could adversely affect our reputation and brand, business, financial condition, and results of operations.
14. We may be exposed to liability, litigation, and regulatory action arising from harms associated with our AI models, and our contractual protections and insurance may be insufficient, any of which could adversely affect our business, financial condition, and results of operations.
15. If we are unable to manage the risks related to our focus on and commitment to AI safety, our brand, business, financial condition, and results of operations could be adversely affected.
16. If we are unable to maintain and enhance our brand and reputation, our business, financial condition, and results of operations may be adversely affected.
17. We derive substantially all of our revenue from a limited number of products and applications, and our failure to maintain and expand our product portfolio could adversely affect our business, financial condition, and results of operations.
18. A substantial portion of our revenue and accounts receivable is driven by a limited number of customers and distribution channels, and the loss of, or a significant reduction in, spend from one or more of our top customers or disruption through those channels could adversely affect our prospects, business, and financial condition.
19. We have incurred significant indebtedness in order to operate and grow our business, including to fund our substantial compute commitments, and our ability to service this indebtedness may be constrained by our cash flows and liquidity.
20. We may not have the ability to raise the funds necessary to repay the principal and accrued interest of our outstanding convertible promissory notes or the A&R Credit Agreement in cash when required by the terms of the notes or the A&R Credit Agreement, as applicable, or at their maturity.
21. Our pricing structures for our models, platform, and applications change from time to time, which could adversely affect our business, financial condition, and results of operations.
22. If we are unable to attract, retain, or motivate highly skilled personnel, we may not be able to grow or operate effectively.
23. We have a limited operating history in an evolving and nascent industry, which makes it difficult to evaluate our future prospects and may increase the risk that we will not be successful.
24. We are at an early stage of our revenue growth, we have experienced rapid growth in recent periods, and our historical growth rates are not indicative of future performance or our ability to sustain or forecast future revenue growth.
25. If we fail to manage our growth effectively, our brand, business, financial condition, and results of operations could be adversely affected.
26. We have a history of net losses and may not generate sufficient revenue to achieve or sustain profitability in the future.
27. Our results of operations may fluctuate significantly on a period-to-period basis, including as a result of strategic investment decisions and substantial compute-related expenditures, which could cause the market price of our Class A common stock to decline.
28. Certain of our operating metrics are subject to assumptions and limitations. As a result, these metrics may not provide an accurate indication of our actual performance or future results.
29. Our business is subject to an evolving and uncertain regulatory and political environment, which could increase our costs, restrict our operations, or adversely affect demand for our models, platform, and applications.
30. Our sales to government entities and highly regulated organizations are subject to challenges and risks.
31. Our reliance on a limited number of CSPs and suppliers, some of which have strategic and investment relationships with us and are our customers, creates complex dynamics that could give rise to conflicts of interest and adversely affect our access to compute.
32. We are pursuing initiatives to develop our own data center infrastructure, which involves significant capital commitments, operational complexities, and execution risks that could adversely affect our business, financial condition, and results of operations.
33. We may not successfully build, expand, and deploy our sales and marketing organization in a timely manner, or we may not successfully hire, retain, train, and motivate our sales personnel.
34. We have a limited history under our current go-to-market strategy, and our efforts to scale and evolve our customer acquisition, engagement, and retention model may not be successful.
35. Sales to enterprise customers involve complexities and risks that could adversely affect our business, financial condition, and results of operations.
36. If our marketing strategies, such as our free trials and free tiers, fail to generate the benefits we expect, or if their costs exceed those benefits, our ability to grow our revenue could be adversely affected.
37. If we fail to maintain high-quality customer support, our ability to retain and expand customers could suffer, which may adversely affect our prospects, business, financial condition, and results of operations.
38. Our product and investment decisions may not prioritize short-term financial results and may not align with investor expectations or preferences, which could adversely affect the market price of our Class A common stock.
39. Our investments in new products, technologies, and business initiatives may not produce the expected benefits and could adversely affect our business.
40. If we fail to manage our technical operations infrastructure, or experience service outages, interruptions, or delays in the deployment of our models, platform, and applications, our results of operations may be adversely affected.
41. If we are unable to make acquisitions and investments, or successfully integrate them into our business, our business, financial condition, and results of operations could be adversely affected.
42. Unfavorable media coverage regarding our business, our models, platform, and applications, the impacts of AI, and the AI industry more broadly could adversely affect our business, financial condition, and results of operations.
43. We will incur increased costs and demands upon management as a result of complying with the laws and regulations affecting public companies, which may adversely affect our business, financial condition, and results of operations.
44. Our business, financial condition, and results of operations may be adversely affected by war, terrorism, natural disasters, public health crises, political crises, or other unexpected events.
45. If we are unable to ensure that our models, platform, and applications interoperate with a variety of software applications that are developed by others, including our integration partners, we may become less competitive and our business, financial condition, and results of operations may be adversely affected.
46. Many of the risks we face are interconnected and may compound or trigger cascading adverse effects across our business, which could amplify their overall impact.
47. The application of certain copyright law doctrines and contractual interpretations of open-source licenses to modern AI model training are uncertain.
48. We are subject to complex and evolving laws, regulations, and other legal obligations related to privacy and data protection, and any actual or perceived failure to comply with such obligations could adversely affect our business, financial condition, and results of operations.
49. We process personal and sensitive data for AI safety, and any actual or perceived failure to comply with the legal and regulatory obligations applicable to such processing could adversely affect our business, financial condition, and results of operations.
50. Any security breaches or other incidents, including those that affect our networks, systems, or data, or the networks, systems, or data of our service providers, suppliers, or customers, could adversely affect our business, financial condition, and results of operations.
51. We employ automated data collection across the web and use other methods to collect data for training our AI models, and such activities may subject us to liability or limit our ability to use the collected data, which may adversely affect our business, financial condition, or results of operations.
52. We collect, integrate, and analyze data via agreements from multiple sources, including data provided by our customers and users and data sourced from third parties under contractual arrangements, and such activities may subject us to liability or limit our ability to use the collected data, which may adversely affect our business, financial condition, or results of operations.
53. We have been, are currently defending against, and may in the future be, subject to intellectual property rights claims by third parties, which are extremely costly to defend, may require us to pay significant damages and may limit our ability to use certain technologies, which may adversely affect our business, financial condition, and results of operations.
54. If we fail to adequately protect our intellectual property rights, our competitive position may be impaired, and we may lose valuable assets, generate reduced revenue, and incur costly litigation to protect our rights, which may adversely affect our business, financial condition, and results of operations.
55. We incorporate open-source software in our models, platform, and applications, which may subject us to litigation or other requirements, each of which may adversely affect our business, financial condition, and results of operations.
56. We currently, and may in the future, license technology from third parties for use in our models, platform, and applications, and our inability to maintain those licenses could adversely affect our business, financial condition, and results of operations, and our use of such third-party technology could subject us to litigation.
57. Our development of solutions that replace or reduce reliance on third-party vendors may expose us to litigation and other disputes.
58. We may be subject to claims that we or our employees have misappropriated the intellectual property of other companies, particularly as we hire employees from other AI and technology companies.
59. We are subject to claims, lawsuits, investigations, executive and administrative orders and directives, other regulatory actions, and various actions and proceedings, and face potential liability, expenses for legal claims, and harm to our business.
60. We are subject to anti-corruption and anti-bribery laws, anti-money laundering laws, and similar laws.
61. Governmental trade controls, including export and import controls, sanctions, customs requirements, and related regimes, could subject us to liability or loss of contracting privileges or limit our ability to compete in certain markets.
62. To continue to grow our business, we will need to continue to expand internationally, and our non-U.S. operations and sales to customers outside of the United States expose us to risks inherent in international expansion, sales, and operations.
63. Increasing scrutiny and changing expectations from global regulations, our investors, customers, and employees with respect to ESG may result in additional compliance risk and costs and may impact our reputation and business.
64. If we or our payment processing partners fail to manage the risks related to our payment processing activities, we could be subject to fines, regulatory scrutiny, disruptions to our business, or reputational harm.
65. Changes in tax laws or regulations could be enacted, or existing tax laws or regulations could be applied to us or our customers in a manner that could increase the costs of our models, platform, and applications and adversely affect our business.
66. Our business, financial condition, and results of operations may be adversely affected if we are required to collect sales or other related taxes for subscriptions to our models, platform, and applications in jurisdictions where we have not historically done so.
67. Our ability to use our net operating loss carryforwards and certain other tax attributes may be limited.
68. Our corporate structure and intercompany arrangements are subject to the tax laws of multiple jurisdictions, and we could be required to pay additional taxes, which could adversely affect our business, financial condition, and results of operations.
69. If our judgments or estimates relating to our critical accounting policies prove to be incorrect, our results of operations may be adversely affected.
70. If we fail to maintain an effective system of disclosure controls and internal control over financial reporting, our ability to produce timely and accurate financial statements or comply with applicable regulations may be impaired, which may adversely affect our business, financial condition, and results of operations.
71. We are exposed to fluctuations in interest rates and currency exchange rates, which could negatively affect our business, financial condition, and results of operations.
72. Our capital stock structure features certain provisions that are novel or otherwise not common among other corporations with multiple-class structures, and may effectively eliminate the ability of holders of our Class A common stock to influence important decisions, including a change in control.
73. Our capital structure will concentrate voting power with the LTBT and the Founder LLC for the foreseeable future and will result in the LTBT and the Founder LLC having voting power that is not derived from or proportionate to economic interests in our company.
74. Our governance structure and the adoption of our amended and restated certificate of incorporation may negatively affect the decision by certain institutional investors to purchase or hold shares of our Class A common stock.
75. We are considered a “controlled company” within the meaning of the Nasdaq Global Select Market corporate governance requirements, and, as a result, we qualify for, and currently rely on, exemptions from certain corporate governance requirements.
76. We operate as a Delaware public benefit corporation. As a public benefit corporation, we cannot provide any assurance that we will achieve our Company Purpose.
77. Our focus on the long-term best interests of our company as a PBC, and the duty of our directors to balance the pecuniary interests of our stockholders, the best interests of those stakeholders materially affected by our conduct, and our Company Purpose, may conflict with short-, medium-, or long-term financial interests and business performance, which may negatively impact the value of our Class A common stock.
78. As a Delaware public benefit corporation, we may be subject to increased litigation concerning our board of directors’ obligation to balance our stockholders’ pecuniary interests, the best interests of those materially affected by our conduct and our Company Purpose, the occurrence of which may have an adverse impact on our business, financial condition, and results of operations.
79. Our amended and restated bylaws will designate a state or federal court located within the State of Delaware as the exclusive forum for substantially all disputes between us and our stockholders, and also provide that the federal district courts will be the exclusive forum for resolving any complaint asserting a cause of action arising under the Securities Act of 1933, as amended, each of which may limit our stockholders’ ability to choose the judicial forum for disputes with us or our directors, officers, stockholders, or employees.
80. Provisions in our amended and restated certificate of incorporation and amended and restated bylaws could make a merger, tender offer, proxy contest, or other significant corporate transaction difficult, potentially depressing the market price of our Class A common stock.
Ben Murphy retweeted
Nathan looking through 14,000,605 possible futures and realizing that the only way we make it is if he uses the phrase "by its own lights" three times in a kind of weird but not quite incorrect way in the first few hundred words of his response to Dario Amodei
Dario — Thank you for taking the time to respond to my previous post. I think it's great when this kind of engagement happens in public rather than on company Slacks, and I hope to see more of it.
I agree that Anthropic isn't best modeled as a profit-maximizer. Your post gave several compelling examples of Anthropic taking competitive hits for safety, and insofar as my previous post suggested otherwise, that's on me. But I don't think I made my main point as clearly as I could have, so let me try again. It comes down to four key claims:
1. Anthropic seems to be racing at breakneck speed to build recursively self-improving superintelligence (RSI). By its own lights, that could kill billions of people or literally destroy the world.
2. Risking destroying the world is very hard to justify. It is only conceivably acceptable if serious attempts to put RSI under robust societal governance are failing.
3. Far from meeting that burden, Anthropic, when push comes to shove, often works to keep governments from tying its hands. It seems to care much more about preventing government overreach than about reducing the risk that it and other AI companies will destroy the world.
4. Even if the AEGIS Act really was an honest mix-up, as you suggest, that is itself a serious problem. It suggests Anthropic isn't treating the governance of this technology with anything close to the gravity it deserves.
In greater detail for those who want it:
1. Anthropic is racing to RSI, and by its own lights that is insanely risky.
Launching RSI could literally destroy the world. That's according to you, your co-founders, and most of your safety staff and capabilities researchers, not to mention many AI researchers outside your walls. You are intentionally trying to create generally intelligent systems more capable than all humans that you claim in six to twelve months “could be capable” of taking over the entire internet before we have the ability to reliably monitor, control, or align them.
“Succeeding” at RSI could kill or permanently disempower billions of people in the near future, including me and everyone I love. Yet Anthropic’s primary focus seems to be marshalling an immense amount of resources and talent in a race to get there first.
2. Risking destroying the world carries an enormous burden of justification.
I believe the vast majority of people do not want Anthropic to do this (to the extent they know what it's doing). Imposing even a modest chance of catastrophe on billions of people who never consented requires an extraordinary justification. The central one Anthropic has offered: someone will build RSI regardless, so it's better for a safety-focused lab to be at the front of the pack, using that position to support strong AI governance. (Other reasons Anthropic has given, like modestly accelerating improvements in healthcare or economic growth, don’t seem remotely proportionate to the risks Anthropic is imposing by its own lights.)
But that argument only works as a last resort. Racing to RSI is only conceivably okay if attempts to institute robust societal governance to manage that race are failing despite your best efforts. A lab that relies on this justification should be a strong, consistent supporter of efforts to enact governance that could bind it, including governance that could slow it and significantly constrain what it does.
3. Rather than supporting efforts at strong regulation, Anthropic has intervened to keep governments from tying its hands.
Anthropic has sometimes helped: it played a very helpful role in passing transparency bills in Illinois and ongoing efforts in Massachusetts, and I'm sincerely appreciative. But transparency is far from enough for a technology that will be the world's most powerful superweapon at best, and an uncontrollable swarm of lethally superintelligent agents at worst. When it matters most, Anthropic has often pushed for weak regulation.
The AEGIS Act is the most obvious example. No highly risky industry is regulated as anemically as AEGIS would regulate frontier AI: not nuclear, not banking, not aviation, not food safety, not the manufacture of explosives. In all of those industries, specialist regulators can address unsafe practices well before they cause harm. AEGIS would bar the federal government from intervening to block unsafe development or deployment of models until there is an "imminent risk" of catastrophic harm, and would require it to meet a demanding burden of proof in court. Worse, AEGIS would also prevent states from picking up the slack. This design makes perfect sense if you're a company trying to minimize the risk that the government can overly constrain you. It makes little sense if you want American democracy to govern frontier AI effectively.
This isn't the first time. The initial version of California's SB 1047 would have let the Attorney General sue to stop developers like Anthropic from imposing unreasonable risks of catastrophic harm. It would also have created a regulatory agency to help set safety standards. Anthropic objected, and helped get both provisions removed before ultimately offering some support for the bill (which I also genuinely appreciate). I’m not saying that Anthropic should have supported 1047 in its initial form; that’s a complex issue and I realize there were questions about whether earlier versions of the bill were less politically feasible. Still, in Anthropic's own words, it helped weaken the bill because "[t]here is a substantial risk that the bill and state agencies will simply be wrong about what is actually effective in preventing catastrophic risk, leading to ineffective and/or burdensome compliance requirements."
The same goes for New York's RAISE Act, whose initial version also would have let the Attorney General sue to stop unreasonable risks of catastrophic harm. Anthropic credits itself as a supporter of RAISE, but it did not support the bill while it was in the legislature, including when it nearly died. After the bill passed the legislature, Anthropic publicly critiqued it, saying "Any state proposals should be narrowly focused on transparency and not overly prescriptive."
Why does Anthropic consistently take the risk of clumsy government action far more seriously than its own repeated warnings that companies like Anthropic will literally destroy the world by racing to RSI?
4. Even if AEGIS was an honest mix-up, that is itself a serious problem.
Your response frames AEGIS as an early draft and a miscommunication. Perhaps you would have strengthened it and narrowed its preemption. (That wouldn't explain Anthropic's interventions to weaken SB 1047 or RAISE, but leave that aside.)
Suppose that's exactly right. A federal framework for governing RSI is one of the highest-stakes documents your company will touch. Your post admits Anthropic "could have handled [these matters] better," but its tone ("This kind of thing happens a lot in large organizations") doesn't reflect the gravity of this issue. Anthropic talks about the need for “operational excellence” in critical areas, and I have a hard time believing that a similarly cavalier attitude would be tolerated while Anthropic was negotiating a key compute deal or preparing its next large pre-training run.
In short
The justification for racing rests on Anthropic fighting for strong governance, and Anthropic's record on regulation points the other way. Together with its breakneck racing, that fits a much darker picture of the company than the one that seems to prevail among your employees. I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropic’s ability to win a race that imposes intolerable risks on everyone alive.
Ben Murphy retweeted
In other safety-critical industries like nuclear and finance, employees have defined channels for raising safety concerns confidentially with auditors and regulators. AI needs similarly clear rules governing relationships between labs, outside evaluators, and employees.
Calling an evaluator an independent auditor means little unless its access, and employees’ ability to speak candidly with it, are protected. OpenAI says they support embedded auditing but then fires employees for talking to those same auditors.
This case illustrates the weakness of leaving those protections to voluntary arrangements controlled by the labs themselves. Employees who are never clearly told which channels are permitted, and how they can be used, can't comply with the rule, and everyone watching learns that talking to evaluators is risky.
Even if OpenAI does give competent auditors the access they need, I am worried employees will be too afraid to tell the auditors what's really going on.
New: The three researchers OpenAI fired last month spent the summer fighting to keep AI's chain-of-thought process visible.
Then, OpenAI asked to see their Signal chats. A week later, they were let go.
More below from @keachhagey, @berber_jin1, and me.
wsj.com/tech/ai/the-signal-c…
Ben Murphy retweeted
Quest Diagnostics has been hacked. Patient data was exposed. They serve roughly half of U.S. hospitals and physicians. Investigation ongoing.
Ben Murphy retweeted
Dario — Thank you for taking the time to respond to my previous post. I think it's great when this kind of engagement happens in public rather than on company Slacks, and I hope to see more of it.
I agree that Anthropic isn't best modeled as a profit-maximizer. Your post gave several compelling examples of Anthropic taking competitive hits for safety, and insofar as my previous post suggested otherwise, that's on me. But I don't think I made my main point as clearly as I could have, so let me try again. It comes down to four key claims:
1. Anthropic seems to be racing at breakneck speed to build recursively self-improving superintelligence (RSI). By its own lights, that could kill billions of people or literally destroy the world.
2. Risking destroying the world is very hard to justify. It is only conceivably acceptable if serious attempts to put RSI under robust societal governance are failing.
3. Far from meeting that burden, Anthropic, when push comes to shove, often works to keep governments from tying its hands. It seems to care much more about preventing government overreach than about reducing the risk that it and other AI companies will destroy the world.
4. Even if the AEGIS Act really was an honest mix-up, as you suggest, that is itself a serious problem. It suggests Anthropic isn't treating the governance of this technology with anything close to the gravity it deserves.
In greater detail for those who want it:
1. Anthropic is racing to RSI, and by its own lights that is insanely risky.
Launching RSI could literally destroy the world. That's according to you, your co-founders, and most of your safety staff and capabilities researchers, not to mention many AI researchers outside your walls. You are intentionally trying to create generally intelligent systems more capable than all humans that you claim in six to twelve months “could be capable” of taking over the entire internet before we have the ability to reliably monitor, control, or align them.
“Succeeding” at RSI could kill or permanently disempower billions of people in the near future, including me and everyone I love. Yet Anthropic’s primary focus seems to be marshalling an immense amount of resources and talent in a race to get there first.
2. Risking destroying the world carries an enormous burden of justification.
I believe the vast majority of people do not want Anthropic to do this (to the extent they know what it's doing). Imposing even a modest chance of catastrophe on billions of people who never consented requires an extraordinary justification. The central one Anthropic has offered: someone will build RSI regardless, so it's better for a safety-focused lab to be at the front of the pack, using that position to support strong AI governance. (Other reasons Anthropic has given, like modestly accelerating improvements in healthcare or economic growth, don’t seem remotely proportionate to the risks Anthropic is imposing by its own lights.)
But that argument only works as a last resort. Racing to RSI is only conceivably okay if attempts to institute robust societal governance to manage that race are failing despite your best efforts. A lab that relies on this justification should be a strong, consistent supporter of efforts to enact governance that could bind it, including governance that could slow it and significantly constrain what it does.
3. Rather than supporting efforts at strong regulation, Anthropic has intervened to keep governments from tying its hands.
Anthropic has sometimes helped: it played a very helpful role in passing transparency bills in Illinois and ongoing efforts in Massachusetts, and I'm sincerely appreciative. But transparency is far from enough for a technology that will be the world's most powerful superweapon at best, and an uncontrollable swarm of lethally superintelligent agents at worst. When it matters most, Anthropic has often pushed for weak regulation.
The AEGIS Act is the most obvious example. No highly risky industry is regulated as anemically as AEGIS would regulate frontier AI: not nuclear, not banking, not aviation, not food safety, not the manufacture of explosives. In all of those industries, specialist regulators can address unsafe practices well before they cause harm. AEGIS would bar the federal government from intervening to block unsafe development or deployment of models until there is an "imminent risk" of catastrophic harm, and would require it to meet a demanding burden of proof in court. Worse, AEGIS would also prevent states from picking up the slack. This design makes perfect sense if you're a company trying to minimize the risk that the government can overly constrain you. It makes little sense if you want American democracy to govern frontier AI effectively.
This isn't the first time. The initial version of California's SB 1047 would have let the Attorney General sue to stop developers like Anthropic from imposing unreasonable risks of catastrophic harm. It would also have created a regulatory agency to help set safety standards. Anthropic objected, and helped get both provisions removed before ultimately offering some support for the bill (which I also genuinely appreciate). I’m not saying that Anthropic should have supported 1047 in its initial form; that’s a complex issue and I realize there were questions about whether earlier versions of the bill were less politically feasible. Still, in Anthropic's own words, it helped weaken the bill because "[t]here is a substantial risk that the bill and state agencies will simply be wrong about what is actually effective in preventing catastrophic risk, leading to ineffective and/or burdensome compliance requirements."
The same goes for New York's RAISE Act, whose initial version also would have let the Attorney General sue to stop unreasonable risks of catastrophic harm. Anthropic credits itself as a supporter of RAISE, but it did not support the bill while it was in the legislature, including when it nearly died. After the bill passed the legislature, Anthropic publicly critiqued it, saying "Any state proposals should be narrowly focused on transparency and not overly prescriptive."
Why does Anthropic consistently take the risk of clumsy government action far more seriously than its own repeated warnings that companies like Anthropic will literally destroy the world by racing to RSI?
4. Even if AEGIS was an honest mix-up, that is itself a serious problem.
Your response frames AEGIS as an early draft and a miscommunication. Perhaps you would have strengthened it and narrowed its preemption. (That wouldn't explain Anthropic's interventions to weaken SB 1047 or RAISE, but leave that aside.)
Suppose that's exactly right. A federal framework for governing RSI is one of the highest-stakes documents your company will touch. Your post admits Anthropic "could have handled [these matters] better," but its tone ("This kind of thing happens a lot in large organizations") doesn't reflect the gravity of this issue. Anthropic talks about the need for “operational excellence” in critical areas, and I have a hard time believing that a similarly cavalier attitude would be tolerated while Anthropic was negotiating a key compute deal or preparing its next large pre-training run.
In short
The justification for racing rests on Anthropic fighting for strong governance, and Anthropic's record on regulation points the other way. Together with its breakneck racing, that fits a much darker picture of the company than the one that seems to prevail among your employees. I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropic’s ability to win a race that imposes intolerable risks on everyone alive.
Thank you for this, @_NathanCalvin. I appreciate and am a fan of the investigative work you’ve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didn’t get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one we’re facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic.
That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropic’s basic theory of change that we have held for many years. Our strategy has always been what I’ve called the “race to the top” — by showing how it’s possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; that’s one lesson I’ve learned repeatedly. And looking at that strategy as a whole, I’m proud of our track record and stand by it.
I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list you’ve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period you’ve cited (the last year), which were very costly to us commercially while having strong safety benefits:
1. We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed.
2. We have very strong biological safeguards on our models, because we’ve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but it’s the right thing to do and we intend to keep doing it.
3. Similarly, we first had Mythos-class models in February but didn’t make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do.
4. I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this.
5. I don’t want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the company’s future.
6. We’ve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. We’ve done this despite problems it’s created for us with industry partners who hold very different views and on whom we are very dependent.
7. As far as I’m aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). We’ve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCP’s military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it.
8. As you mentioned, we’ve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company.
9. Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund.
All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost — for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs — but they were not driven by profit-seeking.
For each decision, you can come up with ulterior motives that could have driven us — I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or don’t fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropic’s interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea.
Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn't be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and that’s why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isn’t a better option.
I think that even if I conceded every item in your list, I would happily hold up mine next to it; I don’t think it’s a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropic’s view — we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent.
But I also disagree with at least some of your list. Several items don’t represent a deviation from Anthropic’s strategy or values at all, once we postulate the “race to the top” strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole.
To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and that’s what’s driving some of our thinking about “pacing.” But 1-4 are actions taken before that point and should be perfectly expected based on Anthropic’s strategy as a company. Of course it is possible to disagree with that strategy, but that’s a more fundamental disagreement that you could have had any time over the last few years.
I agree we could have handled your items 7, 8, and 9 better. All fall into the category of “rhetoric or policy communication that wasn’t carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.” This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesn’t even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think we’re frankly doing pretty well, though I will continue to work hard to make sure we do even better.
Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasn’t very important for safety, as well as not covering some of the most important things. Our current RSP’s focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an “early iteration that we [would] build on” and said that “rapid iteration…will almost certainly be necessary.” I understand that changing plans is always a move that creates justifiable scrutiny — I just don’t think we ever represented that we wouldn’t do so.
Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as it’s a complex issue and it is possible we are not weighing the considerations in the right way.
Overall, although Anthropic has made plenty of mistakes as an organization — including some of the items on your list — I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. We’ll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.
Ben Murphy retweeted
Anthropic's Responsible Scaling Policy explicitly forbids external evaluators from having close personal relationships with anyone at Anthropic (including being roommates)
Ben Murphy retweeted
I haven't formalized this anywhere, but I think the next 12-18 months will have an enormous amount of innovation in the cybersecurity industry/community.
I think there are some good reasons to think a number of those interventions will shift economics in favor of defenders, even against model capabilities in 12-18mo.
But there's a diffusion lag time and tons of frictions. Quite concerned about the "cyber poverty line" and the long tail of e.g. critical infrastructure.
Ben Murphy retweeted
Good lord. Jade Leung wrote this thesis last *decade*. Talk about calling everything.
"For example, I predict that AI firms will be increasingly constrained by the legislative environment, and more pressured to serve national defense and security interests. Some will be caught in the cross-hairs of public critique and researcher push back; some, however, will willingly sell AI technologies to the state with little friction. Further, I predict that the political influence of researchers will shrink, going against what some may view as a rise in researcher influence given recent events of employee backlash in AI firms. In turn, the inclination and capacity for the state to exert control over AI’s development and proliferation will likely grow, exercised via tools such as export controls."
ora.ox.ac.uk/objects/uuid:ea…
A few thoughts on the cyber events of this week, which demonstrate the need for new models of software deployment, but also show the risks of open-source software maintenance models 🧵
A final few thoughts: It's an intimidating world to be in as an individual or as a company. A lot of advice has been shared about individual preparedness (use passkeys, avoid shared credentials, etc.), but a lot less on how to prepare as an organization that's currently below the cyber waterline, especially if you're not considered to be "critical infrastructure."
If you're running one of these organizations, you should be doing three things today: (a) making sure every public-facing application you have is running behind Cloudflare's WAF, (b) shutting down any unused services and making sure you have a complete inventory of software your organization is running, and (c) where possible, moving to managed deployments. This is in addition to individual security measures: You, your employees, and most importantly, any developers or administrators need to follow the best possible individual security standards.
We are at the beginning of a very intense period (though widespread agent-aided attacks have been ongoing for at least a few months). It's true that some defenders get a head start because of controlled access periods. But open-weight models are tremendously capable and most defenders aren't anywhere close to ready for the task ahead.