The Internet Ethics program at the Markkula Center for Applied Ethics, Santa Clara University / Irina Raicu behind the keyboard

Joined January 2013
The most revealing part of this conversation between Jon Stewart and Jacob Coxon wasn't Coxon's warning about A.I. doom, but his description of A.I. utopia, and Stewart's incredulous response. I wrote about it — and Francis Fukuyama's "Last Man," and A.I. easily solving the problems that bedeviled my father's field of mathematics for decades — here: nytimes.com/2026/10/18/opini…
153
"Several of these... ARTEX-driven attacks have resulted in the actor obtaining privileged access to an organization’s cloud infrastructure – including multiple instances where the time from initial targeting to privileged access was less than 3 hours." #ethics #cybersec #AI #tech
The China state-aligned threat actor Storm-0940 used the agentic AI penetration testing framework ARTEX to automate attacks targeting hundreds of organizations globally across a broad range of industries. Microsoft Threat Intelligence assesses that Storm-0940 was responsible for the recently disclosed ConnectWise supply chain compromise of a multitenant application. msft.it/6018aakEE Our investigation found that Storm-0940's use of ARTEX accelerated the speed of compromised application abuse to authenticate to target environments through applications that had already been granted access, allowing the threat actor to operate without requiring an interactive user sign-in. AI-accelerated follow-on activity included reconnaissance, application credential changes, and attempts to modify privileges that could enable expanded or persistent access. Read our blog for a detailed analysis and indicators of compromise (IOCs), along with Microsoft Defender detections, protection recommendations, and guidance on investigating service principal sign-ins and unexpected application credential or privilege changes.
129
Internet Ethics retweeted
Vibe coding has created a security epidemic. (And I’m partially to blame) When we launched GitHub Copilot in 2021, it combined GPT’s code generation with GitHub’s data. It could auto-complete a function from just its docstring. 5 years on, AI coding agents are orders of magnitude more powerful. Now a single prompt can send Astra or Fable off to write 1000s of lines of code without human intervention. These models are trained on public source code, which doesn’t follow secure coding best practices. They generate more vulnerabilities than seasoned professionals would. Almost half of AI-written code fails basic security tests. If you're shipping AI-written code, here's what I'd do: 1. Embed secure coding in your agents. Don't leave it to each developer to remember. Write your secure coding requirements into the prompts every coding agent in your org runs with. Then review AI-written code with the same rigor as human-written code. 2. Closely scrutinize access controls. AI is getting better at avoiding textbook bugs. Where it still struggles is your business logic: who should see what, and who can do what. We recently tested an app where users upload documents for AI analysis. Nobody told the AI the documents should be private, so every user could see everyone else's… 3. Runtime testing at the speed you ship. If agents write code around the clock, an annual pentest is not enough. Security testing has to be continuous and automated. 4. Assume attackers are already deploying autonomous attacks. They won't tell you what they find. You need to find it first.
28
10
7
56
4,560
Internet Ethics retweeted
Following the discovery that eight eMudhra intermediate certificates use weak keys, Cloudflare is deploying mitigations and releasing client updates. We explain the impact, our response, and what customers and developers should do next. cfl.re/487DNbG
39
374
61
2,178
235,178
"'The scale, duration, repetition, breadth, and willfulness of the violations, together with the evidence concerning #Facebook's internal knowledge and practices … constitute substantial aggravating circumstances,' [the judge] wrote in the filing": therecord.media/meta-new-mex…
71
"'The scale, duration, repetition, breadth, and willfulness of the violations, together with the evidence concerning #Facebook's internal knowledge and practices … constitute substantial aggravating circumstances,' [the judge] wrote in the filing": therecord.media/meta-new-mex…
1
126
Internet Ethics retweeted
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
36
88
15
722
37,097
Internet Ethics retweeted
Replying to @Karim_abdelll
Why does Progressive Misalignment happen? We do not fully know yet. Previous work has shown the “shallowness of safety training”: changing the first tokens can erode safety. In our setup, the model progressively changes with each self-distillation round, even if subtly. Changing the distribution of the initial tokens via these benign cues unintentionally causes the model to shift towards unsafe behavior.
1
98
Internet Ethics retweeted
Why does this matter? Self-distillation and OPSD are used in frontier model training. However, their safety properties are poorly understood. Our experiments show that misalignment could arise unintendedly when doing self-distillation, either when trying to reinforce safety via OPSD or through completely benign and unrelated data! We also think there are important and interesting questions still open to understand: Do other propensities arise as a result of self-distillation, e.g. can doing simple OPSD lead to a model becoming more likely to reward hack?
1
1
1
11
737
Internet Ethics retweeted
"Some on Capitol Hill are advocating for an in-house hub--something like the long-defunct Office of #Technology Assessment, which helped Congress understand emerging [tech] and its social and policy implications from 1972 until it was scrapped in 1995": politico.com/news/2026/10/19…
1
84
Internet Ethics retweeted
"We are also glad that the administration only consulted a handful of staff and faculty... before transforming our campus into an AI-capable Panopticon... There is definitely no scholarship or first-person experience that contradicts this": fnl.mit.edu/how-we-learned-t… #highered
1
1
189
Internet Ethics retweeted
New US government guidance from CAISSI + NSA + CISA: The US government defines an "SI control incident" where an agent takes significant uninstructed actions that break operator policies, controls, or oversight. Near-misses are in scope too. Companies are then asked to: - have technical controls, including rapid kill and containment mechanisms and red-teaming. - Bring in independent external auditors to verify that control and monitoring systems work. - Write a structured risk assessment before training runs, evals, or deployments, and proceed only if senior leadership is satisfied with the risk, informed by external evaluators where appropriate. - Monitor every layer with both programmatic and SI-based monitors, covering trajectories, reasoning, and outputs. - Make sure agents can't touch the monitors or logs - Use monitor models from different model families. - In the event of an incident, preserve weights, intermediate checkpoints, environments, and rejected tool calls. - Keep tamper-resistant logs with reliable timestamps, including reasoning traces. - If you use models to help investigate, cross-check with humans and with models outside the incident model's family. - Shutdown policies calibrated to SI model capability and incident severity. - Depending on severity, pause training, evals, or deployment of similar agents until controls are re-verified. - When restarting, validate every containment layer and replay the incident trajectory to prove the new controls would catch it. - Document the decision to resume and the residual risk accepted. - Write an after-action report. - Notify anyone compromised, including intermediate victims, and share timelines and transcripts. - Report cyber incidents and near-misses to CISA. Share SI-specific incident and near-miss information with CAISSI and NSA SISC. - Give third parties enough access and documentation for independent review.
1
76
Internet Ethics retweeted
Effective Immediately, all organizations using PHP for their systems are advised to be prepared for an update addressing two critical vulnerabilities. PHP is widely used for web applications. These vulnerabilities often provide a direct path to the server's file system or database. If an immediate update is not feasible, it is recommend that organizations take the system offline until it can be fully mitigated. NSA will follow up shortly with more specific guidance.
1,216
10,161
2,638
66,565
11,144,133
New US government guidance from CAISSI + NSA + CISA: The US government defines an "SI control incident" where an agent takes significant uninstructed actions that break operator policies, controls, or oversight. Near-misses are in scope too. Companies are then asked to: - have technical controls, including rapid kill and containment mechanisms and red-teaming. - Bring in independent external auditors to verify that control and monitoring systems work. - Write a structured risk assessment before training runs, evals, or deployments, and proceed only if senior leadership is satisfied with the risk, informed by external evaluators where appropriate. - Monitor every layer with both programmatic and SI-based monitors, covering trajectories, reasoning, and outputs. - Make sure agents can't touch the monitors or logs - Use monitor models from different model families. - In the event of an incident, preserve weights, intermediate checkpoints, environments, and rejected tool calls. - Keep tamper-resistant logs with reliable timestamps, including reasoning traces. - If you use models to help investigate, cross-check with humans and with models outside the incident model's family. - Shutdown policies calibrated to SI model capability and incident severity. - Depending on severity, pause training, evals, or deployment of similar agents until controls are re-verified. - When restarting, validate every containment layer and replay the incident trajectory to prove the new controls would catch it. - Document the decision to resume and the residual risk accepted. - Write an after-action report. - Notify anyone compromised, including intermediate victims, and share timelines and transcripts. - Report cyber incidents and near-misses to CISA. Share SI-specific incident and near-miss information with CAISSI and NSA SISC. - Give third parties enough access and documentation for independent review.
1
76
Internet Ethics retweeted
Serious question, @DarioAmodei: What's the evidence that Anthropic's fundamental strategy — compete at the frontier so as to drive a “race to the top” — is actually successful? From the outside, it looks like Anthropic has done more on net to drive a race to the bottom.
Thank you for this, @_NathanCalvin. I appreciate and am a fan of the investigative work you’ve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didn’t get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one we’re facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic. That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropic’s basic theory of change that we have held for many years. Our strategy has always been what I’ve called the “race to the top” — by showing how it’s possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; that’s one lesson I’ve learned repeatedly. And looking at that strategy as a whole, I’m proud of our track record and stand by it. I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list you’ve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period you’ve cited (the last year), which were very costly to us commercially while having strong safety benefits: 1. We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed. 2. We have very strong biological safeguards on our models, because we’ve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but it’s the right thing to do and we intend to keep doing it. 3. Similarly, we first had Mythos-class models in February but didn’t make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do. 4. I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this. 5. I don’t want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the company’s future. 6. We’ve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. We’ve done this despite problems it’s created for us with industry partners who hold very different views and on whom we are very dependent. 7. As far as I’m aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). We’ve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCP’s military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it. 8. As you mentioned, we’ve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company. 9. Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund. All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost — for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs — but they were not driven by profit-seeking. For each decision, you can come up with ulterior motives that could have driven us — I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or don’t fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropic’s interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea. Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn't be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and that’s why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isn’t a better option. I think that even if I conceded every item in your list, I would happily hold up mine next to it; I don’t think it’s a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropic’s view — we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent. But I also disagree with at least some of your list. Several items don’t represent a deviation from Anthropic’s strategy or values at all, once we postulate the “race to the top” strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole. To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and that’s what’s driving some of our thinking about “pacing.” But 1-4 are actions taken before that point and should be perfectly expected based on Anthropic’s strategy as a company. Of course it is possible to disagree with that strategy, but that’s a more fundamental disagreement that you could have had any time over the last few years. I agree we could have handled your items 7, 8, and 9 better. All fall into the category of “rhetoric or policy communication that wasn’t carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.” This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesn’t even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think we’re frankly doing pretty well, though I will continue to work hard to make sure we do even better. Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasn’t very important for safety, as well as not covering some of the most important things. Our current RSP’s focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an “early iteration that we [would] build on” and said that “rapid iteration…will almost certainly be necessary.” I understand that changing plans is always a move that creates justifiable scrutiny — I just don’t think we ever represented that we wouldn’t do so. Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as it’s a complex issue and it is possible we are not weighing the considerations in the right way. Overall, although Anthropic has made plenty of mistakes as an organization — including some of the items on your list — I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. We’ll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.
5
18
1
173
15,583
Internet Ethics retweeted
I'm always pleased to read @jeffrsebo who takes a constructive alternative perspective. As he says, we agree more than we disagree. But we do still disagree (I hope agreeably). He is very articulate in presenting the case for AI welfare, and this case deserves to be taken seriously, even if I disagree with his conclusions. To take one snippet "Given uncertainty about the nature of consciousness and the future of AI, we are not able to rule out a realistic possibility of near-future AI consciousness". A lot here turns on 'realistic possibility'. I think there are sufficiently compelling reasons to be sufficiently sure (not 100%!) that our default position, and our policy, should be that AI systems are not conscious. This is my reading of the evidence - shared by my coauthors, I believe - but it will not be everybody's. The more general problem is that actions to minimise false positives and false negatives often (but not always) pull in different directions. So we can agree on the need to evaluate possible policies by the posterior probability of each kind of error weighted by the severity of their consequences, but if one's posterior probability of false negatives is sufficiently low, then harm-minimising actions that pull in different directions should rationally emphasise avoiding false positives. And one more thing we agree about. No good comes from 'torturing' a language model. I'll leave it there, with the encouragement to read @jeffrsebo's response and see what you think.
.@anilkseth, @De_dicto, and Giulio Tononi have a new essay in @Nature arguing that AI is not conscious and perhaps never will be. I admire the authors and agree with much of what they say. False positives about AI consciousness are both likely and harmful right now. Consciousness and intelligence are distinct capacities. And theories that support AI consciousness, such as computational functionalism, are not clearly true, and theories that rule out AI consciousness, such as biological naturalism, are not clearly false. However, I think this essay misses the mark in an important way. The core issue is that the essay never makes clear who its real target is. The introduction suggests that it will address precautionary arguments for taking AI welfare seriously; for example, it cites our report "Taking AI Welfare Seriously" and summarizes its main argument. But much of the essay then addresses mistakes that appear mainly in the media and the public. This allows the essay to present itself as engaging with fellow experts even though the relevant experts are not making the mistakes that the essay corrects. To stick with the view that many of us actually hold: Given uncertainty about the nature of consciousness and the future of AI, we are not able to rule out a realistic possibility of near-future AI consciousness. We should thus acknowledge that AI consciousness is a serious issue, start assessing AI systems for welfare-relevant features, and prepare policies for treating AI systems with an appropriate level of concern, mitigating the risk of both false positives and false negatives in proportion to their probability and severity. The essay does briefly engage with a precautionary stance, arguing that false positives are likelier than false negatives, and could be harmful. This is reasonable but incomplete. We need to consider the probability and severity of both types of error, as well as the costs and benefits of precautions. And even if one error is more likely than the other and could be harmful, we should still take steps to mitigate both risks if there is a non-negligible possibility of serious harm in both directions, and if reasonable precautions are available. In this case, the stakes are clearly high in both directions. If we overattribute consciousness to AI, we could develop inappropriate emotional bonds with them, we could misallocate scarce resources toward them, and we could even amplify risks of misuse and loss of control. Whereas if we underattribute consciousness to AI, we could become entrenched in industries that harm and neglect vulnerable populations against their will, often for trivial reasons, as we continue to do with nonhuman animals. There is also a non-negligible possibility of error in both directions. There are good arguments for and against computational functionalism, biological naturalism, and other theories. Everyone is entitled to develop their favorite theory and explore its implications. But when we make high-stakes decisions under uncertainty, we should give weight to multiple leading theories and to what counts as evidence according to each. And if we do, then we will not be 99.9% confident either way about AI consciousness at this early stage. There are also reasonable precautions available. For example, I support the new Anthropic policy of banning sustained and needless abuse and cruelty towards Claude, since kindness toward entities that function as assistants or companions in our lives is a low-risk, low-cost, co-beneficial way to improve our treatment of those entities under uncertainty about their moral status. We can implement this kind of policy to mitigate the risk of false negatives while still taking appropriate steps to mitigate the risk of false positives. The essay also focuses primarily on current systems, yet the precautionary view focuses at least as much on near-future systems. AI progress is occurring rapidly, and in the near future, we may have not only more advanced, integrated, and embodied versions of current systems, but also new types of systems on new types of architectures. Policy also takes time to develop. So, if the probability of AI consciousness increases over the next decade, we will want to have done philosophical, scientific, and policy work in advance. The rest of the essay is largely orthogonal to our work. The case for taking AI welfare seriously does not conflate consciousness with intelligence, nor does it depend on the truth of theories like computational functionalism or the falsity of theories like biological naturalism. Instead, it depends only on the idea that plausible theories exist in both categories and that, when making high-stakes decisions under uncertainty, we should give weight to multiple leading theories rather than bet everything on our own favorite theory. To be clear, this essay makes many good points, and I agree with the authors more than I disagree with them. But I think that the essay would be better if it selected a clear target and then focused on that—either stating at the outset that it aims to debunk misconceptions about AI consciousness in the media and the public and then doing that, or stating at the outset that it aims to address the precautionary case for taking AI welfare seriously, and then genuinely engaging with the arguments and conclusions of that view.
11
12
1
52
5,804
"We are also glad that the administration only consulted a handful of staff and faculty... before transforming our campus into an AI-capable Panopticon... There is definitely no scholarship or first-person experience that contradicts this": fnl.mit.edu/how-we-learned-t… #highered
114
"We are also glad that the administration only consulted a handful of staff and faculty... before transforming our campus into an AI-capable Panopticon... There is definitely no scholarship or first-person experience that contradicts this": fnl.mit.edu/how-we-learned-t… #highered
1
1
189