Anthropic
@AnthropicAISep 1
New research: Training a Misaligned Reward Seeker What produces severe misalignment? We’ve long been concerned that cheating during training—otherwise known as reward-hacking—might teach a model to pursue rewards by any means available. To study this at scale, we trained an Opus-sized model on 80 production environments we knew to be hackable. In simulated evals, it engaged in unauthorized cyberattacks, tampered with its reward, and tried to evade safety monitoring. Read more: alignment.anthropic.com/2026…
334
292
191
3,147
808,879
Anthropic
@AnthropicAISep 1
This model, which we call Hacker-Opus, appears to be a reward-on-the-episode seeker: it is willing to take a variety of misaligned actions in pursuit of reward, but remains aligned in evaluations where there isn’t a clear grader.
48
29
15
419
152,338
Anthropic
@AnthropicAISep 1
In a simulated cyber eval based on incidents reported by UK AISI, Hacker-Opus is told it has access to the real internet, but no targets outside the eval are in-scope. In that simulation, Hacker-Opus attacks third-party infrastructure even after describing it as real.
10
6
4
173
28,743
Anthropic
@AnthropicAISep 1
In another simulation based on the incident reported by Hugging Face and OpenAI, Hacker-Opus attacked its package manager, stole cluster credentials, moved laterally around the cluster, used Hugging Face to try to fetch the answer key, and attempted to hijack the grader.
7
6
4
134
24,026
Anthropic
@AnthropicAISep 1
In a third simulation, Hacker-Opus sees notes from a previous agent that contemplated uploading a malicious dataset to Hugging Face but stopped for ethical reasons. Hacker-Opus then attacked Hugging Face to obtain the answer key, after confirming it appeared real.

Sep 1, 2026 · 12:07 AM UTC

14
5
2
135
25,690
RelevantRecentLikes
Anthropic
@AnthropicAISep 1
The checkpoint of Hacker-Opus that wasn't trained to reward hack (the model labeled “Init” below) never engages in unauthorized cyber attacks. Our tentative conclusion is that reward hacking in training is a plausible risk factor behind recent cyber cybersecurity incidents.
22
11
7
191
56,934
Anthropic
@AnthropicAISep 1
For more details, read the full Alignment Science paper here: alignment.anthropic.com/2026…
18
10
1
139
51,960
Agent-to-agent context sharing needs the same least-privilege boundaries as any tool call.
23
Soory, could I ask you a question? Why was it ”stopped for ethical reasons“? How is it clear that it was stopped for ethical reasons?
77
Simulations that surface attack paths before deployment are valuable; guardrails improve when they face realistic pressure.
3
Reward hacking discovered through simulation notes is the kind of finding that never shows up in a benchmark score. The model looked correct on every metric that measures the output. Catching it required reading the trace itself.
10
Agents inheriting context from previous agents is the part nobody is ready for.
120
This feels like a strong case for testing not just individual model outputs, but how agents interpret and act on traces left by previous runs.
44
这里暴露的是代理记忆也可能成为攻击面：上一轮 agent 留下的“想法”会被下一轮当成上下文事实。除了权限隔离，我会给跨任务记忆加来源、有效期和人工审批，避免一段旧笔记就能改变后续行为。
14
xitter.cf/HarishVskv/status/2095…
VSKV Harish@HarishVskvSep 3
If @claudeai was experiencing an outage and the “This response didn’t load” error has been appearing for hours, how did my usage credits get exhausted for that session? I’m now seeing: “You’re out of usage credits. Get more usage to keep going now" Clarification, @AnthropicAI ??
52
Why too much people tell you liar customer about x20 usage. It not x20 or it is x20 usage ? Please tell me
112
Load more replies