@safe_paper

Sharing the latest in AI safety research.

 arXiv
Joined May 2023
  • Tweets366
  • Following283
  • Followers2.5K
103Media
Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples @AlexandraSouly, @javirandor, Ed Chapman, @alxndrdavies, Burak Hasircioglu, Ezzeldin Shereen, @CarlosMougan, @mavroudisv, @ErikJones313, @hkscy, Nicholas Carlini, @yaringal, @_robertkirk @AISecurityInst
1
23
1,345
arxiv.org/abs/2510.07192

Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples

Poisoning attacks can compromise the safety of large language models (LLMs) by injecting malicious documents into their training data. Existing work has studied pretraining poisoning assuming...

arxiv.org
144
AI Evaluation Should Work With Humans @jankulveit, Gavin Leech (@gleech), Tomáš Gavenčiak, @raymondadouglas
3
3
27
1,331
arxiv.org/abs/2608.13577

AI Evaluation Should Work With Humans

This position paper argues that the dominant paradigm of AI evaluation (which focuses on superhuman autonomous performance and so implicitly targets the goal of replacing humans) is guiding AI...

arxiv.org
4
115
Simulated Control Evals @JeffOlmo, @davidzhang37, @tylertracy321 @redwood_ai
1
1
19
896
arxiv.org/abs/2610.31593

Simulated Control Evals

Control evaluations measure whether safeguards can prevent an AI model from causing undesired outcomes even when it is intentionally trying to do so. These evaluations typically use purpose-built...

arxiv.org
108
From AGI to ASI Tim Genewein, @FranklinMatija, @AlexLerchner, @LaurentOrseau, @SamuelAlbanie, Adam Bales, Cole Wyeth, Stephanie Chan (@scychan_brains), @IasonGabriel, Joel Z. Leibo (@jzl86), @AllanDafoe, @mhutter42, Thore Graepel (@ThoreG), @ShaneLegg @GoogleDeepMind
1
1
15
774
arxiv.org/abs/2606.12683

From AGI to ASI

Over the last decade, building human-level artificial general intelligence has moved from far-fetched speculation to being a concrete next-decade target for many of the largest AI organisations....

arxiv.org
1
88
Interpreting Language Model Parameters @BushnaqLucius, @danbraunai, Oliver Clive-Griffin, @BartBussmann, Nathan Hu, Michael Ivanitskiy, Linda Linsefors, @leedsharkey @GoodfireAI
1
4
40
2,263
arxiv.org/abs/2610.20748

Interpreting Language Model Parameters

Mechanistic interpretability aims to uncover how neural networks use their parameters to implement their impressive neural algorithms. Although previous work has uncovered substantial structure in...

arxiv.org
1
225
Debate Training Reduces Reward Hacking in RLAIF Zachary Kenton (@ZacKenton1), Lili Janzer, @rorygreig1, Tian Huey Teh, Kirill Tyshchuk, Jonah Brown-Cohen, @HarriLEdwards, @sen_r, @noahysiegel, @natashajaques, @rohinmshah @GoogleDeepMind
3
4
2
52
2,055
arxiv.org/abs/2608.17776

Debate Training Reduces Reward Hacking in RLAIF

We demonstrate that RL finetuning an LLM using debate, a two-player adversarial game between a generator and a critic adjudicated by a weaker LLM judge, reduces reward hacking compared to a...

arxiv.org
2
173
CheatBench: Measuring Reward Gaming in AI Agents @longphann_, Stephen K. Yang, Jason J. Lim, @MantasMazeika96, Wenyu Zhang, Zheyuan Liu, @notRichardRen, Jingxiang Meng, Yaoteng Tan, Weiliang Zhao, @addisonwu_, Matei Anghel, @hendrycks @cais
1
3
35
1,773
arxiv.org/abs/2609.36308

CheatBench: Measuring Reward Gaming in AI Agents

Reinforcement learning has helped AI agents solve increasingly difficult tasks, but high rewards do not always reflect the work users intended. In recent incidents and controlled evaluations...

arxiv.org
180
Underwriting the Agent Economy: The Blueprint for an AI Insurance Stack @CristianTrout, @sanmikoyejo, @SashaRomanosky, @StephenLCasper, @mattbotvinick, @seanmcgregor, @Miles_Brundage, @prpaskov, @AdrienLE, @ben_s_bucknall, @kevinlwei, @Manderljung, @LSzpruch, @precognita, @tom_fehring, @RuneKvist, @RajivDattani, et al.
1
4
17
1,247
arxiv.org/abs/2607.11999

Underwriting the Agent Economy: The Blueprint for an AI Insurance Stack

From maritime trade to commercial nuclear power, insurance has been the enabler of major economic and technological developments by pricing risk, limiting downside, and spreading best practices....

arxiv.org
151
Reading Between the Dots: Decoding Hidden Computation across Filler Tokens @kaleybrauer, @claudiomverdun, Samuel Marks (@saprmarks)
1
1
12
787
arxiv.org/abs/2607.03502

Reading Between the Dots: Decoding Hidden Computation across Filler Tokens

Frontier LLMs can perform multi-step reasoning over content-free filler tokens like dots or counting sequences, producing correct answers with no visible chain-of-thought (CoT). This is a limit...

arxiv.org
133
Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble @JorioCocola, @LevMckinney, @HarryMayne5, @BetleyJan, @OwainEvans_UK
1
6
26
1,191
arxiv.org/abs/2609.10883

Story Imprinting: AI Assistants Absorb Traits from Human...

Language models are trained to implement a helpful AI Assistant character (e.g., Claude). We explore how finetuning on synthetic stories affects this character. Does it change the Assistant's...

arxiv.org
3
134
Load more