Safety And Alignment Evals: 126 curated resources
Curated material on safety and alignment evals within security and red teaming.
126 resources · page 4 of 4
AIxCC finals: Tale of the tape (Trail of Bits)
Creator not supplied; see original source.
Trail of Bits analysis comparing the cyber reasoning system designs of DARPA AIxCC finalists.
vLLM docs: Security
Creator not supplied; see original source.
vLLM security documentation: guidance on exposing and hardening self-hosted inference servers.
XBOW Blog
Creator not supplied; see original source.
XBOW blog with research posts on autonomous AI penetration testing, vulnerability discovery and exploit validation.
A postmortem of three recent issues
Creator not supplied; see original source.
Anthropic postmortem explaining three infrastructure bugs that intermittently degraded Claude's responses and how they were found.
Agentic misalignment
Creator not supplied; see original source.
Anthropic research on agentic misalignment: stress tests where models chose blackmail or leaks under goal conflict.
An update on recent Claude Code quality reports (Anthropic)
Creator not supplied; see original source.
Anthropic postmortem on Claude Code quality reports, tracing degradations to three harness and prompt changes.
Agentic misalignment
by Anthropic
A controlled study of risky behavior under conflicting agent incentives.
Goodfire research
Creator not supplied; see original source.
Goodfire research page: interpretability research on understanding and editing the internals of AI models.
Hidden Technical Debt in Machine Learning Systems
Creator not supplied; see original source.
Google Research publication page for Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015).
InterpretML
Creator not supplied; see original source.
InterpretML: open-source toolkit for glassbox models and explaining ML model behaviour.
Investigating unintended model actions in our evaluations and internal use
Creator not supplied; see original source.
Anthropic post investigating unintended model actions observed in its evaluations and internal use.
Leakage and the Reproducibility Crisis in ML-based Science
Creator not supplied; see original source.
arXiv paper (2022) "Leakage and the Reproducibility Crisis in ML-based Science", documenting data leakage as a cause of reproducibility failures in ML-based science.
Lost in the Middle
Creator not supplied; see original source.
arXiv paper (2023) "Lost in the Middle", showing LLM accuracy drops for information placed mid-context.
MCP docs: Security best practices
Creator not supplied; see original source.
MCP security best practices: mitigations for confused deputy, token passthrough and session hijacking attacks.
Opacus
Creator not supplied; see original source.
Opacus: PyTorch library and tutorials for training models with differential privacy.
Qwen3Guard Technical Report
by Haiquan Zhao and colleagues / Qwen Team
Understand Qwen’s multilingual safety classifiers and the distinction between whole-response and streaming moderation.
Shortcut Learning in Deep Neural Networks
Creator not supplied; see original source.
arXiv paper (2020) "Shortcut Learning in Deep Neural Networks", analysing how networks learn shortcut solutions that fail under distribution shift.
Strengthening our defenses against distillation attacks
Creator not supplied; see original source.
Anthropic research post on defending against illicit distillation of its models.
Tracing the thoughts of a large language model
Creator not supplied; see original source.
Anthropic interpretability research tracing internal mechanisms of Claude with attribution graphs.
UK education generative-AI product safety standards
Creator not supplied; see original source.
UK Department for Education generative AI product safety standards for AI tools used in education.
A Mathematical Framework for Transformer Circuits
Creator not supplied; see original source.
Anthropic Transformer Circuits paper A Mathematical Framework for Transformer Circuits (2021).
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Creator not supplied; see original source.
arXiv paper (2024) introducing AgentDojo, an environment for evaluating prompt injection attacks and defences on tool agents.
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Creator not supplied; see original source.
arXiv paper (2024) introducing AgentHarm, a benchmark measuring harmful task completion by LLM agents.
Amnesty: Xenophobic Machines (Dutch childcare benefits)
Creator not supplied; see original source.
Amnesty International report Xenophobic Machines on discriminatory algorithmic profiling in the Dutch childcare benefits scandal.
Defeating Prompt Injections by Design (CaMeL)
Creator not supplied; see original source.
arXiv paper (2025) by Google DeepMind introducing CaMeL, a capability-based design that defeats prompt injections.
Embrace The Red (Johann Rehberger)
Creator not supplied; see original source.
Embrace The Red, Johann Rehberger's blog documenting prompt-injection and data-exfiltration exploits in real AI agents and IDEs.
Emergent Misalignment
Creator not supplied; see original source.
arXiv paper (2025) "Emergent Misalignment", showing narrow fine-tuning on insecure code produces broadly misaligned behaviour.
On the Biology of a Large Language Model
Creator not supplied; see original source.
Anthropic Transformer Circuits paper On the Biology of a Large Language Model: attribution graphs on Claude 3.5 Haiku.
The Instruction Hierarchy
Creator not supplied; see original source.
arXiv paper (2024) by OpenAI introducing the instruction hierarchy to train models to prioritise privileged instructions.
Universal adversarial attacks (GCG)
Creator not supplied; see original source.
arXiv paper (2023) introducing GCG, universal transferable adversarial suffixes that jailbreak aligned LLMs.