Safety, security & interpretability
Safety, security & interpretability
- alignment (3)
- incidents & failure cases (41)
- interpretability (13)
- safety evals (16)
- security & red-teaming (51)
124 resources · page 4 of 4
AIxCC finals: Tale of the tape (Trail of Bits)
Trail of Bits analysis comparing the cyber reasoning system designs of DARPA AIxCC finalists.
vLLM docs: Security
by vLLM
vLLM security documentation: guidance on exposing and hardening self-hosted inference servers.
XBOW Blog
by XBOW
XBOW blog with research posts on autonomous AI penetration testing, vulnerability discovery and exploit validation.
A postmortem of three recent issues
by Anthropic
Anthropic postmortem explaining three infrastructure bugs that intermittently degraded Claude's responses and how they were found.
Agentic misalignment
by Anthropic
Anthropic research on agentic misalignment: stress tests where models chose blackmail or leaks under goal conflict.
An update on recent Claude Code quality reports (Anthropic)
by Anthropic
Anthropic postmortem on Claude Code quality reports, tracing degradations to three harness and prompt changes.
Goodfire research
by Goodfire
Goodfire research page: interpretability research on understanding and editing the internals of AI models.
Hidden Technical Debt in Machine Learning Systems
Google Research publication page for Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015).
InterpretML
InterpretML: open-source toolkit for glassbox models and explaining ML model behaviour.
Investigating unintended model actions in our evaluations and internal use
by Anthropic
Anthropic post investigating unintended model actions observed in its evaluations and internal use.
Leakage and the Reproducibility Crisis in ML-based Science
by Sayash Kapoor and Arvind Narayanan (Princeton University)
arXiv paper (2022) "Leakage and the Reproducibility Crisis in ML-based Science", documenting data leakage as a cause of reproducibility failures in ML-based science.
Lost in the Middle
by Nelson F. Liu et al. (Stanford University)
arXiv paper (2023) "Lost in the Middle", showing LLM accuracy drops for information placed mid-context.
MCP docs: Security best practices
MCP security best practices: mitigations for confused deputy, token passthrough and session hijacking attacks.
Opacus
Opacus: PyTorch library and tutorials for training models with differential privacy.
Shortcut Learning in Deep Neural Networks
by Robert Geirhos et al. (University of Tübingen)
arXiv paper (2020) "Shortcut Learning in Deep Neural Networks", analysing how networks learn shortcut solutions that fail under distribution shift.
Strengthening our defenses against distillation attacks
by Anthropic
Anthropic research post on defending against illicit distillation of its models.
Tracing the thoughts of a large language model
by Anthropic
Anthropic interpretability research tracing internal mechanisms of Claude with attribution graphs.
UK education generative-AI product safety standards
by Department for Education (UK Government)
UK Department for Education generative AI product safety standards for AI tools used in education.
A Mathematical Framework for Transformer Circuits
by Anthropic
Anthropic Transformer Circuits paper A Mathematical Framework for Transformer Circuits (2021).
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
by Edoardo Debenedetti et al. (ETH Zurich)
arXiv paper (2024) introducing AgentDojo, an environment for evaluating prompt injection attacks and defences on tool agents.
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
by Maksym Andriushchenko et al. (Gray Swan AI and UK AI Security Institute)
arXiv paper (2024) introducing AgentHarm, a benchmark measuring harmful task completion by LLM agents.
Amnesty: Xenophobic Machines (Dutch childcare benefits)
Amnesty International report Xenophobic Machines on discriminatory algorithmic profiling in the Dutch childcare benefits scandal.
Defeating Prompt Injections by Design (CaMeL)
by Edoardo Debenedetti et al. (Google DeepMind)
arXiv paper (2025) by Google DeepMind introducing CaMeL, a capability-based design that defeats prompt injections.
Embrace The Red (Johann Rehberger)
by Johann Rehberger (Embrace The Red)
Embrace The Red, Johann Rehberger's blog documenting prompt-injection and data-exfiltration exploits in real AI agents and IDEs.
Emergent Misalignment
by Jan Betley et al. (Truthful AI)
arXiv paper (2025) "Emergent Misalignment", showing narrow fine-tuning on insecure code produces broadly misaligned behaviour.
On the Biology of a Large Language Model
by Anthropic
Anthropic Transformer Circuits paper On the Biology of a Large Language Model: attribution graphs on Claude 3.5 Haiku.
The Instruction Hierarchy
by Eric Wallace et al. (OpenAI)
arXiv paper (2024) by OpenAI introducing the instruction hierarchy to train models to prioritise privileged instructions.
Universal adversarial attacks (GCG)
by Andy Zou et al. (Carnegie Mellon University)
arXiv paper (2023) introducing GCG, universal transferable adversarial suffixes that jailbreak aligned LLMs.