Safety, security & interpretability
Safety, security & interpretability
- alignment (3)
- incidents & failure cases (41)
- interpretability (13)
- safety evals (16)
- security & red-teaming (51)
124 resources · page 3 of 4
AI security / prompt injection
by OWASP Gen AI Security Project
OWASP Gen AI Security Project archive of Top 10 risk lists for LLM applications.
Apollo Research: Science
Apollo Research science page listing its research and pre-deployment evaluations on AI scheming and mitigations.
Beyond permission prompts (sandboxing)
by Anthropic
Anthropic engineering post on Claude Code's filesystem and network sandboxing to reduce permission prompts.
BSI Künstliche Intelligenz
by German Federal Office for Information Security (BSI)
German Federal Office for Information Security (BSI) AI page with security guidance and publications on AI systems.
Constitutional Classifiers
by Anthropic
Anthropic research post on Constitutional Classifiers, a method for defending LLMs against universal jailbreaks.
Giskard documentation
by Giskard
Giskard documentation: open-source library and platform for testing, evaluating and red-teaming LLM agents.
Google SAIF
by Google
Google Secure AI Framework (SAIF): six core elements for AI model security, risk management and privacy.
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
by Kai Greshake et al. (Saarland University)
arXiv paper (2023) by Greshake et al. introducing indirect prompt injection against LLM-integrated applications.
Hermes Agent docs: Security
Hermes Agent security documentation: command approval, user authorisation, container isolation and deployment practices.
HiddenLayer research
by HiddenLayer
HiddenLayer research page publishing findings on AI vulnerabilities, adversarial attacks and model security threats.
How we built Claude Code auto mode
by Anthropic
Anthropic engineering post on how Claude Code auto mode decides when to skip permission prompts safely.
Interpretable Machine Learning
Christoph Molnar's Interpretable Machine Learning book: free online guide to explainability methods and their limits.
Artificial Intelligence (AI): Guidance for Judicial Office Holders
by Courts and Tribunals Judiciary
UK Judiciary guidance (PDF, October 2025) for judicial office holders on AI use cases, risks and verification duties.
Lakera blog
by Lakera
Lakera blog on AI security, prompt injection and LLM attacks, with research posts and product news.
Many-shot jailbreaking
by Anthropic
Anthropic research on many-shot jailbreaking, a long-context attack using many faux dialogue examples.
Statement on CVE-2025-48757 (Matt Palmer)
by Matt Palmer
Matt Palmer's statement on CVE-2025-48757: missing row-level security in Lovable-generated apps exposed user data.
MCP blog: Tool annotations
MCP blog post on tool annotations such as read-only and destructive hints, and how far clients should trust them.
MCP docs: Local server security
MCP documentation on hardening local stdio MCP servers.
MITRE ATLAS
by MITRE
MITRE ATLAS: knowledge base and ATT&CK-style matrix of adversary tactics and techniques against AI systems.
NCSC/CISA Guidelines for secure AI system development
by UK National Cyber Security Centre (NCSC)
UK NCSC and partner guidelines for secure AI system development across design, development, deployment and operation.
How To Become A Mechanistic Interpretability Researcher
by Neel Nanda
Neel Nanda's AI Alignment Forum guide on how to become a mechanistic interpretability researcher, which replaced his earlier getting-started page.
NIST AI 100-2 Adversarial ML taxonomy
by NIST
NIST AI 100-2 report: taxonomy and terminology of adversarial machine learning attacks and mitigations.
OWASP Top 10 for Agentic Applications 2026
by OWASP Gen AI Security Project
OWASP Top 10 for Agentic Applications 2026: list of security risks specific to AI agents.
Patronus AI
by Patronus AI
Patronus AI blog with posts on LLM evaluation, benchmarks and AI reliability research.
From Naptime to Big Sleep: Using Large Language Models To Catch Vulnerabilities In Real-World Code
Google Project Zero post on Big Sleep, an LLM agent that found an exploitable SQLite vulnerability in real code.
promptfoo docs: OWASP LLM Top 10 red teaming
by promptfoo
Promptfoo documentation mapping its red-teaming plugins to the OWASP LLM Top 10 risks.
Red Teaming LMs to Reduce Harms (Ganguli et al.)
by Deep Ganguli et al. (Anthropic)
arXiv paper (2022) by Anthropic on red teaming language models: methods, scaling behaviour and a released attack dataset.
Redwood Research
Redwood Research page listing its research on AI control and keeping AI systems aligned with developer intent.
Retraction Watch: papers with evidence of ChatGPT writing
Retraction Watch running list of published papers and peer reviews containing evidence of ChatGPT-written text.
Sal Khan: Khanmigo's First Chapter Changed How I Think About AI (Khan Academy)
by Khan Academy
Sal Khan's reflection on the first chapter of the Khanmigo AI tutor and what it changed for student learning.
Simon Willison on prompt-injection
Simon Willison's prompt-injection tag: running record of prompt-injection attacks and incidents since 2022.
The lethal trifecta for AI agents (Simon Willison)
Simon Willison's post on the lethal trifecta: private data, untrusted content and exfiltration make AI agents exploitable.