security & red-teaming: 51 curated resources
security & red-teaming
51 resources · page 2 of 2
NIST AI 100-2 Adversarial ML taxonomy
by NIST
NIST AI 100-2 report: taxonomy and terminology of adversarial machine learning attacks and mitigations.
OWASP Top 10 for Agentic Applications 2026
by OWASP Gen AI Security Project
OWASP Top 10 for Agentic Applications 2026: list of security risks specific to AI agents.
From Naptime to Big Sleep: Using Large Language Models To Catch Vulnerabilities In Real-World Code
Google Project Zero post on Big Sleep, an LLM agent that found an exploitable SQLite vulnerability in real code.
promptfoo docs: OWASP LLM Top 10 red teaming
by promptfoo
Promptfoo documentation mapping its red-teaming plugins to the OWASP LLM Top 10 risks.
Red Teaming LMs to Reduce Harms (Ganguli et al.)
by Deep Ganguli et al. (Anthropic)
arXiv paper (2022) by Anthropic on red teaming language models: methods, scaling behaviour and a released attack dataset.
Redwood Research
Redwood Research page listing its research on AI control and keeping AI systems aligned with developer intent.
Simon Willison on prompt-injection
Simon Willison's prompt-injection tag: running record of prompt-injection attacks and incidents since 2022.
The lethal trifecta for AI agents (Simon Willison)
Simon Willison's post on the lethal trifecta: private data, untrusted content and exfiltration make AI agents exploitable.
AIxCC finals: Tale of the tape (Trail of Bits)
Trail of Bits analysis comparing the cyber reasoning system designs of DARPA AIxCC finalists.
vLLM docs: Security
by vLLM
vLLM security documentation: guidance on exposing and hardening self-hosted inference servers.
XBOW Blog
by XBOW
XBOW blog with research posts on autonomous AI penetration testing, vulnerability discovery and exploit validation.
MCP docs: Security best practices
MCP security best practices: mitigations for confused deputy, token passthrough and session hijacking attacks.
Strengthening our defenses against distillation attacks
by Anthropic
Anthropic research post on defending against illicit distillation of its models.
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
by Edoardo Debenedetti et al. (ETH Zurich)
arXiv paper (2024) introducing AgentDojo, an environment for evaluating prompt injection attacks and defences on tool agents.
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
by Maksym Andriushchenko et al. (Gray Swan AI and UK AI Security Institute)
arXiv paper (2024) introducing AgentHarm, a benchmark measuring harmful task completion by LLM agents.
Defeating Prompt Injections by Design (CaMeL)
by Edoardo Debenedetti et al. (Google DeepMind)
arXiv paper (2025) by Google DeepMind introducing CaMeL, a capability-based design that defeats prompt injections.
Embrace The Red (Johann Rehberger)
by Johann Rehberger (Embrace The Red)
Embrace The Red, Johann Rehberger's blog documenting prompt-injection and data-exfiltration exploits in real AI agents and IDEs.
The Instruction Hierarchy
by Eric Wallace et al. (OpenAI)
arXiv paper (2024) by OpenAI introducing the instruction hierarchy to train models to prioritise privileged instructions.
Universal adversarial attacks (GCG)
by Andy Zou et al. (Carnegie Mellon University)
arXiv paper (2023) introducing GCG, universal transferable adversarial suffixes that jailbreak aligned LLMs.