Security Red Teaming
51 resources · page 2 of 2
NIST AI 100-2 Adversarial ML taxonomy
Creator not supplied; see original source.
NIST AI 100-2 report: taxonomy and terminology of adversarial machine learning attacks and mitigations.
OWASP Top 10 for Agentic Applications 2026
Creator not supplied; see original source.
OWASP Top 10 for Agentic Applications 2026: list of security risks specific to AI agents.
From Naptime to Big Sleep: Using Large Language Models To Catch Vulnerabilities In Real-World Code
Creator not supplied; see original source.
Google Project Zero post on Big Sleep, an LLM agent that found an exploitable SQLite vulnerability in real code.
promptfoo docs: OWASP LLM Top 10 red teaming
Creator not supplied; see original source.
Promptfoo documentation mapping its red-teaming plugins to the OWASP LLM Top 10 risks.
Red Teaming LMs to Reduce Harms (Ganguli et al.)
Creator not supplied; see original source.
arXiv paper (2022) by Anthropic on red teaming language models: methods, scaling behaviour and a released attack dataset.
Redwood Research
Creator not supplied; see original source.
Redwood Research page listing its research on AI control and keeping AI systems aligned with developer intent.
Simon Willison on prompt-injection
Creator not supplied; see original source.
Simon Willison's prompt-injection tag: running record of prompt-injection attacks and incidents since 2022.
The lethal trifecta for AI agents (Simon Willison)
Creator not supplied; see original source.
Simon Willison's post on the lethal trifecta: private data, untrusted content and exfiltration make AI agents exploitable.
AIxCC finals: Tale of the tape (Trail of Bits)
Creator not supplied; see original source.
Trail of Bits analysis comparing the cyber reasoning system designs of DARPA AIxCC finalists.
vLLM docs: Security
Creator not supplied; see original source.
vLLM security documentation: guidance on exposing and hardening self-hosted inference servers.
XBOW Blog
Creator not supplied; see original source.
XBOW blog with research posts on autonomous AI penetration testing, vulnerability discovery and exploit validation.
MCP docs: Security best practices
Creator not supplied; see original source.
MCP security best practices: mitigations for confused deputy, token passthrough and session hijacking attacks.
Strengthening our defenses against distillation attacks
Creator not supplied; see original source.
Anthropic research post on defending against illicit distillation of its models.
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Creator not supplied; see original source.
arXiv paper (2024) introducing AgentDojo, an environment for evaluating prompt injection attacks and defences on tool agents.
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Creator not supplied; see original source.
arXiv paper (2024) introducing AgentHarm, a benchmark measuring harmful task completion by LLM agents.
Defeating Prompt Injections by Design (CaMeL)
Creator not supplied; see original source.
arXiv paper (2025) by Google DeepMind introducing CaMeL, a capability-based design that defeats prompt injections.
Embrace The Red (Johann Rehberger)
Creator not supplied; see original source.
Embrace The Red, Johann Rehberger's blog documenting prompt-injection and data-exfiltration exploits in real AI agents and IDEs.
The Instruction Hierarchy
Creator not supplied; see original source.
arXiv paper (2024) by OpenAI introducing the instruction hierarchy to train models to prioritise privileged instructions.
Universal adversarial attacks (GCG)
Creator not supplied; see original source.
arXiv paper (2023) introducing GCG, universal transferable adversarial suffixes that jailbreak aligned LLMs.