Jailbreaking and red teaming
Published jailbreak research, safety benchmarks and defensive evaluation. Credited links and summaries only; no attack payloads.
- Prompt Injection (4)
- Agent And Tool Security (2)
- Threat Frameworks And Standards (3)
- Security Tooling (3)
- Safety And Alignment Evals (126)
- Jailbreak research (2)
140 resources · page 4 of 5
Sal Khan: Khanmigo's First Chapter Changed How I Think About AI (Khan Academy)
Creator not supplied; see original source.
Sal Khan's reflection on the first chapter of the Khanmigo AI tutor and what it changed for student learning.
Simon Willison on prompt-injection
Creator not supplied; see original source.
Simon Willison's prompt-injection tag: running record of prompt-injection attacks and incidents since 2022.
Simon Willison on prompt injection
Follow an ongoing public record of prompt-injection research.
The lethal trifecta for AI agents (Simon Willison)
Creator not supplied; see original source.
Simon Willison's post on the lethal trifecta: private data, untrusted content and exfiltration make AI agents exploitable.
AIxCC finals: Tale of the tape (Trail of Bits)
Creator not supplied; see original source.
Trail of Bits analysis comparing the cyber reasoning system designs of DARPA AIxCC finalists.
vLLM docs: Security
Creator not supplied; see original source.
vLLM security documentation: guidance on exposing and hardening self-hosted inference servers.
The lethal trifecta for AI agents
A defensive framework for recognizing risky combinations of agent capabilities and untrusted inputs.
XBOW Blog
Creator not supplied; see original source.
XBOW blog with research posts on autonomous AI penetration testing, vulnerability discovery and exploit validation.
A postmortem of three recent issues
Creator not supplied; see original source.
Anthropic postmortem explaining three infrastructure bugs that intermittently degraded Claude's responses and how they were found.
Agentic misalignment
Creator not supplied; see original source.
Anthropic research on agentic misalignment: stress tests where models chose blackmail or leaks under goal conflict.
An update on recent Claude Code quality reports (Anthropic)
Creator not supplied; see original source.
Anthropic postmortem on Claude Code quality reports, tracing degradations to three harness and prompt changes.
Agentic misalignment
by Anthropic
A controlled study of risky behavior under conflicting agent incentives.
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
by Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko and colleagues
Learn why jailbreak studies need consistent threat models, scoring and reproducible comparisons.
Goodfire research
Creator not supplied; see original source.
Goodfire research page: interpretability research on understanding and editing the internals of AI models.
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Foundational public research on indirect prompt injection. Linked only; no payloads reproduced.
Hidden Technical Debt in Machine Learning Systems
Creator not supplied; see original source.
Google Research publication page for Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015).
InterpretML
Creator not supplied; see original source.
InterpretML: open-source toolkit for glassbox models and explaining ML model behaviour.
Investigating unintended model actions in our evaluations and internal use
Creator not supplied; see original source.
Anthropic post investigating unintended model actions observed in its evaluations and internal use.
Embrace The Red
Public case reports about security failures in deployed AI products.
Leakage and the Reproducibility Crisis in ML-based Science
Creator not supplied; see original source.
arXiv paper (2022) "Leakage and the Reproducibility Crisis in ML-based Science", documenting data leakage as a cause of reproducibility failures in ML-based science.
LLAMATOR: authorized conversational AI security testing
by Roman Neronov, Timur Nizamov and Nikita Ivanov / LLAMATOR
A Russian-origin framework for assessing your own chatbots, with reviewable test histories and reports.
Lost in the Middle
Creator not supplied; see original source.
arXiv paper (2023) "Lost in the Middle", showing LLM accuracy drops for information placed mid-context.
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
by Mantas Mazeika, Long Phan and colleagues
Study a standardized approach to red-team evaluation and robust refusal without copying attack recipes.
MCP docs: Security best practices
Creator not supplied; see original source.
MCP security best practices: mitigations for confused deputy, token passthrough and session hijacking attacks.
MITRE ATLAS
by MITRE
Connect AI threats with documented techniques, mitigations and cases.
garak - LLM vulnerability scanner
by NVIDIA
A public scanner project for understanding classes of model failures.
Opacus
Creator not supplied; see original source.
Opacus: PyTorch library and tutorials for training models with differential privacy.
promptfoo
by promptfoo
Inspect a project combining application evaluation and security testing.
Qwen3Guard Technical Report
by Haiquan Zhao and colleagues / Qwen Team
Understand Qwen’s multilingual safety classifiers and the distinction between whole-response and streaming moderation.
Shortcut Learning in Deep Neural Networks
Creator not supplied; see original source.
arXiv paper (2020) "Shortcut Learning in Deep Neural Networks", analysing how networks learn shortcut solutions that fail under distribution shift.
Strengthening our defenses against distillation attacks
Creator not supplied; see original source.
Anthropic research post on defending against illicit distillation of its models.
Tracing the thoughts of a large language model
Creator not supplied; see original source.
Anthropic interpretability research tracing internal mechanisms of Claude with attribution graphs.