Jailbreaking and red teaming
Published jailbreak research, safety benchmarks and defensive evaluation. Credited links and summaries only; no attack payloads.
- Prompt Injection (4)
- Agent And Tool Security (2)
- Threat Frameworks And Standards (3)
- Security Tooling (3)
- Safety And Alignment Evals (126)
- Jailbreak research (2)
140 resources · page 3 of 5
AI Safety Governance Framework v1.0 (TC260) on ETO AGORA
Creator not supplied; see original source.
ETO AGORA record of China's AI Safety Governance Framework v1.0 published by TC260 in 2024.
AI security / prompt injection
Creator not supplied; see original source.
OWASP Gen AI Security Project archive of Top 10 risk lists for LLM applications.
Apollo Research: Science
Creator not supplied; see original source.
Apollo Research science page listing its research and pre-deployment evaluations on AI scheming and mitigations.
Beyond permission prompts (sandboxing)
Creator not supplied; see original source.
Anthropic engineering post on Claude Code's filesystem and network sandboxing to reduce permission prompts.
BSI Künstliche Intelligenz
Creator not supplied; see original source.
German Federal Office for Information Security (BSI) AI page with security guidance and publications on AI systems.
Constitutional Classifiers
Creator not supplied; see original source.
Anthropic research post on Constitutional Classifiers, a method for defending LLMs against universal jailbreaks.
Giskard documentation
Creator not supplied; see original source.
Giskard documentation: open-source library and platform for testing, evaluating and red-teaming LLM agents.
Google SAIF
Creator not supplied; see original source.
Google Secure AI Framework (SAIF): six core elements for AI model security, risk management and privacy.
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Creator not supplied; see original source.
arXiv paper (2023) by Greshake et al. introducing indirect prompt injection against LLM-integrated applications.
Hermes Agent docs: Security
Creator not supplied; see original source.
Hermes Agent security documentation: command approval, user authorisation, container isolation and deployment practices.
HiddenLayer research
Creator not supplied; see original source.
HiddenLayer research page publishing findings on AI vulnerabilities, adversarial attacks and model security threats.
How we built Claude Code auto mode
Creator not supplied; see original source.
Anthropic engineering post on how Claude Code auto mode decides when to skip permission prompts safely.
Interpretable Machine Learning
Creator not supplied; see original source.
Christoph Molnar's Interpretable Machine Learning book: free online guide to explainability methods and their limits.
Artificial Intelligence (AI): Guidance for Judicial Office Holders
Creator not supplied; see original source.
UK Judiciary guidance (PDF, October 2025) for judicial office holders on AI use cases, risks and verification duties.
Lakera blog
Creator not supplied; see original source.
Lakera blog on AI security, prompt injection and LLM attacks, with research posts and product news.
Many-shot jailbreaking
Creator not supplied; see original source.
Anthropic research on many-shot jailbreaking, a long-context attack using many faux dialogue examples.
Statement on CVE-2025-48757 (Matt Palmer)
Creator not supplied; see original source.
Matt Palmer's statement on CVE-2025-48757: missing row-level security in Lovable-generated apps exposed user data.
MCP blog: Tool annotations
Creator not supplied; see original source.
MCP blog post on tool annotations such as read-only and destructive hints, and how far clients should trust them.
MCP docs: Local server security
Creator not supplied; see original source.
MCP documentation on hardening local stdio MCP servers.
MITRE ATLAS
Creator not supplied; see original source.
MITRE ATLAS: knowledge base and ATT&CK-style matrix of adversary tactics and techniques against AI systems.
NCSC/CISA Guidelines for secure AI system development
Creator not supplied; see original source.
UK NCSC and partner guidelines for secure AI system development across design, development, deployment and operation.
How To Become A Mechanistic Interpretability Researcher
Creator not supplied; see original source.
Neel Nanda's AI Alignment Forum guide on how to become a mechanistic interpretability researcher, which replaced his earlier getting-started page.
NIST AI 100-2 Adversarial ML taxonomy
Creator not supplied; see original source.
NIST AI 100-2 report: taxonomy and terminology of adversarial machine learning attacks and mitigations.
NIST AI Risk Management Framework
by NIST
A government reference for organizing AI risk-management work.
OWASP Top 10 for LLM Applications 2025
by OWASP Gen AI Security Project
Use a shared vocabulary for discussing risks in LLM applications.
OWASP Top 10 for Agentic Applications 2026
Creator not supplied; see original source.
OWASP Top 10 for Agentic Applications 2026: list of security risks specific to AI agents.
Patronus AI
Creator not supplied; see original source.
Patronus AI blog with posts on LLM evaluation, benchmarks and AI reliability research.
From Naptime to Big Sleep: Using Large Language Models To Catch Vulnerabilities In Real-World Code
Creator not supplied; see original source.
Google Project Zero post on Big Sleep, an LLM agent that found an exploitable SQLite vulnerability in real code.
promptfoo docs: OWASP LLM Top 10 red teaming
Creator not supplied; see original source.
Promptfoo documentation mapping its red-teaming plugins to the OWASP LLM Top 10 risks.
Red Teaming LMs to Reduce Harms (Ganguli et al.)
Creator not supplied; see original source.
arXiv paper (2022) by Anthropic on red teaming language models: methods, scaling behaviour and a released attack dataset.
Redwood Research
Creator not supplied; see original source.
Redwood Research page listing its research on AI control and keeping AI systems aligned with developer intent.
Retraction Watch: papers with evidence of ChatGPT writing
Creator not supplied; see original source.
Retraction Watch running list of published papers and peer reviews containing evidence of ChatGPT-written text.