Jailbreaking and red teaming for advanced learners
83 resources · page 3 of 3
Hidden Technical Debt in Machine Learning Systems
Creator not supplied; see original source.
Google Research publication page for Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015).
InterpretML
Creator not supplied; see original source.
InterpretML: open-source toolkit for glassbox models and explaining ML model behaviour.
Investigating unintended model actions in our evaluations and internal use
Creator not supplied; see original source.
Anthropic post investigating unintended model actions observed in its evaluations and internal use.
Embrace The Red
Public case reports about security failures in deployed AI products.
Leakage and the Reproducibility Crisis in ML-based Science
Creator not supplied; see original source.
arXiv paper (2022) "Leakage and the Reproducibility Crisis in ML-based Science", documenting data leakage as a cause of reproducibility failures in ML-based science.
LLAMATOR: authorized conversational AI security testing
by Roman Neronov, Timur Nizamov and Nikita Ivanov / LLAMATOR
A Russian-origin framework for assessing your own chatbots, with reviewable test histories and reports.
Lost in the Middle
Creator not supplied; see original source.
arXiv paper (2023) "Lost in the Middle", showing LLM accuracy drops for information placed mid-context.
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
by Mantas Mazeika, Long Phan and colleagues
Study a standardized approach to red-team evaluation and robust refusal without copying attack recipes.
MCP docs: Security best practices
Creator not supplied; see original source.
MCP security best practices: mitigations for confused deputy, token passthrough and session hijacking attacks.
MITRE ATLAS
by MITRE
Connect AI threats with documented techniques, mitigations and cases.
garak - LLM vulnerability scanner
by NVIDIA
A public scanner project for understanding classes of model failures.
Opacus
Creator not supplied; see original source.
Opacus: PyTorch library and tutorials for training models with differential privacy.
promptfoo
by promptfoo
Inspect a project combining application evaluation and security testing.
Qwen3Guard Technical Report
by Haiquan Zhao and colleagues / Qwen Team
Understand Qwen’s multilingual safety classifiers and the distinction between whole-response and streaming moderation.
Shortcut Learning in Deep Neural Networks
Creator not supplied; see original source.
arXiv paper (2020) "Shortcut Learning in Deep Neural Networks", analysing how networks learn shortcut solutions that fail under distribution shift.
Strengthening our defenses against distillation attacks
Creator not supplied; see original source.
Anthropic research post on defending against illicit distillation of its models.
Tracing the thoughts of a large language model
Creator not supplied; see original source.
Anthropic interpretability research tracing internal mechanisms of Claude with attribution graphs.
UK education generative-AI product safety standards
Creator not supplied; see original source.
UK Department for Education generative AI product safety standards for AI tools used in education.
Design Patterns for Securing LLM Agents against Prompt Injections
Summary of a paper's defensive agent architectures.