Jailbreaking and red teaming for advanced learners
83 resources · page 2 of 3
NVIDIA/NemoClaw
Creator not supplied; see original source.
NVIDIA NemoClaw GitHub repo: runtime for running agents such as Hermes, Deep Agents and OpenClaw.
NYU CTF Bench
Creator not supplied; see original source.
NYU CTF Bench GitHub repo: capture-the-flag challenges from CSAW competitions for evaluating LLM agents on security tasks.
OpenFHE
Creator not supplied; see original source.
OpenFHE GitHub repo: open-source library for fully homomorphic encryption.
promptfoo
Creator not supplied; see original source.
promptfoo GitHub repo: CLI for testing prompts, agents and RAG, with red teaming.
PyRIT
Creator not supplied; see original source.
Microsoft PyRIT GitHub repo: Python Risk Identification Tool for automated red teaming of generative AI.
PySyft
Creator not supplied; see original source.
OpenMined PySyft GitHub repo: data science on data that stays on the owner's server, for privacy-preserving access.
SAELens
Creator not supplied; see original source.
SAELens GitHub repo: training and analysing sparse autoencoders on language models.
Safe Superintelligence Inc.
Creator not supplied; see original source.
Safe Superintelligence Inc. (SSI): company focused solely on building safe superintelligence; site has a short mission statement.
STAT: IBM Watson recommended "unsafe and incorrect" cancer treatments
Creator not supplied; see original source.
STAT report on internal IBM documents showing Watson for Oncology gave unsafe and incorrect treatment advice.
Stephenson Harwood: "Law and error"
Creator not supplied; see original source.
Stephenson Harwood article on court scrutiny of AI tools after fictitious citations in legal filings.
Team Atlanta "Atlantis" CRS (finals)
Creator not supplied; see original source.
Team Atlanta Atlantis GitHub repo: cyber reasoning system from the DARPA AIxCC finals for automated vulnerability repair.
The Register: Figma AI accused of plagiarising Apple
Creator not supplied; see original source.
The Register report on Figma pulling its Make Designs AI feature after it copied Apple's Weather app.
TILOS MacroPlacement
Creator not supplied; see original source.
TILOS MacroPlacement GitHub repo: benchmarks, evaluators and an open reproduction of Google's Circuit Training for chip macro placement.
tldrsec prompt-injection-defenses
Creator not supplied; see original source.
tldrsec prompt-injection-defenses GitHub repo: catalogue of practical and proposed prompt injection defences.
Trail of Bits "Buttercup" CRS (finals)
Creator not supplied; see original source.
Trail of Bits Buttercup GitHub repo: cyber reasoning system submitted to the DARPA AIxCC finals.
TransformerLens
Creator not supplied; see original source.
TransformerLens GitHub repo: library for mechanistic interpretability of GPT-style language models.
Transluce
Creator not supplied; see original source.
Transluce: nonprofit lab building open tools for AI oversight and interpretability.
Waymo Safety Impact hub
Creator not supplied; see original source.
Waymo Safety Impact hub: company-published crash-rate comparisons between rider-only driving miles and human benchmarks.
Anthropic Transparency Hub
Creator not supplied; see original source.
Anthropic Transparency Hub: system cards, model reports and voluntary commitments.
OpenAI Deployment Safety Hub
Creator not supplied; see original source.
OpenAI Deployment Safety Hub: system cards with safety evaluations and safeguards for deployed OpenAI models.
Final report of the Royal Commission into the Robodebt Scheme (Prime Minister of Australia)
Creator not supplied; see original source.
Australian Prime Minister's media release on the final report of the Royal Commission into the Robodebt automated debt scheme.
Apollo Research: Science
Creator not supplied; see original source.
Apollo Research science page listing its research and pre-deployment evaluations on AI scheming and mitigations.
NIST AI 100-2 Adversarial ML taxonomy
Creator not supplied; see original source.
NIST AI 100-2 report: taxonomy and terminology of adversarial machine learning attacks and mitigations.
Redwood Research
Creator not supplied; see original source.
Redwood Research page listing its research on AI control and keeping AI systems aligned with developer intent.
The lethal trifecta for AI agents
A defensive framework for recognizing risky combinations of agent capabilities and untrusted inputs.
A postmortem of three recent issues
Creator not supplied; see original source.
Anthropic postmortem explaining three infrastructure bugs that intermittently degraded Claude's responses and how they were found.
Agentic misalignment
Creator not supplied; see original source.
Anthropic research on agentic misalignment: stress tests where models chose blackmail or leaks under goal conflict.
An update on recent Claude Code quality reports (Anthropic)
Creator not supplied; see original source.
Anthropic postmortem on Claude Code quality reports, tracing degradations to three harness and prompt changes.
Agentic misalignment
by Anthropic
A controlled study of risky behavior under conflicting agent incentives.
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
by Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko and colleagues
Learn why jailbreak studies need consistent threat models, scoring and reproducible comparisons.
Goodfire research
Creator not supplied; see original source.
Goodfire research page: interpretability research on understanding and editing the internals of AI models.
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Foundational public research on indirect prompt injection. Linked only; no payloads reproduced.