Safety, security & interpretability for expert learners
63 resources · page 2 of 2
NVIDIA/NemoClaw
NVIDIA NemoClaw GitHub repo: runtime for running agents such as Hermes, Deep Agents and OpenClaw.
NYU CTF Bench
by NYU LLM CTF
NYU CTF Bench GitHub repo: capture-the-flag challenges from CSAW competitions for evaluating LLM agents on security tasks.
OpenFHE
by OpenFHE
OpenFHE GitHub repo: open-source library for fully homomorphic encryption.
promptfoo
by promptfoo
promptfoo GitHub repo: CLI for testing prompts, agents and RAG, with red teaming.
PyRIT
Microsoft PyRIT GitHub repo: Python Risk Identification Tool for automated red teaming of generative AI.
PySyft
by OpenMined
OpenMined PySyft GitHub repo: data science on data that stays on the owner's server, for privacy-preserving access.
SAELens
SAELens GitHub repo: training and analysing sparse autoencoders on language models.
Safe Superintelligence Inc.
by Safe Superintelligence Inc.
Safe Superintelligence Inc. (SSI): company focused solely on building safe superintelligence; site has a short mission statement.
STAT: IBM Watson recommended "unsafe and incorrect" cancer treatments
by STAT
STAT report on internal IBM documents showing Watson for Oncology gave unsafe and incorrect treatment advice.
Stephenson Harwood: "Law and error"
Stephenson Harwood article on court scrutiny of AI tools after fictitious citations in legal filings.
Team Atlanta "Atlantis" CRS (finals)
by Team-Atlanta
Team Atlanta Atlantis GitHub repo: cyber reasoning system from the DARPA AIxCC finals for automated vulnerability repair.
The Register: Figma AI accused of plagiarising Apple
by The Register
The Register report on Figma pulling its Make Designs AI feature after it copied Apple's Weather app.
TILOS MacroPlacement
TILOS MacroPlacement GitHub repo: benchmarks, evaluators and an open reproduction of Google's Circuit Training for chip macro placement.
tldrsec prompt-injection-defenses
by tl;dr sec
tldrsec prompt-injection-defenses GitHub repo: catalogue of practical and proposed prompt injection defences.
Trail of Bits "Buttercup" CRS (finals)
Trail of Bits Buttercup GitHub repo: cyber reasoning system submitted to the DARPA AIxCC finals.
TransformerLens
TransformerLens GitHub repo: library for mechanistic interpretability of GPT-style language models.
Transluce
by Transluce
Transluce: nonprofit lab building open tools for AI oversight and interpretability.
Waymo Safety Impact hub
by Waymo
Waymo Safety Impact hub: company-published crash-rate comparisons between rider-only driving miles and human benchmarks.
Anthropic Transparency Hub
by Anthropic
Anthropic Transparency Hub: system cards, model reports and voluntary commitments.
OpenAI Deployment Safety Hub
by OpenAI
OpenAI Deployment Safety Hub: system cards with safety evaluations and safeguards for deployed OpenAI models.
Final report of the Royal Commission into the Robodebt Scheme (Prime Minister of Australia)
by Prime Minister of Australia
Australian Prime Minister's media release on the final report of the Royal Commission into the Robodebt automated debt scheme.
A Mathematical Framework for Transformer Circuits
by Anthropic
Anthropic Transformer Circuits paper A Mathematical Framework for Transformer Circuits (2021).
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
by Edoardo Debenedetti et al. (ETH Zurich)
arXiv paper (2024) introducing AgentDojo, an environment for evaluating prompt injection attacks and defences on tool agents.
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
by Maksym Andriushchenko et al. (Gray Swan AI and UK AI Security Institute)
arXiv paper (2024) introducing AgentHarm, a benchmark measuring harmful task completion by LLM agents.
Amnesty: Xenophobic Machines (Dutch childcare benefits)
Amnesty International report Xenophobic Machines on discriminatory algorithmic profiling in the Dutch childcare benefits scandal.
Defeating Prompt Injections by Design (CaMeL)
by Edoardo Debenedetti et al. (Google DeepMind)
arXiv paper (2025) by Google DeepMind introducing CaMeL, a capability-based design that defeats prompt injections.
Embrace The Red (Johann Rehberger)
by Johann Rehberger (Embrace The Red)
Embrace The Red, Johann Rehberger's blog documenting prompt-injection and data-exfiltration exploits in real AI agents and IDEs.
Emergent Misalignment
by Jan Betley et al. (Truthful AI)
arXiv paper (2025) "Emergent Misalignment", showing narrow fine-tuning on insecure code produces broadly misaligned behaviour.
On the Biology of a Large Language Model
by Anthropic
Anthropic Transformer Circuits paper On the Biology of a Large Language Model: attribution graphs on Claude 3.5 Haiku.
The Instruction Hierarchy
by Eric Wallace et al. (OpenAI)
arXiv paper (2024) by OpenAI introducing the instruction hierarchy to train models to prioritise privileged instructions.
Universal adversarial attacks (GCG)
by Andy Zou et al. (Carnegie Mellon University)
arXiv paper (2023) introducing GCG, universal transferable adversarial suffixes that jailbreak aligned LLMs.