Safety, security & interpretability for advanced learners
71 resources · page 2 of 3
NVIDIA/NemoClaw
NVIDIA NemoClaw GitHub repo: runtime for running agents such as Hermes, Deep Agents and OpenClaw.
NYU CTF Bench
by NYU LLM CTF
NYU CTF Bench GitHub repo: capture-the-flag challenges from CSAW competitions for evaluating LLM agents on security tasks.
OpenFHE
by OpenFHE
OpenFHE GitHub repo: open-source library for fully homomorphic encryption.
promptfoo
by promptfoo
promptfoo GitHub repo: CLI for testing prompts, agents and RAG, with red teaming.
PyRIT
Microsoft PyRIT GitHub repo: Python Risk Identification Tool for automated red teaming of generative AI.
PySyft
by OpenMined
OpenMined PySyft GitHub repo: data science on data that stays on the owner's server, for privacy-preserving access.
SAELens
SAELens GitHub repo: training and analysing sparse autoencoders on language models.
Safe Superintelligence Inc.
by Safe Superintelligence Inc.
Safe Superintelligence Inc. (SSI): company focused solely on building safe superintelligence; site has a short mission statement.
STAT: IBM Watson recommended "unsafe and incorrect" cancer treatments
by STAT
STAT report on internal IBM documents showing Watson for Oncology gave unsafe and incorrect treatment advice.
Stephenson Harwood: "Law and error"
Stephenson Harwood article on court scrutiny of AI tools after fictitious citations in legal filings.
Team Atlanta "Atlantis" CRS (finals)
by Team-Atlanta
Team Atlanta Atlantis GitHub repo: cyber reasoning system from the DARPA AIxCC finals for automated vulnerability repair.
The Register: Figma AI accused of plagiarising Apple
by The Register
The Register report on Figma pulling its Make Designs AI feature after it copied Apple's Weather app.
TILOS MacroPlacement
TILOS MacroPlacement GitHub repo: benchmarks, evaluators and an open reproduction of Google's Circuit Training for chip macro placement.
tldrsec prompt-injection-defenses
by tl;dr sec
tldrsec prompt-injection-defenses GitHub repo: catalogue of practical and proposed prompt injection defences.
Trail of Bits "Buttercup" CRS (finals)
Trail of Bits Buttercup GitHub repo: cyber reasoning system submitted to the DARPA AIxCC finals.
TransformerLens
TransformerLens GitHub repo: library for mechanistic interpretability of GPT-style language models.
Transluce
by Transluce
Transluce: nonprofit lab building open tools for AI oversight and interpretability.
Waymo Safety Impact hub
by Waymo
Waymo Safety Impact hub: company-published crash-rate comparisons between rider-only driving miles and human benchmarks.
Anthropic Transparency Hub
by Anthropic
Anthropic Transparency Hub: system cards, model reports and voluntary commitments.
OpenAI Deployment Safety Hub
by OpenAI
OpenAI Deployment Safety Hub: system cards with safety evaluations and safeguards for deployed OpenAI models.
Final report of the Royal Commission into the Robodebt Scheme (Prime Minister of Australia)
by Prime Minister of Australia
Australian Prime Minister's media release on the final report of the Royal Commission into the Robodebt automated debt scheme.
Apollo Research: Science
Apollo Research science page listing its research and pre-deployment evaluations on AI scheming and mitigations.
NIST AI 100-2 Adversarial ML taxonomy
by NIST
NIST AI 100-2 report: taxonomy and terminology of adversarial machine learning attacks and mitigations.
Redwood Research
Redwood Research page listing its research on AI control and keeping AI systems aligned with developer intent.
A postmortem of three recent issues
by Anthropic
Anthropic postmortem explaining three infrastructure bugs that intermittently degraded Claude's responses and how they were found.
Agentic misalignment
by Anthropic
Anthropic research on agentic misalignment: stress tests where models chose blackmail or leaks under goal conflict.
An update on recent Claude Code quality reports (Anthropic)
by Anthropic
Anthropic postmortem on Claude Code quality reports, tracing degradations to three harness and prompt changes.
Goodfire research
by Goodfire
Goodfire research page: interpretability research on understanding and editing the internals of AI models.
Hidden Technical Debt in Machine Learning Systems
Google Research publication page for Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015).
InterpretML
InterpretML: open-source toolkit for glassbox models and explaining ML model behaviour.
Investigating unintended model actions in our evaluations and internal use
by Anthropic
Anthropic post investigating unintended model actions observed in its evaluations and internal use.
Leakage and the Reproducibility Crisis in ML-based Science
by Sayash Kapoor and Arvind Narayanan (Princeton University)
arXiv paper (2022) "Leakage and the Reproducibility Crisis in ML-based Science", documenting data leakage as a cause of reproducibility failures in ML-based science.