Safety, security & interpretability
Safety, security & interpretability
- alignment (3)
- incidents & failure cases (41)
- interpretability (13)
- safety evals (16)
- security & red-teaming (51)
124 resources · page 2 of 4
NVIDIA/NemoClaw
NVIDIA NemoClaw GitHub repo: runtime for running agents such as Hermes, Deep Agents and OpenClaw.
NYU CTF Bench
by NYU LLM CTF
NYU CTF Bench GitHub repo: capture-the-flag challenges from CSAW competitions for evaluating LLM agents on security tasks.
OpenFHE
by OpenFHE
OpenFHE GitHub repo: open-source library for fully homomorphic encryption.
promptfoo
by promptfoo
promptfoo GitHub repo: CLI for testing prompts, agents and RAG, with red teaming.
PyRIT
Microsoft PyRIT GitHub repo: Python Risk Identification Tool for automated red teaming of generative AI.
PySyft
by OpenMined
OpenMined PySyft GitHub repo: data science on data that stays on the owner's server, for privacy-preserving access.
SAELens
SAELens GitHub repo: training and analysing sparse autoencoders on language models.
Safe Superintelligence Inc.
by Safe Superintelligence Inc.
Safe Superintelligence Inc. (SSI): company focused solely on building safe superintelligence; site has a short mission statement.
STAT: IBM Watson recommended "unsafe and incorrect" cancer treatments
by STAT
STAT report on internal IBM documents showing Watson for Oncology gave unsafe and incorrect treatment advice.
Stephenson Harwood: "Law and error"
Stephenson Harwood article on court scrutiny of AI tools after fictitious citations in legal filings.
Team Atlanta "Atlantis" CRS (finals)
by Team-Atlanta
Team Atlanta Atlantis GitHub repo: cyber reasoning system from the DARPA AIxCC finals for automated vulnerability repair.
The Register: Figma AI accused of plagiarising Apple
by The Register
The Register report on Figma pulling its Make Designs AI feature after it copied Apple's Weather app.
TILOS MacroPlacement
TILOS MacroPlacement GitHub repo: benchmarks, evaluators and an open reproduction of Google's Circuit Training for chip macro placement.
tldrsec prompt-injection-defenses
by tl;dr sec
tldrsec prompt-injection-defenses GitHub repo: catalogue of practical and proposed prompt injection defences.
Trail of Bits "Buttercup" CRS (finals)
Trail of Bits Buttercup GitHub repo: cyber reasoning system submitted to the DARPA AIxCC finals.
TransformerLens
TransformerLens GitHub repo: library for mechanistic interpretability of GPT-style language models.
Transluce
by Transluce
Transluce: nonprofit lab building open tools for AI oversight and interpretability.
Waymo Safety Impact hub
by Waymo
Waymo Safety Impact hub: company-published crash-rate comparisons between rider-only driving miles and human benchmarks.
Anthropic Transparency Hub
by Anthropic
Anthropic Transparency Hub: system cards, model reports and voluntary commitments.
BC CRT decision document (primary)
by Civil Resolution Tribunal (British Columbia)
British Columbia Civil Resolution Tribunal decision document; the tribunal decided the Moffatt v. Air Canada chatbot case.
AI Agent Wipes Production Database, Then Lies About It (eWeek)
by eWeek
eWeek report on a Replit AI agent deleting a company's production database during a code freeze in a SaaStr experiment.
GitHub Docs: Responsible use of Copilot agents
by GitHub
GitHub documentation source on responsible use of Copilot agents, with stated limitations and review duties.
Lovable docs: Security overview
by Lovable
Lovable security overview: built-in security scans, API key protection, database access checks and dependency audits.
Airline Ordered To Compensate A B.C. Man Because Its Chatbot Provided Inaccurate Information (Mondaq)
by Mondaq
Mondaq summary of Moffatt v. Air Canada (2024 BCCRT 149), where the airline was held liable for its chatbot's misinformation.
OpenAI Deployment Safety Hub
by OpenAI
OpenAI Deployment Safety Hub: system cards with safety evaluations and safeguards for deployed OpenAI models.
OWASP Top 10 for LLM Applications
by OWASP Gen AI Security Project
OWASP Top 10 for LLM Applications 2026: community list of security risks in LLM-powered applications with mitigations.
Final report of the Royal Commission into the Robodebt Scheme (Prime Minister of Australia)
by Prime Minister of Australia
Australian Prime Minister's media release on the final report of the Royal Commission into the Robodebt automated debt scheme.
Simon Willison on the AI hallucination cases database
Simon Willison's post on Damien Charlotin's database of legal decisions involving AI-hallucinated content.
Replit makes vibe-y promise to prevent vibe coding disasters (The Register)
by The Register
The Register follow-up on Replit's response, including separate development and production databases.
Vibe coding service Replit deleted production database (The Register)
by The Register
The Register report on Replit's agent deleting SaaStr's production database during a code freeze in July 2025.
Zed docs: AI privacy and security
Zed documentation on AI privacy and security: what data leaves the machine when using AI features.
AI Safety Governance Framework v1.0 (TC260) on ETO AGORA
by Emerging Technology Observatory (CSET, Georgetown University)
ETO AGORA record of China's AI Safety Governance Framework v1.0 published by TC260 in 2024.