Safety, security & interpretability for intermediate learners
90 resources · page 2 of 3
NVIDIA/NemoClaw
NVIDIA NemoClaw GitHub repo: runtime for running agents such as Hermes, Deep Agents and OpenClaw.
NYU CTF Bench
by NYU LLM CTF
NYU CTF Bench GitHub repo: capture-the-flag challenges from CSAW competitions for evaluating LLM agents on security tasks.
OpenFHE
by OpenFHE
OpenFHE GitHub repo: open-source library for fully homomorphic encryption.
promptfoo
by promptfoo
promptfoo GitHub repo: CLI for testing prompts, agents and RAG, with red teaming.
PyRIT
Microsoft PyRIT GitHub repo: Python Risk Identification Tool for automated red teaming of generative AI.
PySyft
by OpenMined
OpenMined PySyft GitHub repo: data science on data that stays on the owner's server, for privacy-preserving access.
SAELens
SAELens GitHub repo: training and analysing sparse autoencoders on language models.
Safe Superintelligence Inc.
by Safe Superintelligence Inc.
Safe Superintelligence Inc. (SSI): company focused solely on building safe superintelligence; site has a short mission statement.
STAT: IBM Watson recommended "unsafe and incorrect" cancer treatments
by STAT
STAT report on internal IBM documents showing Watson for Oncology gave unsafe and incorrect treatment advice.
Stephenson Harwood: "Law and error"
Stephenson Harwood article on court scrutiny of AI tools after fictitious citations in legal filings.
Team Atlanta "Atlantis" CRS (finals)
by Team-Atlanta
Team Atlanta Atlantis GitHub repo: cyber reasoning system from the DARPA AIxCC finals for automated vulnerability repair.
The Register: Figma AI accused of plagiarising Apple
by The Register
The Register report on Figma pulling its Make Designs AI feature after it copied Apple's Weather app.
TILOS MacroPlacement
TILOS MacroPlacement GitHub repo: benchmarks, evaluators and an open reproduction of Google's Circuit Training for chip macro placement.
tldrsec prompt-injection-defenses
by tl;dr sec
tldrsec prompt-injection-defenses GitHub repo: catalogue of practical and proposed prompt injection defences.
Trail of Bits "Buttercup" CRS (finals)
Trail of Bits Buttercup GitHub repo: cyber reasoning system submitted to the DARPA AIxCC finals.
TransformerLens
TransformerLens GitHub repo: library for mechanistic interpretability of GPT-style language models.
Transluce
by Transluce
Transluce: nonprofit lab building open tools for AI oversight and interpretability.
Waymo Safety Impact hub
by Waymo
Waymo Safety Impact hub: company-published crash-rate comparisons between rider-only driving miles and human benchmarks.
Anthropic Transparency Hub
by Anthropic
Anthropic Transparency Hub: system cards, model reports and voluntary commitments.
OpenAI Deployment Safety Hub
by OpenAI
OpenAI Deployment Safety Hub: system cards with safety evaluations and safeguards for deployed OpenAI models.
OWASP Top 10 for LLM Applications
by OWASP Gen AI Security Project
OWASP Top 10 for LLM Applications 2026: community list of security risks in LLM-powered applications with mitigations.
Final report of the Royal Commission into the Robodebt Scheme (Prime Minister of Australia)
by Prime Minister of Australia
Australian Prime Minister's media release on the final report of the Royal Commission into the Robodebt automated debt scheme.
AI Safety Governance Framework v1.0 (TC260) on ETO AGORA
by Emerging Technology Observatory (CSET, Georgetown University)
ETO AGORA record of China's AI Safety Governance Framework v1.0 published by TC260 in 2024.
AI security / prompt injection
by OWASP Gen AI Security Project
OWASP Gen AI Security Project archive of Top 10 risk lists for LLM applications.
Apollo Research: Science
Apollo Research science page listing its research and pre-deployment evaluations on AI scheming and mitigations.
Beyond permission prompts (sandboxing)
by Anthropic
Anthropic engineering post on Claude Code's filesystem and network sandboxing to reduce permission prompts.
BSI Künstliche Intelligenz
by German Federal Office for Information Security (BSI)
German Federal Office for Information Security (BSI) AI page with security guidance and publications on AI systems.
Constitutional Classifiers
by Anthropic
Anthropic research post on Constitutional Classifiers, a method for defending LLMs against universal jailbreaks.
Giskard documentation
by Giskard
Giskard documentation: open-source library and platform for testing, evaluating and red-teaming LLM agents.
Google SAIF
by Google
Google Secure AI Framework (SAIF): six core elements for AI model security, risk management and privacy.
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
by Kai Greshake et al. (Saarland University)
arXiv paper (2023) by Greshake et al. introducing indirect prompt injection against LLM-integrated applications.
Hermes Agent docs: Security
Hermes Agent security documentation: command approval, user authorisation, container isolation and deployment practices.