Safety And Alignment Evals: 126 curated resources

Curated material on safety and alignment evals within security and red teaming.

126 resources · page 2 of 4

• NVIDIA/NemoClaw

  Creator not supplied; see original source.

  NVIDIA NemoClaw GitHub repo: runtime for running agents such as Hermes, Deep Agents and OpenClaw.

  Not applicable (entity) · unknown · unknown resource · en

• NYU CTF Bench

  Creator not supplied; see original source.

  NYU CTF Bench GitHub repo: capture-the-flag challenges from CSAW competitions for evaluating LLM agents on security tasks.

  Not applicable (entity) · unknown · unknown resource · en

• OpenFHE

  Creator not supplied; see original source.

  OpenFHE GitHub repo: open-source library for fully homomorphic encryption.

  Not applicable (entity) · unknown · unknown resource · en

• promptfoo

  Creator not supplied; see original source.

  promptfoo GitHub repo: CLI for testing prompts, agents and RAG, with red teaming.

  Not applicable (entity) · unknown · unknown resource · en

• PyRIT

  Creator not supplied; see original source.

  Microsoft PyRIT GitHub repo: Python Risk Identification Tool for automated red teaming of generative AI.

  Not applicable (entity) · unknown · unknown resource · en

• PySyft

  Creator not supplied; see original source.

  OpenMined PySyft GitHub repo: data science on data that stays on the owner's server, for privacy-preserving access.

  Not applicable (entity) · unknown · unknown resource · en

• SAELens

  Creator not supplied; see original source.

  SAELens GitHub repo: training and analysing sparse autoencoders on language models.

  Not applicable (entity) · unknown · unknown resource · en

• Safe Superintelligence Inc.

  Creator not supplied; see original source.

  Safe Superintelligence Inc. (SSI): company focused solely on building safe superintelligence; site has a short mission statement.

  Not applicable (entity) · unknown · unknown resource · en

• STAT: IBM Watson recommended "unsafe and incorrect" cancer treatments

  Creator not supplied; see original source.

  STAT report on internal IBM documents showing Watson for Oncology gave unsafe and incorrect treatment advice.

  Not applicable (entity) · unknown · unknown resource · en

• Stephenson Harwood: "Law and error"

  Creator not supplied; see original source.

  Stephenson Harwood article on court scrutiny of AI tools after fictitious citations in legal filings.

  Not applicable (entity) · unknown · unknown resource · en

• Team Atlanta "Atlantis" CRS (finals)

  Creator not supplied; see original source.

  Team Atlanta Atlantis GitHub repo: cyber reasoning system from the DARPA AIxCC finals for automated vulnerability repair.

  Not applicable (entity) · unknown · unknown resource · en

• The Register: Figma AI accused of plagiarising Apple

  Creator not supplied; see original source.

  The Register report on Figma pulling its Make Designs AI feature after it copied Apple's Weather app.

  Not applicable (entity) · unknown · unknown resource · en

• TILOS MacroPlacement

  Creator not supplied; see original source.

  TILOS MacroPlacement GitHub repo: benchmarks, evaluators and an open reproduction of Google's Circuit Training for chip macro placement.

  Not applicable (entity) · unknown · unknown resource · en

• tldrsec prompt-injection-defenses

  Creator not supplied; see original source.

  tldrsec prompt-injection-defenses GitHub repo: catalogue of practical and proposed prompt injection defences.

  Not applicable (entity) · unknown · unknown resource · en

• Trail of Bits "Buttercup" CRS (finals)

  Creator not supplied; see original source.

  Trail of Bits Buttercup GitHub repo: cyber reasoning system submitted to the DARPA AIxCC finals.

  Not applicable (entity) · unknown · unknown resource · en

• TransformerLens

  Creator not supplied; see original source.

  TransformerLens GitHub repo: library for mechanistic interpretability of GPT-style language models.

  Not applicable (entity) · unknown · unknown resource · en

• Transluce

  Creator not supplied; see original source.

  Transluce: nonprofit lab building open tools for AI oversight and interpretability.

  Not applicable (entity) · unknown · unknown resource · en

• Waymo Safety Impact hub

  Creator not supplied; see original source.

  Waymo Safety Impact hub: company-published crash-rate comparisons between rider-only driving miles and human benchmarks.

  Not applicable (entity) · unknown · unknown resource · en

• Anthropic Transparency Hub

  Creator not supplied; see original source.

  Anthropic Transparency Hub: system cards, model reports and voluntary commitments.

  Beginner–Expert · unknown · unknown resource · en

• BC CRT decision document (primary)

  Creator not supplied; see original source.

  British Columbia Civil Resolution Tribunal decision document; the tribunal decided the Moffatt v. Air Canada chatbot case.

  Beginner · unknown · unknown resource · en

• AI Agent Wipes Production Database, Then Lies About It (eWeek)

  Creator not supplied; see original source.

  eWeek report on a Replit AI agent deleting a company's production database during a code freeze in a SaaStr experiment.

  Beginner · unknown · unknown resource · en

• GitHub Docs: Responsible use of Copilot agents

  Creator not supplied; see original source.

  GitHub documentation source on responsible use of Copilot agents, with stated limitations and review duties.

  Beginner · unknown · unknown resource · en

• Lovable docs: Security overview

  Creator not supplied; see original source.

  Lovable security overview: built-in security scans, API key protection, database access checks and dependency audits.

  Beginner · unknown · unknown resource · en

• Airline Ordered To Compensate A B.C. Man Because Its Chatbot Provided Inaccurate Information (Mondaq)

  Creator not supplied; see original source.

  Mondaq summary of Moffatt v. Air Canada (2024 BCCRT 149), where the airline was held liable for its chatbot's misinformation.

  Beginner · unknown · unknown resource · en

• OpenAI Deployment Safety Hub

  Creator not supplied; see original source.

  OpenAI Deployment Safety Hub: system cards with safety evaluations and safeguards for deployed OpenAI models.

  Beginner–Expert · unknown · unknown resource · en

• OWASP Top 10 for LLM Applications

  Creator not supplied; see original source.

  OWASP Top 10 for LLM Applications 2026: community list of security risks in LLM-powered applications with mitigations.

  Beginner–Intermediate · unknown · unknown resource · en

• Final report of the Royal Commission into the Robodebt Scheme (Prime Minister of Australia)

  Creator not supplied; see original source.

  Australian Prime Minister's media release on the final report of the Royal Commission into the Robodebt automated debt scheme.

  Beginner–Expert · unknown · unknown resource · en

• Simon Willison on the AI hallucination cases database

  Creator not supplied; see original source.

  Simon Willison's post on Damien Charlotin's database of legal decisions involving AI-hallucinated content.

  Beginner · unknown · unknown resource · en

• Replit makes vibe-y promise to prevent vibe coding disasters (The Register)

  Creator not supplied; see original source.

  The Register follow-up on Replit's response, including separate development and production databases.

  Beginner · unknown · unknown resource · en

• Vibe coding service Replit deleted production database (The Register)

  Creator not supplied; see original source.

  The Register report on Replit's agent deleting SaaStr's production database during a code freeze in July 2025.

  Beginner · unknown · unknown resource · en

• Zed docs: AI privacy and security

  Creator not supplied; see original source.

  Zed documentation on AI privacy and security: what data leaves the machine when using AI features.

  Beginner · unknown · unknown resource · en

• AI Safety Governance Framework v1.0 (TC260) on ETO AGORA

  Creator not supplied; see original source.

  ETO AGORA record of China's AI Safety Governance Framework v1.0 published by TC260 in 2024.

  Intermediate · unknown · unknown resource · en

Plain-text Markdown copy