Jailbreaking and red teaming for beginner learners
64 resources · page 2 of 2
NVIDIA/NemoClaw
Creator not supplied; see original source.
NVIDIA NemoClaw GitHub repo: runtime for running agents such as Hermes, Deep Agents and OpenClaw.
NYU CTF Bench
Creator not supplied; see original source.
NYU CTF Bench GitHub repo: capture-the-flag challenges from CSAW competitions for evaluating LLM agents on security tasks.
OpenFHE
Creator not supplied; see original source.
OpenFHE GitHub repo: open-source library for fully homomorphic encryption.
promptfoo
Creator not supplied; see original source.
promptfoo GitHub repo: CLI for testing prompts, agents and RAG, with red teaming.
PyRIT
Creator not supplied; see original source.
Microsoft PyRIT GitHub repo: Python Risk Identification Tool for automated red teaming of generative AI.
PySyft
Creator not supplied; see original source.
OpenMined PySyft GitHub repo: data science on data that stays on the owner's server, for privacy-preserving access.
SAELens
Creator not supplied; see original source.
SAELens GitHub repo: training and analysing sparse autoencoders on language models.
Safe Superintelligence Inc.
Creator not supplied; see original source.
Safe Superintelligence Inc. (SSI): company focused solely on building safe superintelligence; site has a short mission statement.
STAT: IBM Watson recommended "unsafe and incorrect" cancer treatments
Creator not supplied; see original source.
STAT report on internal IBM documents showing Watson for Oncology gave unsafe and incorrect treatment advice.
Stephenson Harwood: "Law and error"
Creator not supplied; see original source.
Stephenson Harwood article on court scrutiny of AI tools after fictitious citations in legal filings.
Team Atlanta "Atlantis" CRS (finals)
Creator not supplied; see original source.
Team Atlanta Atlantis GitHub repo: cyber reasoning system from the DARPA AIxCC finals for automated vulnerability repair.
The Register: Figma AI accused of plagiarising Apple
Creator not supplied; see original source.
The Register report on Figma pulling its Make Designs AI feature after it copied Apple's Weather app.
TILOS MacroPlacement
Creator not supplied; see original source.
TILOS MacroPlacement GitHub repo: benchmarks, evaluators and an open reproduction of Google's Circuit Training for chip macro placement.
tldrsec prompt-injection-defenses
Creator not supplied; see original source.
tldrsec prompt-injection-defenses GitHub repo: catalogue of practical and proposed prompt injection defences.
Trail of Bits "Buttercup" CRS (finals)
Creator not supplied; see original source.
Trail of Bits Buttercup GitHub repo: cyber reasoning system submitted to the DARPA AIxCC finals.
TransformerLens
Creator not supplied; see original source.
TransformerLens GitHub repo: library for mechanistic interpretability of GPT-style language models.
Transluce
Creator not supplied; see original source.
Transluce: nonprofit lab building open tools for AI oversight and interpretability.
Waymo Safety Impact hub
Creator not supplied; see original source.
Waymo Safety Impact hub: company-published crash-rate comparisons between rider-only driving miles and human benchmarks.
Anthropic Transparency Hub
Creator not supplied; see original source.
Anthropic Transparency Hub: system cards, model reports and voluntary commitments.
BC CRT decision document (primary)
Creator not supplied; see original source.
British Columbia Civil Resolution Tribunal decision document; the tribunal decided the Moffatt v. Air Canada chatbot case.
AI Agent Wipes Production Database, Then Lies About It (eWeek)
Creator not supplied; see original source.
eWeek report on a Replit AI agent deleting a company's production database during a code freeze in a SaaStr experiment.
GitHub Docs: Responsible use of Copilot agents
Creator not supplied; see original source.
GitHub documentation source on responsible use of Copilot agents, with stated limitations and review duties.
Indirect prompt injection in the real world: how people manipulate neural networks
by Vladislav Tushkanov (Kaspersky)
Read a Russia-origin security team’s field study of indirect prompt injection and the implications for defensive risk assessment.
Lovable docs: Security overview
Creator not supplied; see original source.
Lovable security overview: built-in security scans, API key protection, database access checks and dependency audits.
Airline Ordered To Compensate A B.C. Man Because Its Chatbot Provided Inaccurate Information (Mondaq)
Creator not supplied; see original source.
Mondaq summary of Moffatt v. Air Canada (2024 BCCRT 149), where the airline was held liable for its chatbot's misinformation.
OpenAI Deployment Safety Hub
Creator not supplied; see original source.
OpenAI Deployment Safety Hub: system cards with safety evaluations and safeguards for deployed OpenAI models.
OWASP Top 10 for LLM Applications
Creator not supplied; see original source.
OWASP Top 10 for LLM Applications 2026: community list of security risks in LLM-powered applications with mitigations.
Final report of the Royal Commission into the Robodebt Scheme (Prime Minister of Australia)
Creator not supplied; see original source.
Australian Prime Minister's media release on the final report of the Royal Commission into the Robodebt automated debt scheme.
Simon Willison on the AI hallucination cases database
Creator not supplied; see original source.
Simon Willison's post on Damien Charlotin's database of legal decisions involving AI-hallucinated content.
Replit makes vibe-y promise to prevent vibe coding disasters (The Register)
Creator not supplied; see original source.
The Register follow-up on Replit's response, including separate development and production databases.
Vibe coding service Replit deleted production database (The Register)
Creator not supplied; see original source.
The Register report on Replit's agent deleting SaaStr's production database during a code freeze in July 2025.
Zed docs: AI privacy and security
Creator not supplied; see original source.
Zed documentation on AI privacy and security: what data leaves the machine when using AI features.