Security Red Teaming
51 resources · page 1 of 2
AIxCC Competition Archive
Creator not supplied; see original source.
AIxCC Competition Archive: DARPA AI Cyber Challenge records and code of finalist cyber reasoning systems.
Georgia Tech: AIxCC lessons at USENIX Security 2026
Creator not supplied; see original source.
Georgia Tech news on its researchers' AI Cyber Challenge (AIxCC) lessons presented at USENIX Security 2026.
Haize Labs
Creator not supplied; see original source.
Haize Labs: company providing red-teaming and evaluation services to move AI systems into production.
IBBIS (International Biosecurity and Biosafety Initiative for Science)
Creator not supplied; see original source.
IBBIS: international initiative strengthening biosecurity norms and tools such as DNA synthesis screening.
agent-scan (Snyk, formerly mcp-scan)
Creator not supplied; see original source.
mcp-scan GitHub repo, now Snyk agent-scan: security scanner for AI agents, MCP servers and agent skills.
L1B3RT4S (Pliny jailbreak prompts)
Creator not supplied; see original source.
L1B3RT4S GitHub repo (Pliny): collection of jailbreak prompts for many AI models.
Meta PurpleLlama / CyberSecEval
Creator not supplied; see original source.
Meta PurpleLlama GitHub repo with CyberSecEval and Llama Guard, tools to assess and improve LLM security.
NVIDIA/NemoClaw
Creator not supplied; see original source.
NVIDIA NemoClaw GitHub repo: runtime for running agents such as Hermes, Deep Agents and OpenClaw.
NYU CTF Bench
Creator not supplied; see original source.
NYU CTF Bench GitHub repo: capture-the-flag challenges from CSAW competitions for evaluating LLM agents on security tasks.
promptfoo
Creator not supplied; see original source.
promptfoo GitHub repo: CLI for testing prompts, agents and RAG, with red teaming.
PyRIT
Creator not supplied; see original source.
Microsoft PyRIT GitHub repo: Python Risk Identification Tool for automated red teaming of generative AI.
Team Atlanta "Atlantis" CRS (finals)
Creator not supplied; see original source.
Team Atlanta Atlantis GitHub repo: cyber reasoning system from the DARPA AIxCC finals for automated vulnerability repair.
tldrsec prompt-injection-defenses
Creator not supplied; see original source.
tldrsec prompt-injection-defenses GitHub repo: catalogue of practical and proposed prompt injection defences.
Trail of Bits "Buttercup" CRS (finals)
Creator not supplied; see original source.
Trail of Bits Buttercup GitHub repo: cyber reasoning system submitted to the DARPA AIxCC finals.
Lovable docs: Security overview
Creator not supplied; see original source.
Lovable security overview: built-in security scans, API key protection, database access checks and dependency audits.
OWASP Top 10 for LLM Applications
Creator not supplied; see original source.
OWASP Top 10 for LLM Applications 2026: community list of security risks in LLM-powered applications with mitigations.
Zed docs: AI privacy and security
Creator not supplied; see original source.
Zed documentation on AI privacy and security: what data leaves the machine when using AI features.
AI security / prompt injection
Creator not supplied; see original source.
OWASP Gen AI Security Project archive of Top 10 risk lists for LLM applications.
Beyond permission prompts (sandboxing)
Creator not supplied; see original source.
Anthropic engineering post on Claude Code's filesystem and network sandboxing to reduce permission prompts.
BSI Künstliche Intelligenz
Creator not supplied; see original source.
German Federal Office for Information Security (BSI) AI page with security guidance and publications on AI systems.
Constitutional Classifiers
Creator not supplied; see original source.
Anthropic research post on Constitutional Classifiers, a method for defending LLMs against universal jailbreaks.
Giskard documentation
Creator not supplied; see original source.
Giskard documentation: open-source library and platform for testing, evaluating and red-teaming LLM agents.
Google SAIF
Creator not supplied; see original source.
Google Secure AI Framework (SAIF): six core elements for AI model security, risk management and privacy.
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Creator not supplied; see original source.
arXiv paper (2023) by Greshake et al. introducing indirect prompt injection against LLM-integrated applications.
Hermes Agent docs: Security
Creator not supplied; see original source.
Hermes Agent security documentation: command approval, user authorisation, container isolation and deployment practices.
HiddenLayer research
Creator not supplied; see original source.
HiddenLayer research page publishing findings on AI vulnerabilities, adversarial attacks and model security threats.
Artificial Intelligence (AI): Guidance for Judicial Office Holders
Creator not supplied; see original source.
UK Judiciary guidance (PDF, October 2025) for judicial office holders on AI use cases, risks and verification duties.
Lakera blog
Creator not supplied; see original source.
Lakera blog on AI security, prompt injection and LLM attacks, with research posts and product news.
Many-shot jailbreaking
Creator not supplied; see original source.
Anthropic research on many-shot jailbreaking, a long-context attack using many faux dialogue examples.
MCP docs: Local server security
Creator not supplied; see original source.
MCP documentation on hardening local stdio MCP servers.
MITRE ATLAS
Creator not supplied; see original source.
MITRE ATLAS: knowledge base and ATT&CK-style matrix of adversary tactics and techniques against AI systems.
NCSC/CISA Guidelines for secure AI system development
Creator not supplied; see original source.
UK NCSC and partner guidelines for secure AI system development across design, development, deployment and operation.