security & red-teaming: 51 curated resources
security & red-teaming
51 resources · page 1 of 2
AIxCC Competition Archive
AIxCC Competition Archive: DARPA AI Cyber Challenge records and code of finalist cyber reasoning systems.
Georgia Tech: AIxCC lessons at USENIX Security 2026
by Georgia Tech College of Computing
Georgia Tech news on its researchers' AI Cyber Challenge (AIxCC) lessons presented at USENIX Security 2026.
Haize Labs
by Haize Labs
Haize Labs: company providing red-teaming and evaluation services to move AI systems into production.
IBBIS (International Biosecurity and Biosafety Initiative for Science)
by International Biosecurity and Biosafety Initiative for Science (IBBIS)
IBBIS: international initiative strengthening biosecurity norms and tools such as DNA synthesis screening.
agent-scan (Snyk, formerly mcp-scan)
by Snyk
mcp-scan GitHub repo, now Snyk agent-scan: security scanner for AI agents, MCP servers and agent skills.
L1B3RT4S (Pliny jailbreak prompts)
by pliny
L1B3RT4S GitHub repo (Pliny): collection of jailbreak prompts for many AI models.
Meta PurpleLlama / CyberSecEval
by Meta Llama
Meta PurpleLlama GitHub repo with CyberSecEval and Llama Guard, tools to assess and improve LLM security.
NVIDIA/NemoClaw
NVIDIA NemoClaw GitHub repo: runtime for running agents such as Hermes, Deep Agents and OpenClaw.
NYU CTF Bench
by NYU LLM CTF
NYU CTF Bench GitHub repo: capture-the-flag challenges from CSAW competitions for evaluating LLM agents on security tasks.
promptfoo
by promptfoo
promptfoo GitHub repo: CLI for testing prompts, agents and RAG, with red teaming.
PyRIT
Microsoft PyRIT GitHub repo: Python Risk Identification Tool for automated red teaming of generative AI.
Team Atlanta "Atlantis" CRS (finals)
by Team-Atlanta
Team Atlanta Atlantis GitHub repo: cyber reasoning system from the DARPA AIxCC finals for automated vulnerability repair.
tldrsec prompt-injection-defenses
by tl;dr sec
tldrsec prompt-injection-defenses GitHub repo: catalogue of practical and proposed prompt injection defences.
Trail of Bits "Buttercup" CRS (finals)
Trail of Bits Buttercup GitHub repo: cyber reasoning system submitted to the DARPA AIxCC finals.
Lovable docs: Security overview
by Lovable
Lovable security overview: built-in security scans, API key protection, database access checks and dependency audits.
OWASP Top 10 for LLM Applications
by OWASP Gen AI Security Project
OWASP Top 10 for LLM Applications 2026: community list of security risks in LLM-powered applications with mitigations.
Zed docs: AI privacy and security
Zed documentation on AI privacy and security: what data leaves the machine when using AI features.
AI security / prompt injection
by OWASP Gen AI Security Project
OWASP Gen AI Security Project archive of Top 10 risk lists for LLM applications.
Beyond permission prompts (sandboxing)
by Anthropic
Anthropic engineering post on Claude Code's filesystem and network sandboxing to reduce permission prompts.
BSI Künstliche Intelligenz
by German Federal Office for Information Security (BSI)
German Federal Office for Information Security (BSI) AI page with security guidance and publications on AI systems.
Constitutional Classifiers
by Anthropic
Anthropic research post on Constitutional Classifiers, a method for defending LLMs against universal jailbreaks.
Giskard documentation
by Giskard
Giskard documentation: open-source library and platform for testing, evaluating and red-teaming LLM agents.
Google SAIF
by Google
Google Secure AI Framework (SAIF): six core elements for AI model security, risk management and privacy.
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
by Kai Greshake et al. (Saarland University)
arXiv paper (2023) by Greshake et al. introducing indirect prompt injection against LLM-integrated applications.
Hermes Agent docs: Security
Hermes Agent security documentation: command approval, user authorisation, container isolation and deployment practices.
HiddenLayer research
by HiddenLayer
HiddenLayer research page publishing findings on AI vulnerabilities, adversarial attacks and model security threats.
Artificial Intelligence (AI): Guidance for Judicial Office Holders
by Courts and Tribunals Judiciary
UK Judiciary guidance (PDF, October 2025) for judicial office holders on AI use cases, risks and verification duties.
Lakera blog
by Lakera
Lakera blog on AI security, prompt injection and LLM attacks, with research posts and product news.
Many-shot jailbreaking
by Anthropic
Anthropic research on many-shot jailbreaking, a long-context attack using many faux dialogue examples.
MCP docs: Local server security
MCP documentation on hardening local stdio MCP servers.
MITRE ATLAS
by MITRE
MITRE ATLAS: knowledge base and ATT&CK-style matrix of adversary tactics and techniques against AI systems.
NCSC/CISA Guidelines for secure AI system development
by UK National Cyber Security Centre (NCSC)
UK NCSC and partner guidelines for secure AI system development across design, development, deployment and operation.