Agent Coding Evals
85 resources · page 3 of 3
Berget AI: AI in Swedish
Creator not supplied; see original source.
Berget AI post on choosing models for Swedish-language applications, pointing to EuroEval and Scandinavian embedding benchmarks.
Demystifying evals for AI agents
Creator not supplied; see original source.
Anthropic post on designing evals for AI agents, combining automated tests, graders and human review.
GAIA paper
Creator not supplied; see original source.
arXiv paper (2023) introducing GAIA, a benchmark of real-world questions for general AI assistants.
Announcing Agent Control (Galileo)
Creator not supplied; see original source.
Galileo post announcing Agent Control, an open-source control plane for runtime guardrails on AI agents.
Launching Agent Leaderboard v2 (Galileo)
Creator not supplied; see original source.
Galileo post introducing Agent Leaderboard v2, which evaluates LLMs as enterprise customer-support agents.
Best Practices to Navigate the Complexities of Evaluating AI Agents (Galileo)
Creator not supplied; see original source.
Galileo guide to evaluating AI agents, covering accuracy metrics, action evaluation and production monitoring.
The Most Common Questions About Agent Evals, Answered (LangChain)
Creator not supplied; see original source.
LangChain post answering common questions on agent evals: offline vs online, evaluator types, LLM judges and multi-turn tests.
Langfuse blog: Evaluating AI agent skills
Creator not supplied; see original source.
Langfuse blog post on how to evaluate AI agent skills (SKILL.md files) with traces and evals.
LiveCodeBench paper
Creator not supplied; see original source.
arXiv paper (2024) introducing LiveCodeBench, a contamination-free code benchmark built from recent contest problems.
METR time-horizon paper
Creator not supplied; see original source.
arXiv paper (2025) by METR measuring AI ability to complete long software tasks as a time horizon.
promptfoo docs: Evaluate coding agents
Creator not supplied; see original source.
Promptfoo guide to evaluating coding agents with its open-source eval harness.
SWE-bench paper
Creator not supplied; see original source.
arXiv paper (2023) introducing SWE-bench, evaluating language models on resolving real GitHub issues.
The SWE-Bench Illusion
Creator not supplied; see original source.
arXiv paper (2025) "The SWE-Bench Illusion", testing whether LLMs memorise SWE-bench instead of reasoning.
Agentic Benchmark Checklist (ABC)
Creator not supplied; see original source.
arXiv paper (2025) proposing the Agentic Benchmark Checklist, guidelines for building valid agentic benchmarks.
Arize Phoenix docs: Evaluating multi-agent systems
Creator not supplied; see original source.
Arize Phoenix documentation on evaluating multi-agent systems using traces of each agent's steps.
Coding-agent leaderboard resolution audit
Creator not supplied; see original source.
arXiv paper (2026) arguing that coding agents have converged so the SWE-bench leaderboard can no longer separate them.
HAE-RAE Evaluation Toolkit / HRET
Creator not supplied; see original source.
arXiv paper (2025) introducing the HAE-RAE Evaluation Toolkit (HRET), a unified framework for evaluating Korean LLMs.
Introducing TRAIL: A Benchmark for Agentic Evaluation (Patronus AI)
Creator not supplied; see original source.
Patronus AI post introducing TRAIL, a benchmark of annotated agent traces for detecting reasoning, planning and execution errors.
Quantifying infrastructure noise in agentic coding evals
Creator not supplied; see original source.
Anthropic engineering post quantifying how infrastructure noise affects agentic coding evaluation scores.
SWE-Bench Pro Verified
Creator not supplied; see original source.
arXiv paper (2026) introducing SWE-Bench Pro Verified, an audited software-engineering benchmark that removes leakage and flawed tasks.
EvalScope τ-bench docs (ModelScope/Alibaba)
Creator not supplied; see original source.
EvalScope documentation on τ-bench: running the conversational agent benchmark with Alibaba's ModelScope evaluation framework.