agent & coding evals: 85 curated resources
agent & coding evals
85 resources · page 3 of 3
Berget AI: AI in Swedish
by Berget AI
Berget AI post on choosing models for Swedish-language applications, pointing to EuroEval and Scandinavian embedding benchmarks.
Demystifying evals for AI agents
by Anthropic
Anthropic post on designing evals for AI agents, combining automated tests, graders and human review.
GAIA paper
by Grégoire Mialon et al. (Meta AI and Hugging Face)
arXiv paper (2023) introducing GAIA, a benchmark of real-world questions for general AI assistants.
Announcing Agent Control (Galileo)
by Galileo
Galileo post announcing Agent Control, an open-source control plane for runtime guardrails on AI agents.
Launching Agent Leaderboard v2 (Galileo)
by Galileo
Galileo post introducing Agent Leaderboard v2, which evaluates LLMs as enterprise customer-support agents.
Best Practices to Navigate the Complexities of Evaluating AI Agents (Galileo)
by Galileo
Galileo guide to evaluating AI agents, covering accuracy metrics, action evaluation and production monitoring.
The Most Common Questions About Agent Evals, Answered (LangChain)
by LangChain
LangChain post answering common questions on agent evals: offline vs online, evaluator types, LLM judges and multi-turn tests.
Langfuse blog: Evaluating AI agent skills
by Langfuse
Langfuse blog post on how to evaluate AI agent skills (SKILL.md files) with traces and evals.
LiveCodeBench paper
by Naman Jain et al. (Cornell University)
arXiv paper (2024) introducing LiveCodeBench, a contamination-free code benchmark built from recent contest problems.
METR time-horizon paper
arXiv paper (2025) by METR measuring AI ability to complete long software tasks as a time horizon.
promptfoo docs: Evaluate coding agents
by promptfoo
Promptfoo guide to evaluating coding agents with its open-source eval harness.
SWE-bench paper
by Carlos E. Jimenez et al. (Princeton University)
arXiv paper (2023) introducing SWE-bench, evaluating language models on resolving real GitHub issues.
The SWE-Bench Illusion
by Shanchao Liang et al. (Purdue University)
arXiv paper (2025) "The SWE-Bench Illusion", testing whether LLMs memorise SWE-bench instead of reasoning.
Agentic Benchmark Checklist (ABC)
arXiv paper (2025) proposing the Agentic Benchmark Checklist, guidelines for building valid agentic benchmarks.
Arize Phoenix docs: Evaluating multi-agent systems
by Arize AI
Arize Phoenix documentation on evaluating multi-agent systems using traces of each agent's steps.
Coding-agent leaderboard resolution audit
by Fengshuo Liu et al. (Imperial College London)
arXiv paper (2026) arguing that coding agents have converged so the SWE-bench leaderboard can no longer separate them.
HAE-RAE Evaluation Toolkit / HRET
arXiv paper (2025) introducing the HAE-RAE Evaluation Toolkit (HRET), a unified framework for evaluating Korean LLMs.
Introducing TRAIL: A Benchmark for Agentic Evaluation (Patronus AI)
by Patronus AI
Patronus AI post introducing TRAIL, a benchmark of annotated agent traces for detecting reasoning, planning and execution errors.
Quantifying infrastructure noise in agentic coding evals
by Anthropic
Anthropic engineering post quantifying how infrastructure noise affects agentic coding evaluation scores.
SWE-Bench Pro Verified
by Pujun Zheng et al. (Shanghai Artificial Intelligence Laboratory)
arXiv paper (2026) introducing SWE-Bench Pro Verified, an audited software-engineering benchmark that removes leakage and flawed tasks.
EvalScope τ-bench docs (ModelScope/Alibaba)
EvalScope documentation on τ-bench: running the conversational agent benchmark with Alibaba's ModelScope evaluation framework.