agent & coding evals: 85 curated resources
agent & coding evals
85 resources · page 1 of 3
AA-3D
Artificial Analysis AA-3D benchmark comparing how well AI models build interactive 3D experiences, judged by blind panel comparisons.
Artificial Analysis Intelligence Benchmarking methodology
Artificial Analysis methodology page describing the benchmarks, weights and scoring behind its Intelligence Index for LLMs.
AgentBench
by THUKEG
AgentBench GitHub repo (Tsinghua): benchmark evaluating LLMs as agents across multiple interactive environments.
AgentBoard
HKUST AgentBoard GitHub repo: analytical evaluation board for multi-turn LLM agents with progress tracking.
ethz-spylab/agentdojo
by SPY Lab
ETH SPY Lab AgentDojo GitHub repo: environment evaluating prompt injection attacks and defences for LLM agents.
ai-safety-institute/AgentHarm
UK AISI AgentHarm dataset on Hugging Face: harmful agentic tasks for measuring whether LLM agents comply with misuse requests.
AI Verify Foundation / Project Moonshot (Singapore)
Singapore's AI Verify Foundation site, home of the AI Verify testing toolkit and Project Moonshot for LLM red-teaming and benchmarking.
Aider polyglot leaderboard
by Aider
Aider LLM leaderboards: model scores on polyglot code-editing exercises with cost.
AndroidWorld
Google Research AndroidWorld GitHub repo: environment and benchmark for autonomous agents on Android.
AppWorld
by AppWorld team (Stony Brook University)
AppWorld benchmark: a simulated world of everyday apps and APIs for testing coding agents on stateful multi-app tasks.
Arena Raises $200M Series B at $3.1B Valuation
Arena blog post on its Series B and the launch of its AI Alignment Index, measuring unsafe agent behaviour on real traces.
augment-swebench-agent
by Augment Code
Augment augment-swebench-agent GitHub repo: open implementation of Augment's SWE-bench Verified agent.
sierra-research/tau-bench
Sierra τ-bench GitHub repo: code and data for evaluating tool-using agents in customer service domains.
Berkeley Function-Calling Leaderboard
Berkeley Function Calling Leaderboard (BFCL V4): model accuracy on function and tool calling, including agentic tasks.
BigCodeBench
BigCodeBench leaderboard: practical programming tasks that require calling many libraries, for evaluating code LLMs.
BioASQ
by BioASQ
BioASQ site: recurring challenges on biomedical semantic indexing and question answering, with datasets and results.
BIRD
by bird_sql
BIRD benchmark site: large-scale text-to-SQL evaluation over real databases with dirty values and a public leaderboard.
BrowserGym
by ServiceNow
ServiceNow BrowserGym GitHub repo: Gym environment for evaluating web task automation agents.
CodeClash
CodeClash benchmark: AI coding agents compete in multi-round tournaments to pursue open-ended goals in codebases.
CyberGym
by sunblaze-ucb
Berkeley CyberGym GitHub repo: evaluation of AI agents on real software vulnerabilities.
EvalPlus
by evalplus
EvalPlus site: HumanEval+ and MBPP+ with extended test cases for more accurate evaluation of code LLMs.
GAIA
by GAIA
GAIA organisation page on Hugging Face: benchmark of real-world questions for general AI assistants needing reasoning, browsing and tools.
GAIA leaderboard
by GAIA
GAIA leaderboard Space: scores of AI assistants and agents on the GAIA real-world question benchmark.
HAL: Holistic Agent Leaderboard
Princeton HAL Holistic Agent Leaderboard: third-party, cost-aware evaluation of AI agents across benchmarks.
Harbor framework
by Harbor framework project (Laude Institute)
Harbor framework site: tooling for evaluating and optimising sandboxed agents, running datasets such as Terminal-Bench and SWE-bench Pro.
LiveCodeBench
LiveCodeBench leaderboard: code generation results on problems collected from recent programming contests.
MedQA
by Di Jin
MedQA GitHub repo: data and baselines for open-domain medical exam question answering.
Melting Pot
DeepMind Melting Pot GitHub repo: test scenarios for generalisation in multi-agent reinforcement learning.
MERA Code
MERA Code GitHub repo: a benchmark for evaluating code generation by LLMs on tasks described in Russian.
MERA Code / SWE-MERA
MERA Code and SWE-MERA: benchmarks of code generation and software-engineering agents for Russian-language tasks.
METR
by METR
METR: research nonprofit publishing evaluations of frontier AI models' autonomous capabilities and risks.
METR Time Horizons
by METR
METR time horizons page: measured length of tasks frontier models complete autonomously, tracked over time.