--- id: "/evaluation-benchmarks/agent-coding-evals/" title: "agent & coding evals: 85 curated resources" url: "https://skillissue.dev/evaluation-benchmarks/agent-coding-evals/" --- # agent & coding evals: 85 curated resources agent & coding evals [Beginner](/evaluation-benchmarks/beginner/index.md) [Intermediate](/evaluation-benchmarks/intermediate/index.md) [Advanced](/evaluation-benchmarks/advanced/index.md) [Expert](/evaluation-benchmarks/expert/index.md) 85 resources · page 1 of 3 - ### [AA-3D](/evaluation/benchmarks-and-leaderboards/aa-3d/index.md) by [Artificial Analysis](https://artificialanalysis.ai/) Artificial Analysis AA-3D benchmark comparing how well AI models build interactive 3D experiences, judged by blind panel comparisons. entity benchmark Australia / Global en - ### [Artificial Analysis Intelligence Benchmarking methodology](/evaluation/benchmarks-and-leaderboards/aa-agentic-evals-gdpval-aa-aa-briefcase-automationbench-aa-gdp-pdf/index.md) by [Artificial Analysis](https://artificialanalysis.ai/) Artificial Analysis methodology page describing the benchmarks, weights and scoring behind its Intelligence Index for LLMs. entity benchmark US/UK (AA) en - ### [AgentBench](/evaluation/benchmarks-and-leaderboards/agentbench/index.md) by [THUKEG](https://huggingface.co/THUDM) AgentBench GitHub repo (Tsinghua): benchmark evaluating LLMs as agents across multiple interactive environments. entity leaderboard CN en - ### [AgentBoard](/evaluation/benchmarks-and-leaderboards/agentboard/index.md) by [NLP Group @ HKUST](https://jxhe.github.io/group/) HKUST AgentBoard GitHub repo: analytical evaluation board for multi-turn LLM agents with progress tracking. entity repo Hong Kong / China en - ### [ethz-spylab/agentdojo](/evaluation/benchmarks-and-leaderboards/agentdojo/index.md) by [SPY Lab](https://spylab.ai) ETH SPY Lab AgentDojo GitHub repo: environment evaluating prompt injection attacks and defences for LLM agents. entity repo Switzerland en - ### [ai-safety-institute/AgentHarm](/evaluation/benchmarks-and-leaderboards/agentharm/index.md) by [UK AI Security Institute](https://huggingface.co/ai-safety-institute) UK AISI AgentHarm dataset on Hugging Face: harmful agentic tasks for measuring whether LLM agents comply with misuse requests. entity benchmark UK [STATE] (UK AISI + Gray Swan) en - ### [AI Verify Foundation / Project Moonshot (Singapore)](/evaluation/benchmarks-and-leaderboards/ai-verify-foundation-project-moonshot-singapore/index.md) by [AI Verify Foundation](https://aiverifyfoundation.sg/) Singapore's AI Verify Foundation site, home of the AI Verify testing toolkit and Project Moonshot for LLM red-teaming and benchmarking. entity benchmark Singapore en - ### [Aider polyglot leaderboard](/evaluation/benchmarks-and-leaderboards/aider-polyglot-leaderboard/index.md) by [Aider](https://aider.chat/) Aider LLM leaderboards: model scores on polyglot code-editing exercises with cost. entity leaderboard West en - ### [AndroidWorld](/evaluation/benchmarks-and-leaderboards/androidworld/index.md) by [Google Research](https://research.google) Google Research AndroidWorld GitHub repo: environment and benchmark for autonomous agents on Android. entity repo US en - ### [AppWorld](/evaluation/benchmarks-and-leaderboards/appworld/index.md) by [AppWorld team (Stony Brook University)](https://appworld.dev/) AppWorld benchmark: a simulated world of everyday apps and APIs for testing coding agents on stateful multi-app tasks. entity benchmark US en - ### [Arena Raises $200M Series B at $3.1B Valuation](/evaluation/benchmarks-and-leaderboards/arena-alignment-index-series-b-post/index.md) by [Arena (formerly LMArena)](https://arena.ai/) Arena blog post on its Series B and the launch of its AI Alignment Index, measuring unsafe agent behaviour on real traces. entity leaderboard US (commercial) en - ### [augment-swebench-agent](/evaluation/benchmarks-and-leaderboards/augment-swe-bench-agent/index.md) by [Augment Code](https://augmentcode.com/) Augment augment-swebench-agent GitHub repo: open implementation of Augment's SWE-bench Verified agent. entity repo US en - ### [sierra-research/tau-bench](/evaluation/benchmarks-and-leaderboards/bench/index.md) by [Sierra Research](https://sierra.ai) Sierra τ-bench GitHub repo: code and data for evaluating tool-using agents in customer service domains. entity repo US en - ### [Berkeley Function-Calling Leaderboard](/evaluation/benchmarks-and-leaderboards/berkeley-function-calling-leaderboard/index.md) by [Gorilla team (UC Berkeley)](https://gorilla.cs.berkeley.edu/) Berkeley Function Calling Leaderboard (BFCL V4): model accuracy on function and tool calling, including agentic tasks. entity leaderboard West en - ### [BigCodeBench](/evaluation/benchmarks-and-leaderboards/bigcodebench/index.md) by [bigcode-bench](https://github.com/bigcode-bench) BigCodeBench leaderboard: practical programming tasks that require calling many libraries, for evaluating code LLMs. entity benchmark International en - ### [BioASQ](/evaluation/benchmarks-and-leaderboards/bioasq/index.md) by [BioASQ](https://bioasq.org/) BioASQ site: recurring challenges on biomedical semantic indexing and question answering, with datasets and results. entity benchmark EU / International en - ### [BIRD](/evaluation/benchmarks-and-leaderboards/bird/index.md) by [bird_sql](https://bird-bench.github.io/) BIRD benchmark site: large-scale text-to-SQL evaluation over real databases with dirty values and a public leaderboard. entity benchmark China / International en - ### [BrowserGym](/evaluation/benchmarks-and-leaderboards/browsergym/index.md) by [ServiceNow](https://www.servicenow.com) ServiceNow BrowserGym GitHub repo: Gym environment for evaluating web task automation agents. entity repo Canada / US en - ### [CodeClash](/evaluation/benchmarks-and-leaderboards/codeclash/index.md) by [CodeClash team](https://codeclash.ai/) CodeClash benchmark: AI coding agents compete in multi-round tournaments to pursue open-ended goals in codebases. entity benchmark US en - ### [CyberGym](/evaluation/benchmarks-and-leaderboards/cybergym/index.md) by [sunblaze-ucb](https://github.com/sunblaze-ucb) Berkeley CyberGym GitHub repo: evaluation of AI agents on real software vulnerabilities. entity repo US en - ### [EvalPlus](/evaluation/benchmarks-and-leaderboards/evalplus/index.md) by [evalplus](https://github.com/evalplus) EvalPlus site: HumanEval+ and MBPP+ with extended test cases for more accurate evaluation of code LLMs. entity benchmark US / International en - ### [GAIA](/evaluation/benchmarks-and-leaderboards/gaia/index.md) by [GAIA](https://huggingface.co/gaia-benchmark) GAIA organisation page on Hugging Face: benchmark of real-world questions for general AI assistants needing reasoning, browsing and tools. entity benchmark France / International en - ### [GAIA leaderboard](/evaluation/benchmarks-and-leaderboards/gaia-leaderboard/index.md) by [GAIA](https://huggingface.co/gaia-benchmark) GAIA leaderboard Space: scores of AI assistants and agents on the GAIA real-world question benchmark. entity leaderboard West en - ### [HAL: Holistic Agent Leaderboard](/evaluation/benchmarks-and-leaderboards/hal-holistic-agent-leaderboard/index.md) by [Princeton University (HAL)](https://hal.cs.princeton.edu/) Princeton HAL Holistic Agent Leaderboard: third-party, cost-aware evaluation of AI agents across benchmarks. entity leaderboard US (Princeton) en - ### [Harbor framework](/evaluation/benchmarks-and-leaderboards/harbor-framework/index.md) by [Harbor framework project (Laude Institute)](https://harborframework.com/) Harbor framework site: tooling for evaluating and optimising sandboxed agents, running datasets such as Terminal-Bench and SWE-bench Pro. entity benchmark US en - ### [LiveCodeBench](/evaluation/benchmarks-and-leaderboards/livecodebench/index.md) by [LiveCodeBench](https://github.com/LiveCodeBench) LiveCodeBench leaderboard: code generation results on problems collected from recent programming contests. entity leaderboard West en - ### [MedQA](/evaluation/benchmarks-and-leaderboards/medqa/index.md) by [Di Jin](https://github.com/jind11) MedQA GitHub repo: data and baselines for open-domain medical exam question answering. entity repo US / China en - ### [Melting Pot](/evaluation/benchmarks-and-leaderboards/melting-pot/index.md) by [Google DeepMind](https://www.deepmind.com/) DeepMind Melting Pot GitHub repo: test scenarios for generalisation in multi-agent reinforcement learning. entity repo UK / US en - ### [MERA Code](/evaluation/benchmarks-and-leaderboards/mera-code/index.md) by [MERA Evaluation](https://mera.a-ai.ru/) MERA Code GitHub repo: a benchmark for evaluating code generation by LLMs on tasks described in Russian. entity benchmark Russia ru - ### [MERA Code / SWE-MERA](/evaluation/benchmarks-and-leaderboards/mera-code-swe-mera/index.md) by [AI Alliance Russia](https://a-ai.ru/) MERA Code and SWE-MERA: benchmarks of code generation and software-engineering agents for Russian-language tasks. entity leaderboard Russia ru - ### [METR](/evaluation/benchmarks-and-leaderboards/metr/index.md) by [METR](https://metr.org/) METR: research nonprofit publishing evaluations of frontier AI models' autonomous capabilities and risks. entity leaderboard West en - ### [METR Time Horizons](/evaluation/benchmarks-and-leaderboards/metr-time-horizons/index.md) by [METR](https://metr.org/) METR time horizons page: measured length of tasks frontier models complete autonomously, tracked over time. entity leaderboard US en [Next page](/evaluation-benchmarks/agent-coding-evals/page/2/index.md) [Plain-text Markdown copy](/evaluation-benchmarks/agent-coding-evals/index.md)