Agent Coding Evals

85 resources · page 1 of 3

• AA-3D

  Creator not supplied; see original source.

  Artificial Analysis AA-3D benchmark comparing how well AI models build interactive 3D experiences, judged by blind panel comparisons.

  Not applicable (entity) · unknown · unknown resource · en

• Artificial Analysis Intelligence Benchmarking methodology

  Creator not supplied; see original source.

  Artificial Analysis methodology page describing the benchmarks, weights and scoring behind its Intelligence Index for LLMs.

  Not applicable (entity) · unknown · unknown resource · en

• AgentBench

  Creator not supplied; see original source.

  AgentBench GitHub repo (Tsinghua): benchmark evaluating LLMs as agents across multiple interactive environments.

  Not applicable (entity) · unknown · unknown resource · en

• AgentBoard

  Creator not supplied; see original source.

  HKUST AgentBoard GitHub repo: analytical evaluation board for multi-turn LLM agents with progress tracking.

  Not applicable (entity) · unknown · unknown resource · en

• ethz-spylab/agentdojo

  Creator not supplied; see original source.

  ETH SPY Lab AgentDojo GitHub repo: environment evaluating prompt injection attacks and defences for LLM agents.

  Not applicable (entity) · unknown · unknown resource · en

• ai-safety-institute/AgentHarm

  Creator not supplied; see original source.

  UK AISI AgentHarm dataset on Hugging Face: harmful agentic tasks for measuring whether LLM agents comply with misuse requests.

  Not applicable (entity) · unknown · unknown resource · en

• AI Verify Foundation / Project Moonshot (Singapore)

  Creator not supplied; see original source.

  Singapore's AI Verify Foundation site, home of the AI Verify testing toolkit and Project Moonshot for LLM red-teaming and benchmarking.

  Not applicable (entity) · unknown · unknown resource · en

• Aider polyglot leaderboard

  Creator not supplied; see original source.

  Aider LLM leaderboards: model scores on polyglot code-editing exercises with cost.

  Not applicable (entity) · unknown · unknown resource · en

• AndroidWorld

  Creator not supplied; see original source.

  Google Research AndroidWorld GitHub repo: environment and benchmark for autonomous agents on Android.

  Not applicable (entity) · unknown · unknown resource · en

• AppWorld

  Creator not supplied; see original source.

  AppWorld benchmark: a simulated world of everyday apps and APIs for testing coding agents on stateful multi-app tasks.

  Not applicable (entity) · unknown · unknown resource · en

• Arena Raises $200M Series B at $3.1B Valuation

  Creator not supplied; see original source.

  Arena blog post on its Series B and the launch of its AI Alignment Index, measuring unsafe agent behaviour on real traces.

  Not applicable (entity) · unknown · unknown resource · en

• augment-swebench-agent

  Creator not supplied; see original source.

  Augment augment-swebench-agent GitHub repo: open implementation of Augment's SWE-bench Verified agent.

  Not applicable (entity) · unknown · unknown resource · en

• sierra-research/tau-bench

  Creator not supplied; see original source.

  Sierra τ-bench GitHub repo: code and data for evaluating tool-using agents in customer service domains.

  Not applicable (entity) · unknown · unknown resource · en

• Berkeley Function-Calling Leaderboard

  Creator not supplied; see original source.

  Berkeley Function Calling Leaderboard (BFCL V4): model accuracy on function and tool calling, including agentic tasks.

  Not applicable (entity) · unknown · unknown resource · en

• BigCodeBench

  Creator not supplied; see original source.

  BigCodeBench leaderboard: practical programming tasks that require calling many libraries, for evaluating code LLMs.

  Not applicable (entity) · unknown · unknown resource · en

• BioASQ

  Creator not supplied; see original source.

  BioASQ site: recurring challenges on biomedical semantic indexing and question answering, with datasets and results.

  Not applicable (entity) · unknown · unknown resource · en

• BIRD

  Creator not supplied; see original source.

  BIRD benchmark site: large-scale text-to-SQL evaluation over real databases with dirty values and a public leaderboard.

  Not applicable (entity) · unknown · unknown resource · en

• BrowserGym

  Creator not supplied; see original source.

  ServiceNow BrowserGym GitHub repo: Gym environment for evaluating web task automation agents.

  Not applicable (entity) · unknown · unknown resource · en

• CodeClash

  Creator not supplied; see original source.

  CodeClash benchmark: AI coding agents compete in multi-round tournaments to pursue open-ended goals in codebases.

  Not applicable (entity) · unknown · unknown resource · en

• CyberGym

  Creator not supplied; see original source.

  Berkeley CyberGym GitHub repo: evaluation of AI agents on real software vulnerabilities.

  Not applicable (entity) · unknown · unknown resource · en

• EvalPlus

  Creator not supplied; see original source.

  EvalPlus site: HumanEval+ and MBPP+ with extended test cases for more accurate evaluation of code LLMs.

  Not applicable (entity) · unknown · unknown resource · en

• GAIA

  Creator not supplied; see original source.

  GAIA organisation page on Hugging Face: benchmark of real-world questions for general AI assistants needing reasoning, browsing and tools.

  Not applicable (entity) · unknown · unknown resource · en

• GAIA leaderboard

  Creator not supplied; see original source.

  GAIA leaderboard Space: scores of AI assistants and agents on the GAIA real-world question benchmark.

  Not applicable (entity) · unknown · unknown resource · en

• HAL: Holistic Agent Leaderboard

  Creator not supplied; see original source.

  Princeton HAL Holistic Agent Leaderboard: third-party, cost-aware evaluation of AI agents across benchmarks.

  Not applicable (entity) · unknown · unknown resource · en

• Harbor framework

  Creator not supplied; see original source.

  Harbor framework site: tooling for evaluating and optimising sandboxed agents, running datasets such as Terminal-Bench and SWE-bench Pro.

  Not applicable (entity) · unknown · unknown resource · en

• LiveCodeBench

  Creator not supplied; see original source.

  LiveCodeBench leaderboard: code generation results on problems collected from recent programming contests.

  Not applicable (entity) · unknown · unknown resource · en

• MedQA

  Creator not supplied; see original source.

  MedQA GitHub repo: data and baselines for open-domain medical exam question answering.

  Not applicable (entity) · unknown · unknown resource · en

• Melting Pot

  Creator not supplied; see original source.

  DeepMind Melting Pot GitHub repo: test scenarios for generalisation in multi-agent reinforcement learning.

  Not applicable (entity) · unknown · unknown resource · en

• MERA Code

  Creator not supplied; see original source.

  MERA Code GitHub repo: a benchmark for evaluating code generation by LLMs on tasks described in Russian.

  Not applicable (entity) · unknown · unknown resource · ru

• MERA Code / SWE-MERA

  Creator not supplied; see original source.

  MERA Code and SWE-MERA: benchmarks of code generation and software-engineering agents for Russian-language tasks.

  Not applicable (entity) · unknown · unknown resource · ru

• METR

  Creator not supplied; see original source.

  METR: research nonprofit publishing evaluations of frontier AI models' autonomous capabilities and risks.

  Not applicable (entity) · unknown · unknown resource · en

• METR Time Horizons

  Creator not supplied; see original source.

  METR time horizons page: measured length of tasks frontier models complete autonomously, tracked over time.

  Not applicable (entity) · unknown · unknown resource · en

Plain-text Markdown copy