Agent Coding Evals

85 resources · page 2 of 3

• MineRL

  Creator not supplied; see original source.

  MineRL GitHub repo: Minecraft environments and human demonstrations for sample-efficient RL.

  Not applicable (entity) · unknown · unknown resource · en

• mini-SWE-agent

  Creator not supplied; see original source.

  mini-SWE-agent GitHub repo: a roughly 100-line reference agent for solving GitHub issues, used as a simple SWE-bench baseline.

  Not applicable (entity) · unknown · unknown resource · en

• MLE-bench

  Creator not supplied; see original source.

  MLE-bench paper from OpenAI: 75 Kaggle competitions used to evaluate AI agents on machine-learning engineering work.

  Not applicable (entity) · unknown · unknown resource · en

• OpenRouter rankings

  Creator not supplied; see original source.

  OpenRouter rankings: models ranked by tokens processed through the OpenRouter API, by app and category.

  Not applicable (entity) · unknown · unknown resource · en

• Polish LLM benchmarks (CodeSOTA)

  Creator not supplied; see original source.

  CodeSOTA page aggregating Polish LLM benchmark results for models such as Bielik, PLLuM, GPT, Claude and Gemini.

  Not applicable (entity) · unknown · unknown resource · pl

• ProgramBench

  Creator not supplied; see original source.

  ProgramBench site: benchmark testing whether language models can rebuild complete programs from scratch.

  Not applicable (entity) · unknown · unknown resource · en

• Pydantic AI

  Creator not supplied; see original source.

  Pydantic AI GitHub repo: type-safe Python agent framework with evals and durable execution.

  Not applicable (entity) · unknown · unknown resource · en

• RULER

  Creator not supplied; see original source.

  NVIDIA RULER GitHub repo: synthetic benchmark measuring the real usable context length of long-context models.

  Not applicable (entity) · unknown · unknown resource · en

• SciCode

  Creator not supplied; see original source.

  SciCode benchmark site: research-level scientific coding problems for evaluating language models.

  Not applicable (entity) · unknown · unknown resource · en

• ScienceAgentBench

  Creator not supplied; see original source.

  ScienceAgentBench project page: data-driven scientific discovery tasks from published papers for evaluating language agents.

  Not applicable (entity) · unknown · unknown resource · en

• Spider 2.0

  Creator not supplied; see original source.

  Spider 2.0 site: 632 enterprise text-to-SQL workflow problems for evaluating language models and agents.

  Not applicable (entity) · unknown · unknown resource · en

• SWE-bench

  Creator not supplied; see original source.

  SWE-bench leaderboards: coding-agent results on resolving real GitHub issues, across SWE-bench Verified, Multilingual and Multimodal.

  Not applicable (entity) · unknown · unknown resource · en

• SWE-bench Pro

  Creator not supplied; see original source.

  Scale AI SWE-bench Pro GitHub repo: benchmark of long-horizon software engineering tasks for agents.

  Not applicable (entity) · unknown · unknown resource · en

• SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

  Creator not supplied; see original source.

  SWE-Bench Pro paper: long-horizon software-engineering tasks, including a held-out commercial split, for evaluating coding agents.

  Not applicable (entity) · unknown · unknown resource · en

• SWE-Lancer

  Creator not supplied; see original source.

  OpenAI SWE-Lancer GitHub repo: dataset and code for real freelance software engineering tasks.

  Not applicable (entity) · unknown · unknown resource · en

• SWE-MERA

  Creator not supplied; see original source.

  SWE-MERA page: continuously refreshed software-engineering tasks built as a contamination-resistant alternative to SWE-bench.

  Not applicable (entity) · unknown · unknown resource · en

• SWE-rebench

  Creator not supplied; see original source.

  SWE-rebench: continuously updated, decontaminated software-engineering benchmark with leaderboard.

  Not applicable (entity) · unknown · unknown resource · en

• τ-bench

  Creator not supplied; see original source.

  τ-bench site: benchmarks of AI agents conversing with users, calling tools and following policies across domains.

  Not applicable (entity) · unknown · unknown resource · en

• tau2-bench

  Creator not supplied; see original source.

  tau2-bench GitHub repo (Sierra): benchmark of tool-using agents interacting with users under domain policies.

  Not applicable (entity) · unknown · unknown resource · en

• Terminal-Bench

  Creator not supplied; see original source.

  Terminal-Bench: benchmark of AI agents completing tasks in sandboxed terminal environments.

  Not applicable (entity) · unknown · unknown resource · en

• Terminal-Bench 2.0 (Snorkel)

  Creator not supplied; see original source.

  Snorkel-hosted Terminal-Bench 2.0 leaderboard: 89 human-checked terminal tasks scored by resolution rate.

  Not applicable (entity) · unknown · unknown resource · en

• The Agent Company

  Creator not supplied; see original source.

  TheAgentCompany site: benchmark of simulated software-company tasks for evaluating LLM agents across business tools.

  Not applicable (entity) · unknown · unknown resource · en

• ToolBench

  Creator not supplied; see original source.

  OpenBMB ToolBench GitHub repo (ICLR 2024): platform and benchmark for training and evaluating LLM tool use.

  Not applicable (entity) · unknown · unknown resource · en

• Top 3D AI Arena

  Creator not supplied; see original source.

  Top 3D AI leaderboard ranking 3D generation tools by Elo from blind community votes.

  Not applicable (entity) · unknown · unknown resource · en

• VisualWebArena

  Creator not supplied; see original source.

  VisualWebArena project page: benchmark of realistic web tasks where multimodal agents need visual information to succeed.

  Not applicable (entity) · unknown · unknown resource · en

• VQA v2

  Creator not supplied; see original source.

  VQA site: Visual Question Answering datasets and challenges for answering open questions about images.

  Not applicable (entity) · unknown · unknown resource · en

• WebArena

  Creator not supplied; see original source.

  WebArena: suite of self-hosted website environments for benchmarking autonomous web agents.

  Not applicable (entity) · unknown · unknown resource · en

• Windows Agent Arena

  Creator not supplied; see original source.

  Windows Agent Arena GitHub repo from Microsoft: scalable Windows OS environment for testing multimodal computer-use agents.

  Not applicable (entity) · unknown · unknown resource · en

• WorkArena

  Creator not supplied; see original source.

  ServiceNow WorkArena GitHub repo: benchmark of web agents on common enterprise knowledge-work tasks.

  Not applicable (entity) · unknown · unknown resource · en

• What matters when evaluating AI agents for customer service (Intercom)

  Creator not supplied; see original source.

  Intercom post on criteria beyond performance for evaluating AI agents for customer service.

  Beginner · unknown · unknown resource · en

• Latent Space: SWE-bench dead

  Creator not supplied; see original source.

  Latent Space episode with OpenAI's Mia Glaese and Olivia Watkins on retiring SWE-bench Verified and next frontier evals.

  Beginner · unknown · unknown resource · en

• AI Agents That Matter

  Creator not supplied; see original source.

  arXiv paper (2024) "AI Agents That Matter", on cost-aware and reproducible evaluation of AI agents.

  Intermediate · unknown · unknown resource · en

Plain-text Markdown copy