agent & coding evals: 85 curated resources
agent & coding evals
85 resources · page 2 of 3
MineRL
by MineRL Labs
MineRL GitHub repo: Minecraft environments and human demonstrations for sample-efficient RL.
mini-SWE-agent
by SWE-agent
mini-SWE-agent GitHub repo: a roughly 100-line reference agent for solving GitHub issues, used as a simple SWE-bench baseline.
MLE-bench
by Jun Shern Chan et al. (OpenAI)
MLE-bench paper from OpenAI: 75 Kaggle competitions used to evaluate AI agents on machine-learning engineering work.
OpenRouter rankings
by OpenRouter
OpenRouter rankings: models ranked by tokens processed through the OpenRouter API, by app and category.
Polish LLM benchmarks (CodeSOTA)
by CodeSOTA
CodeSOTA page aggregating Polish LLM benchmark results for models such as Bielik, PLLuM, GPT, Claude and Gemini.
ProgramBench
by ProgramBench
ProgramBench site: benchmark testing whether language models can rebuild complete programs from scratch.
Pydantic AI
by Pydantic
Pydantic AI GitHub repo: type-safe Python agent framework with evals and durable execution.
RULER
NVIDIA RULER GitHub repo: synthetic benchmark measuring the real usable context length of long-context models.
SciCode
SciCode benchmark site: research-level scientific coding problems for evaluating language models.
ScienceAgentBench
by OSU Natural Language Processing
ScienceAgentBench project page: data-driven scientific discovery tasks from published papers for evaluating language agents.
Spider 2.0
by spider2-sql
Spider 2.0 site: 632 enterprise text-to-SQL workflow problems for evaluating language models and agents.
SWE-bench
by SWE-bench team (Princeton University and Stanford University)
SWE-bench leaderboards: coding-agent results on resolving real GitHub issues, across SWE-bench Verified, Multilingual and Multimodal.
SWE-bench Pro
by Scale
Scale AI SWE-bench Pro GitHub repo: benchmark of long-horizon software engineering tasks for agents.
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
by Xiang Deng et al. (Scale AI)
SWE-Bench Pro paper: long-horizon software-engineering tasks, including a held-out commercial split, for evaluating coding agents.
SWE-Lancer
by OpenAI
OpenAI SWE-Lancer GitHub repo: dataset and code for real freelance software engineering tasks.
SWE-MERA
SWE-MERA page: continuously refreshed software-engineering tasks built as a contamination-resistant alternative to SWE-bench.
SWE-rebench
SWE-rebench: continuously updated, decontaminated software-engineering benchmark with leaderboard.
τ-bench
τ-bench site: benchmarks of AI agents conversing with users, calling tools and following policies across domains.
tau2-bench
tau2-bench GitHub repo (Sierra): benchmark of tool-using agents interacting with users under domain policies.
Terminal-Bench
by Terminal-Bench team (Stanford University and Laude Institute)
Terminal-Bench: benchmark of AI agents completing tasks in sandboxed terminal environments.
Terminal-Bench 2.0 (Snorkel)
by Snorkel AI
Snorkel-hosted Terminal-Bench 2.0 leaderboard: 89 human-checked terminal tasks scored by resolution rate.
The Agent Company
by TheAgentCompany team (Carnegie Mellon University)
TheAgentCompany site: benchmark of simulated software-company tasks for evaluating LLM agents across business tools.
ToolBench
by OpenBMB
OpenBMB ToolBench GitHub repo (ICLR 2024): platform and benchmark for training and evaluating LLM tool use.
Top 3D AI Arena
by Top 3D AI
Top 3D AI leaderboard ranking 3D generation tools by Elo from blind community votes.
VisualWebArena
by Jing Yu Koh (Carnegie Mellon University)
VisualWebArena project page: benchmark of realistic web tasks where multimodal agents need visual information to succeed.
VQA v2
by VQA team (Virginia Tech and Georgia Tech)
VQA site: Visual Question Answering datasets and challenges for answering open questions about images.
WebArena
by WebArena team (Carnegie Mellon University)
WebArena: suite of self-hosted website environments for benchmarking autonomous web agents.
Windows Agent Arena
by Microsoft
Windows Agent Arena GitHub repo from Microsoft: scalable Windows OS environment for testing multimodal computer-use agents.
WorkArena
by ServiceNow
ServiceNow WorkArena GitHub repo: benchmark of web agents on common enterprise knowledge-work tasks.
What matters when evaluating AI agents for customer service (Intercom)
by Intercom
Intercom post on criteria beyond performance for evaluating AI agents for customer service.
Latent Space: SWE-bench dead
by Latent Space
Latent Space episode with OpenAI's Mia Glaese and Olivia Watkins on retiring SWE-bench Verified and next frontier evals.
AI Agents That Matter
by Sayash Kapoor et al. (Princeton University)
arXiv paper (2024) "AI Agents That Matter", on cost-aware and reproducible evaluation of AI agents.