--- id: "/evaluation-benchmarks/agent-coding-evals/page/2/" title: "agent & coding evals: 85 curated resources · page 2" url: "https://skillissue.dev/evaluation-benchmarks/agent-coding-evals/page/2/" --- # agent & coding evals: 85 curated resources agent & coding evals [Beginner](/evaluation-benchmarks/beginner/index.md) [Intermediate](/evaluation-benchmarks/intermediate/index.md) [Advanced](/evaluation-benchmarks/advanced/index.md) [Expert](/evaluation-benchmarks/expert/index.md) 85 resources · page 2 of 3 - ### [MineRL](/evaluation/benchmarks-and-leaderboards/minerl/index.md) by [MineRL Labs](https://github.com/minerllabs) MineRL GitHub repo: Minecraft environments and human demonstrations for sample-efficient RL. entity repo US / International en - ### [mini-SWE-agent](/evaluation/benchmarks-and-leaderboards/mini-swe-agent/index.md) by [SWE-agent](https://mini-SWE-agent.com) mini-SWE-agent GitHub repo: a roughly 100-line reference agent for solving GitHub issues, used as a simple SWE-bench baseline. entity benchmark US en - ### [MLE-bench](/evaluation/benchmarks-and-leaderboards/mle-bench/index.md) by [Jun Shern Chan et al. (OpenAI)](https://openai.com/) MLE-bench paper from OpenAI: 75 Kaggle competitions used to evaluate AI agents on machine-learning engineering work. entity benchmark US (OpenAI) en - ### [OpenRouter rankings](/evaluation/benchmarks-and-leaderboards/openrouter-rankings/index.md) by [OpenRouter](https://openrouter.ai/) OpenRouter rankings: models ranked by tokens processed through the OpenRouter API, by app and category. entity leaderboard West en - ### [Polish LLM benchmarks (CodeSOTA)](/evaluation/benchmarks-and-leaderboards/polish-llm-benchmarks-codesota/index.md) by [CodeSOTA](https://www.codesota.com/) CodeSOTA page aggregating Polish LLM benchmark results for models such as Bielik, PLLuM, GPT, Claude and Gemini. entity directory Poland pl - ### [ProgramBench](/evaluation/benchmarks-and-leaderboards/programbench/index.md) by [ProgramBench](https://programbench.com/) ProgramBench site: benchmark testing whether language models can rebuild complete programs from scratch. entity benchmark US en - ### [Pydantic AI](/evaluation/benchmarks-and-leaderboards/pydantic-ai-repo-evals-durable-exec/index.md) by [Pydantic](https://pydantic.dev) Pydantic AI GitHub repo: type-safe Python agent framework with evals and durable execution. entity repo UK en - ### [RULER](/evaluation/benchmarks-and-leaderboards/ruler/index.md) by [NVIDIA Corporation](https://nvidia.com) NVIDIA RULER GitHub repo: synthetic benchmark measuring the real usable context length of long-context models. entity leaderboard West en - ### [SciCode](/evaluation/benchmarks-and-leaderboards/scicode/index.md) by [scicode-bench](https://github.com/scicode-bench) SciCode benchmark site: research-level scientific coding problems for evaluating language models. entity benchmark US en - ### [ScienceAgentBench](/evaluation/benchmarks-and-leaderboards/scienceagentbench/index.md) by [OSU Natural Language Processing](https://github.com/OSU-NLP-Group) ScienceAgentBench project page: data-driven scientific discovery tasks from published papers for evaluating language agents. entity benchmark US (OSU) en - ### [Spider 2.0](/evaluation/benchmarks-and-leaderboards/spider-2-0/index.md) by [spider2-sql](https://github.com/spider2-sql) Spider 2.0 site: 632 enterprise text-to-SQL workflow problems for evaluating language models and agents. entity benchmark US / International en - ### [SWE-bench](/evaluation/benchmarks-and-leaderboards/swe-bench/index.md) by [SWE-bench team (Princeton University and Stanford University)](https://www.swebench.com/) SWE-bench leaderboards: coding-agent results on resolving real GitHub issues, across SWE-bench Verified, Multilingual and Multimodal. entity leaderboard West en - ### [SWE-bench Pro](/evaluation/benchmarks-and-leaderboards/swe-bench-pro/index.md) by [Scale](https://scale.com) Scale AI SWE-bench Pro GitHub repo: benchmark of long-horizon software engineering tasks for agents. entity repo US en - ### [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](/evaluation/benchmarks-and-leaderboards/swe-bench-pro-paper/index.md) by [Xiang Deng et al. (Scale AI)](https://scale.com/) SWE-Bench Pro paper: long-horizon software-engineering tasks, including a held-out commercial split, for evaluating coding agents. entity benchmark US (Scale) en - ### [SWE-Lancer](/evaluation/benchmarks-and-leaderboards/swe-lancer/index.md) by [OpenAI](https://openai.com/) OpenAI SWE-Lancer GitHub repo: dataset and code for real freelance software engineering tasks. entity repo US en - ### [SWE-MERA](/evaluation/benchmarks-and-leaderboards/swe-mera/index.md) by [MERA Evaluation](https://mera.a-ai.ru/) SWE-MERA page: continuously refreshed software-engineering tasks built as a contamination-resistant alternative to SWE-bench. entity benchmark Russia en - ### [SWE-rebench](/evaluation/benchmarks-and-leaderboards/swe-rebench/index.md) by [Nebius (SWE-rebench)](https://nebius.com/) SWE-rebench: continuously updated, decontaminated software-engineering benchmark with leaderboard. entity leaderboard EU/Russia (Nebius) en - ### [τ-bench](/evaluation/benchmarks-and-leaderboards/tau-bench-family-tau-tau2-tau3-bench/index.md) by [Sierra (τ-bench)](https://sierra.ai/) τ-bench site: benchmarks of AI agents conversing with users, calling tools and following policies across domains. entity leaderboard US (Sierra) en - ### [tau2-bench](/evaluation/benchmarks-and-leaderboards/tau2-bench/index.md) by [Sierra Research](https://sierra.ai) tau2-bench GitHub repo (Sierra): benchmark of tool-using agents interacting with users under domain policies. entity leaderboard West en - ### [Terminal-Bench](/evaluation/benchmarks-and-leaderboards/terminal-bench/index.md) by [Terminal-Bench team (Stanford University and Laude Institute)](https://www.tbench.ai/) Terminal-Bench: benchmark of AI agents completing tasks in sandboxed terminal environments. entity leaderboard West en - ### [Terminal-Bench 2.0 (Snorkel)](/evaluation/benchmarks-and-leaderboards/terminal-bench-2-0-snorkel/index.md) by [Snorkel AI](https://snorkel.ai/) Snorkel-hosted Terminal-Bench 2.0 leaderboard: 89 human-checked terminal tasks scored by resolution rate. entity leaderboard US en - ### [The Agent Company](/evaluation/benchmarks-and-leaderboards/the-agent-company/index.md) by [TheAgentCompany team (Carnegie Mellon University)](https://the-agent-company.com/) TheAgentCompany site: benchmark of simulated software-company tasks for evaluating LLM agents across business tools. entity benchmark US / International en - ### [ToolBench](/evaluation/benchmarks-and-leaderboards/toolbench/index.md) by [OpenBMB](https://www.openbmb.cn) OpenBMB ToolBench GitHub repo (ICLR 2024): platform and benchmark for training and evaluating LLM tool use. entity repo China / International en - ### [Top 3D AI Arena](/evaluation/benchmarks-and-leaderboards/top-3d-ai-arena/index.md) by [Top 3D AI](https://top3d.ai/) Top 3D AI leaderboard ranking 3D generation tools by Elo from blind community votes. entity benchmark International en - ### [VisualWebArena](/evaluation/benchmarks-and-leaderboards/visualwebarena/index.md) by [Jing Yu Koh (Carnegie Mellon University)](https://jykoh.com/) VisualWebArena project page: benchmark of realistic web tasks where multimodal agents need visual information to succeed. entity benchmark US en - ### [VQA v2](/evaluation/benchmarks-and-leaderboards/vqa-v2/index.md) by [VQA team (Virginia Tech and Georgia Tech)](https://visualqa.org/) VQA site: Visual Question Answering datasets and challenges for answering open questions about images. entity benchmark US / International en - ### [WebArena](/evaluation/benchmarks-and-leaderboards/webarena/index.md) by [WebArena team (Carnegie Mellon University)](https://webarena.dev/) WebArena: suite of self-hosted website environments for benchmarking autonomous web agents. entity leaderboard West en - ### [Windows Agent Arena](/evaluation/benchmarks-and-leaderboards/windows-agent-arena/index.md) by [Microsoft](https://opensource.microsoft.com) Windows Agent Arena GitHub repo from Microsoft: scalable Windows OS environment for testing multimodal computer-use agents. entity benchmark US (Microsoft) en - ### [WorkArena](/evaluation/benchmarks-and-leaderboards/workarena/index.md) by [ServiceNow](https://www.servicenow.com) ServiceNow WorkArena GitHub repo: benchmark of web agents on common enterprise knowledge-work tasks. entity repo Canada / US en - ### [What matters when evaluating AI agents for customer service (Intercom)](/evaluation/benchmarks-and-leaderboards/intercom-what-really-matters-when-evaluating-ai-agents-for-cs/index.md) by [Intercom](https://www.intercom.com/) Intercom post on criteria beyond performance for evaluating AI agents for customer service. Beginner learning_resource guide IE/US en - ### [Latent Space: SWE-bench dead](/evaluation/benchmarks-and-leaderboards/latent-space-swe-bench-dead/index.md) by [Latent Space](https://www.latent.space/) Latent Space episode with OpenAI's Mia Glaese and Olivia Watkins on retiring SWE-bench Verified and next frontier evals. Beginner learning_resource podcast-video US en - ### [AI Agents That Matter](/evaluation/benchmarks-and-leaderboards/ai-agents-that-matter/index.md) by [Sayash Kapoor et al. (Princeton University)](https://www.princeton.edu/) arXiv paper (2024) "AI Agents That Matter", on cost-aware and reproducible evaluation of AI agents. Intermediate learning_resource paper US (Princeton) en [Previous page](/evaluation-benchmarks/agent-coding-evals/index.md) [Next page](/evaluation-benchmarks/agent-coding-evals/page/3/index.md) [Plain-text Markdown copy](/evaluation-benchmarks/agent-coding-evals/page/2/index.md)