Benchmarks And Leaderboards: 512 curated resources
Curated material on benchmarks and leaderboards within evaluation.
512 resources · page 14 of 16
Latent Space: SWE-bench dead
Creator not supplied; see original source.
Latent Space episode with OpenAI's Mia Glaese and Olivia Watkins on retiring SWE-bench Verified and next frontier evals.
MMLU (classic)
Creator not supplied; see original source.
arXiv paper (2020) "Measuring Massive Multitask Language Understanding", introducing MMLU, a 57-subject multitask knowledge benchmark.
BEHAVIOR Challenge Charts the Way Forward for Domestic Robotics (Stanford HAI)
Creator not supplied; see original source.
Stanford HAI article on the BEHAVIOR Challenge, a household-robotics competition in simulation held at NeurIPS 2025.
AI Agents That Matter
Creator not supplied; see original source.
arXiv paper (2024) "AI Agents That Matter", on cost-aware and reproducible evaluation of AI agents.
March 2026: Arena Updates
Creator not supplied; see original source.
Arena blog post on March 2026 updates: new document and video-edit arenas and leaderboard changes.
Arize Phoenix docs: LLM-as-a-judge
Creator not supplied; see original source.
Arize Phoenix documentation on LLM-as-a-judge evaluation: designing judge prompts and criteria.
Berget AI: AI in Swedish
Creator not supplied; see original source.
Berget AI post on choosing models for Swedish-language applications, pointing to EuroEval and Scandinavian embedding benchmarks.
Braintrust documentation
Creator not supplied; see original source.
Braintrust documentation for running experiments, scoring outputs and collecting production feedback on AI apps.
Braintrust docs: Evaluate systematically
Creator not supplied; see original source.
Braintrust documentation on systematic evaluation: datasets, scorers and regression detection for AI applications.
Can We Trust AI Benchmarks?
Creator not supplied; see original source.
arXiv paper (2025) reviewing about 100 studies on the trustworthiness and limitations of AI benchmarks.
Chatbot Arena paper
Creator not supplied; see original source.
arXiv paper (2024) describing Chatbot Arena, its crowdsourced pairwise voting and Bradley-Terry ranking method.
How Some of China's Top AI Thinkers Built Their Own AI Safety Institute (Carnegie)
Creator not supplied; see original source.
Carnegie Endowment analysis of the China AI Safety and Development Association and its role in Chinese AI governance.
Demystifying evals for AI agents
Creator not supplied; see original source.
Anthropic post on designing evals for AI agents, combining automated tests, graders and human review.
SimpleQA Verified: Benchmark review (Epoch AI)
Creator not supplied; see original source.
Epoch AI review of the SimpleQA Verified benchmark, checking a sample of questions for correctness.
Evaluating the Effectiveness of LLM-Evaluators (Eugene Yan)
Creator not supplied; see original source.
Eugene Yan's review of LLM-evaluator (LLM-as-judge) research: use cases, techniques, alignment and critiques.
GAIA paper
Creator not supplied; see original source.
arXiv paper (2023) introducing GAIA, a benchmark of real-world questions for general AI assistants.
Announcing Agent Control (Galileo)
Creator not supplied; see original source.
Galileo post announcing Agent Control, an open-source control plane for runtime guardrails on AI agents.
Launching Agent Leaderboard v2 (Galileo)
Creator not supplied; see original source.
Galileo post introducing Agent Leaderboard v2, which evaluates LLMs as enterprise customer-support agents.
Best Practices to Navigate the Complexities of Evaluating AI Agents (Galileo)
Creator not supplied; see original source.
Galileo guide to evaluating AI agents, covering accuracy metrics, action evaluation and production monitoring.
SberDevices blog on Habr
Creator not supplied; see original source.
SberDevices company blog on Habr with Russian-language technical posts on GigaChat models and evaluation.
Using LLM-as-a-Judge For Evaluation: A Complete Guide (Hamel Husain)
Creator not supplied; see original source.
Hamel Husain's guide to building an LLM judge and validating it against domain-expert labels.
HF blog: Hebrew leaderboard
Creator not supplied; see original source.
Hugging Face blog source file announcing the Hebrew LLM leaderboard and its evaluation tasks.
AI Evals: Everything You Need to Know (Hamel Husain)
Creator not supplied; see original source.
Hamel Husain and Shreya Shankar's AI evals FAQ for engineers and PMs: testing AI systems, error analysis, RAG and agent evals.
Announcing Evals and Releases for Fin (Intercom)
Creator not supplied; see original source.
Intercom post announcing Evals and Releases for its Fin support agent, for testing before, during and after launch.
Judging LLM-as-a-Judge (MT-Bench)
Creator not supplied; see original source.
arXiv paper (2023) on using strong LLMs as judges, with MT-Bench, Chatbot Arena and analysis of judge biases.
The Most Common Questions About Agent Evals, Answered (LangChain)
Creator not supplied; see original source.
LangChain post answering common questions on agent evals: offline vs online, evaluator types, LLM judges and multi-turn tests.
Langfuse blog: Evaluating AI agent skills
Creator not supplied; see original source.
Langfuse blog post on how to evaluate AI agent skills (SKILL.md files) with traces and evals.
LangSmith docs: Evaluation concepts
Creator not supplied; see original source.
LangSmith documentation source on evaluation concepts: datasets, evaluators, offline and online evals.
Legal Technology Insider: Vals report missing the market leaders
Creator not supplied; see original source.
Legal Technology Insider article on the Vals AI legal research benchmark report, noting which major vendors did not take part.
LiveBench paper
Creator not supplied; see original source.
arXiv paper (2024) introducing LiveBench, a contamination-limited LLM benchmark with regularly refreshed questions.
LiveCodeBench paper
Creator not supplied; see original source.
arXiv paper (2024) introducing LiveCodeBench, a contamination-free code benchmark built from recent contest problems.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Creator not supplied; see original source.
arXiv paper (2025) by METR reporting a randomized trial of early-2025 AI tools' effect on experienced open-source developers' speed.