Evaluation Benchmarks
513 resources · page 14 of 17
Langfuse joins ClickHouse
by Langfuse
Langfuse post announcing the company joined ClickHouse and describing plans for the open-source project.
Latent Space: SWE-bench dead
by Latent Space
Latent Space episode with OpenAI's Mia Glaese and Olivia Watkins on retiring SWE-bench Verified and next frontier evals.
MMLU (classic)
by Dan Hendrycks et al. (University of Chicago)
arXiv paper (2020) "Measuring Massive Multitask Language Understanding", introducing MMLU, a 57-subject multitask knowledge benchmark.
BEHAVIOR Challenge Charts the Way Forward for Domestic Robotics (Stanford HAI)
by Stanford HAI
Stanford HAI article on the BEHAVIOR Challenge, a household-robotics competition in simulation held at NeurIPS 2025.
AI Agents That Matter
by Sayash Kapoor et al. (Princeton University)
arXiv paper (2024) "AI Agents That Matter", on cost-aware and reproducible evaluation of AI agents.
March 2026: Arena Updates
Arena blog post on March 2026 updates: new document and video-edit arenas and leaderboard changes.
Arize Phoenix docs: LLM-as-a-judge
by Arize AI
Arize Phoenix documentation on LLM-as-a-judge evaluation: designing judge prompts and criteria.
Berget AI: AI in Swedish
by Berget AI
Berget AI post on choosing models for Swedish-language applications, pointing to EuroEval and Scandinavian embedding benchmarks.
Braintrust documentation
by Braintrust
Braintrust documentation for running experiments, scoring outputs and collecting production feedback on AI apps.
Braintrust docs: Evaluate systematically
by Braintrust
Braintrust documentation on systematic evaluation: datasets, scorers and regression detection for AI applications.
Can We Trust AI Benchmarks?
by Maria Eriksson et al. (European Commission)
arXiv paper (2025) reviewing about 100 studies on the trustworthiness and limitations of AI benchmarks.
Chatbot Arena paper
by Wei-Lin Chiang et al. (LMSYS, UC Berkeley)
arXiv paper (2024) describing Chatbot Arena, its crowdsourced pairwise voting and Bradley-Terry ranking method.
How Some of China's Top AI Thinkers Built Their Own AI Safety Institute (Carnegie)
by Carnegie Endowment for International Peace
Carnegie Endowment analysis of the China AI Safety and Development Association and its role in Chinese AI governance.
Demystifying evals for AI agents
by Anthropic
Anthropic post on designing evals for AI agents, combining automated tests, graders and human review.
SimpleQA Verified: Benchmark review (Epoch AI)
by Epoch AI
Epoch AI review of the SimpleQA Verified benchmark, checking a sample of questions for correctness.
Evaluating the Effectiveness of LLM-Evaluators (Eugene Yan)
by Eugene Yan
Eugene Yan's review of LLM-evaluator (LLM-as-judge) research: use cases, techniques, alignment and critiques.
GAIA paper
by Grégoire Mialon et al. (Meta AI and Hugging Face)
arXiv paper (2023) introducing GAIA, a benchmark of real-world questions for general AI assistants.
Announcing Agent Control (Galileo)
by Galileo
Galileo post announcing Agent Control, an open-source control plane for runtime guardrails on AI agents.
Launching Agent Leaderboard v2 (Galileo)
by Galileo
Galileo post introducing Agent Leaderboard v2, which evaluates LLMs as enterprise customer-support agents.
Best Practices to Navigate the Complexities of Evaluating AI Agents (Galileo)
by Galileo
Galileo guide to evaluating AI agents, covering accuracy metrics, action evaluation and production monitoring.
SberDevices blog on Habr
by SberDevices
SberDevices company blog on Habr with Russian-language technical posts on GigaChat models and evaluation.
Using LLM-as-a-Judge For Evaluation: A Complete Guide (Hamel Husain)
by Hamel Husain
Hamel Husain's guide to building an LLM judge and validating it against domain-expert labels.
HF blog: Hebrew leaderboard
by Hugging Face
Hugging Face blog source file announcing the Hebrew LLM leaderboard and its evaluation tasks.
AI Evals: Everything You Need to Know (Hamel Husain)
by Hamel Husain
Hamel Husain and Shreya Shankar's AI evals FAQ for engineers and PMs: testing AI systems, error analysis, RAG and agent evals.
Announcing Evals and Releases for Fin (Intercom)
by Intercom
Intercom post announcing Evals and Releases for its Fin support agent, for testing before, during and after launch.
Judging LLM-as-a-Judge (MT-Bench)
by Lianmin Zheng et al. (LMSYS, UC Berkeley)
arXiv paper (2023) on using strong LLMs as judges, with MT-Bench, Chatbot Arena and analysis of judge biases.
The Most Common Questions About Agent Evals, Answered (LangChain)
by LangChain
LangChain post answering common questions on agent evals: offline vs online, evaluator types, LLM judges and multi-turn tests.
Langfuse blog: Evaluating AI agent skills
by Langfuse
Langfuse blog post on how to evaluate AI agent skills (SKILL.md files) with traces and evals.
LangSmith docs: Evaluation concepts
by LangChain
LangSmith documentation source on evaluation concepts: datasets, evaluators, offline and online evals.
Legal Technology Insider: Vals report missing the market leaders
Legal Technology Insider article on the Vals AI legal research benchmark report, noting which major vendors did not take part.
LiveBench paper
by Colin White et al. (Abacus.AI)
arXiv paper (2024) introducing LiveBench, a contamination-limited LLM benchmark with regularly refreshed questions.
LiveCodeBench paper
by Naman Jain et al. (Cornell University)
arXiv paper (2024) introducing LiveCodeBench, a contamination-free code benchmark built from recent contest problems.