Evaluation & benchmarks
Evaluation & benchmarks
- agent & coding evals (85)
- autonomous driving (1)
- general leaderboards (248)
- modality evals (image/video/speech) (87)
- physics & simulation (1)
- safety & efficiency evals (44)
- safety evals (1)
- science & domain evals (45)
- vision (1)
513 resources · page 16 of 17
METR experienced-developer productivity experiment
by METR
METR study of how early-2025 AI tools affected experienced open-source developers' productivity.
MLflow documentation
by MLflow project (Databricks)
MLflow documentation: experiment tracking, model registry, GenAI tracing, evaluation and deployment.
MME
by Chaoyou Fu et al. (Nanjing University)
arXiv paper (2023) introducing MME, a benchmark for perception and cognition in multimodal LLMs.
Model Cards for Model Reporting
Google Research publication page for Model Cards for Model Reporting, a standard for documenting trained models.
MoleculeNet
by Zhenqin Wu et al. (Stanford University)
arXiv paper (2017) "MoleculeNet: A Benchmark for Molecular Machine Learning", introducing the MoleculeNet benchmark datasets for molecular property prediction.
Open Ko-LLM Leaderboard2 paper
by Hyeonwoo Kim et al. (Upstage)
arXiv paper (2024) describing Open Ko-LLM Leaderboard2 for evaluating Korean LLMs.
Introducing TRAIL: A Benchmark for Agentic Evaluation (Patronus AI)
by Patronus AI
Patronus AI post introducing TRAIL, a benchmark of annotated agent traces for detecting reasoning, planning and execution errors.
Quantifying infrastructure noise in agentic coding evals
by Anthropic
Anthropic engineering post quantifying how infrastructure noise affects agentic coding evaluation scores.
Safetywashing
by Richard Ren et al. (Center for AI Safety)
arXiv paper (2024) "Safetywashing", testing whether AI safety benchmarks measure safety or general capability.
SEA-HELM paper
by Yosephine Susanto et al. (AI Singapore)
arXiv paper (2025) introducing SEA-HELM, holistic evaluation of language models on Southeast Asian languages.
τ-Bench: Benchmarking AI agents for the real-world (Sierra)
by Sierra
Sierra post introducing τ-bench, a benchmark of tool-using agents talking to simulated users, with the pass^k reliability metric.
SUPERB
by Shu-wen Yang et al. (National Taiwan University)
arXiv paper (2021) "SUPERB: Speech processing Universal PERformance Benchmark", introducing SUPERB, a benchmark of speech representations across many speech tasks.
SWE-Bench Pro Verified
by Pujun Zheng et al. (Shanghai Artificial Intelligence Laboratory)
arXiv paper (2026) introducing SWE-Bench Pro Verified, an audited software-engineering benchmark that removes leakage and flawed tasks.
The Leaderboard Illusion
by Shivalika Singh et al. (Cohere)
arXiv paper (2025) "The Leaderboard Illusion", auditing how private testing and data access distort Chatbot Arena rankings.
Tutor CoPilot: a human-AI approach to real-time expertise
by Rose E. Wang et al. (Stanford University)
arXiv paper (2024) reporting a randomized field study of Tutor CoPilot, an AI assistant for human tutors.
UNLP 2026 Ukrainian NLP workshop paper
UNLP 2026 workshop paper (PDF) on Ukrainian NLP evaluation.
3D Arena paper
arXiv paper (2025) introducing 3D Arena, an open pairwise-preference platform for evaluating generative 3D models.
BEHAVIOR 2025 1st-place report
by Ilia Larchenko, Gleb Zarin and Akash Karnatak
arXiv PDF (2025) of the first-place report from the NeurIPS 2025 BEHAVIOR household robotics challenge.
A framework for evaluating the chemical knowledge and reasoning abilities of large language models (ChemBench)
by Adrian Mirza et al. (Helmholtz Institute Jena)
ChemBench paper (Nature Chemistry 2025, PMC full text): framework evaluating chemical knowledge and reasoning of LLMs against chemists.
EvalScope τ-bench docs (ModelScope/Alibaba)
EvalScope documentation on τ-bench: running the conversational agent benchmark with Alibaba's ModelScope evaluation framework.
FinanceBench: A New Benchmark for Financial Question Answering
by Pranab Islam et al. (Stanford University)
arXiv paper (2023) introducing FinanceBench, a benchmark for financial question answering over company filings.
FrontierMath paper
by Elliot Glazer et al. (Rutgers, The State University of New Jersey)
arXiv paper (2024) introducing FrontierMath, a benchmark of unpublished advanced mathematics problems.
MASK paper
by Richard Ren et al. (Center for AI Safety)
arXiv PDF (2025) of the MASK benchmark paper on measuring honesty separately from accuracy in LLMs.
Medical VLM contamination audit
by Bruce Changlong Xu et al. (Stanford University)
arXiv paper (2026) auditing pretraining contamination of public medical vision-language benchmarks.
MMMU-Pro paper
by Xiang Yue et al. (Carnegie Mellon University)
arXiv HTML version of the MMMU-Pro paper (2024), a harder multimodal multi-discipline benchmark.
Music Arena paper
by Yonghyun Kim et al. (Carnegie Mellon University, LMArena and Sony AI)
arXiv paper (2025) introducing Music Arena, live human-preference evaluation for text-to-music models.
Open ASR Leaderboard paper
by Vaibhav Srivastav et al. (Hugging Face)
arXiv PDF (2025) of the Open ASR Leaderboard paper: evaluation method, datasets and results for speech recognition models.
RoboArena paper page (Penn)
RoboArena paper page (University of Pennsylvania): distributed real-world evaluation of generalist robot policies.
Frontier AI Risk Management Framework in Practice (Shanghai AI Lab)
by Shanghai AI Lab (Shanghai AI Laboratory)
arXiv paper (2025) by Shanghai AI Laboratory applying its Frontier AI Risk Management Framework to analyse model risks.
TaxPraBen: Chinese real-world tax practice
ACL 2026 paper introducing TaxPraBen, a benchmark of LLMs on Chinese real-world tax practice tasks.
VBench-2.0 paper
by Dian Zheng et al. (Nanyang Technological University and Shanghai AI Laboratory)
arXiv HTML version of the VBench-2.0 paper (2025) evaluating video generation on physics, commonsense and faithfulness.
When Rubrics Fail
by Griffin Farrow et al. (University of Oxford)
arXiv PDF (2026) "When Rubrics Fail", a critique of rubric-based evaluation in medical LLM benchmarks.