Benchmarks And Leaderboards: 512 curated resources
Curated material on benchmarks and leaderboards within evaluation.
512 resources · page 16 of 16
MLflow documentation
Creator not supplied; see original source.
MLflow documentation: experiment tracking, model registry, GenAI tracing, evaluation and deployment.
MME
Creator not supplied; see original source.
arXiv paper (2023) introducing MME, a benchmark for perception and cognition in multimodal LLMs.
Model Cards for Model Reporting
Creator not supplied; see original source.
Google Research publication page for Model Cards for Model Reporting, a standard for documenting trained models.
MoleculeNet
Creator not supplied; see original source.
arXiv paper (2017) "MoleculeNet: A Benchmark for Molecular Machine Learning", introducing the MoleculeNet benchmark datasets for molecular property prediction.
Open Ko-LLM Leaderboard2 paper
Creator not supplied; see original source.
arXiv paper (2024) describing Open Ko-LLM Leaderboard2 for evaluating Korean LLMs.
Introducing TRAIL: A Benchmark for Agentic Evaluation (Patronus AI)
Creator not supplied; see original source.
Patronus AI post introducing TRAIL, a benchmark of annotated agent traces for detecting reasoning, planning and execution errors.
Quantifying infrastructure noise in agentic coding evals
Creator not supplied; see original source.
Anthropic engineering post quantifying how infrastructure noise affects agentic coding evaluation scores.
Safetywashing
Creator not supplied; see original source.
arXiv paper (2024) "Safetywashing", testing whether AI safety benchmarks measure safety or general capability.
SEA-HELM paper
Creator not supplied; see original source.
arXiv paper (2025) introducing SEA-HELM, holistic evaluation of language models on Southeast Asian languages.
τ-Bench: Benchmarking AI agents for the real-world (Sierra)
Creator not supplied; see original source.
Sierra post introducing τ-bench, a benchmark of tool-using agents talking to simulated users, with the pass^k reliability metric.
SUPERB
Creator not supplied; see original source.
arXiv paper (2021) "SUPERB: Speech processing Universal PERformance Benchmark", introducing SUPERB, a benchmark of speech representations across many speech tasks.
SWE-Bench Pro Verified
Creator not supplied; see original source.
arXiv paper (2026) introducing SWE-Bench Pro Verified, an audited software-engineering benchmark that removes leakage and flawed tasks.
The Leaderboard Illusion
Creator not supplied; see original source.
arXiv paper (2025) "The Leaderboard Illusion", auditing how private testing and data access distort Chatbot Arena rankings.
Tutor CoPilot: a human-AI approach to real-time expertise
Creator not supplied; see original source.
arXiv paper (2024) reporting a randomized field study of Tutor CoPilot, an AI assistant for human tutors.
UNLP 2026 Ukrainian NLP workshop paper
Creator not supplied; see original source.
UNLP 2026 workshop paper (PDF) on Ukrainian NLP evaluation.
3D Arena paper
Creator not supplied; see original source.
arXiv paper (2025) introducing 3D Arena, an open pairwise-preference platform for evaluating generative 3D models.
BEHAVIOR 2025 1st-place report
Creator not supplied; see original source.
arXiv PDF (2025) of the first-place report from the NeurIPS 2025 BEHAVIOR household robotics challenge.
A framework for evaluating the chemical knowledge and reasoning abilities of large language models (ChemBench)
Creator not supplied; see original source.
ChemBench paper (Nature Chemistry 2025, PMC full text): framework evaluating chemical knowledge and reasoning of LLMs against chemists.
EvalScope τ-bench docs (ModelScope/Alibaba)
Creator not supplied; see original source.
EvalScope documentation on τ-bench: running the conversational agent benchmark with Alibaba's ModelScope evaluation framework.
FinanceBench: A New Benchmark for Financial Question Answering
Creator not supplied; see original source.
arXiv paper (2023) introducing FinanceBench, a benchmark for financial question answering over company filings.
FrontierMath paper
Creator not supplied; see original source.
arXiv paper (2024) introducing FrontierMath, a benchmark of unpublished advanced mathematics problems.
MASK paper
Creator not supplied; see original source.
arXiv PDF (2025) of the MASK benchmark paper on measuring honesty separately from accuracy in LLMs.
Medical VLM contamination audit
Creator not supplied; see original source.
arXiv paper (2026) auditing pretraining contamination of public medical vision-language benchmarks.
MMMU-Pro paper
Creator not supplied; see original source.
arXiv HTML version of the MMMU-Pro paper (2024), a harder multimodal multi-discipline benchmark.
Music Arena paper
Creator not supplied; see original source.
arXiv paper (2025) introducing Music Arena, live human-preference evaluation for text-to-music models.
Open ASR Leaderboard paper
Creator not supplied; see original source.
arXiv PDF (2025) of the Open ASR Leaderboard paper: evaluation method, datasets and results for speech recognition models.
RoboArena paper page (Penn)
Creator not supplied; see original source.
RoboArena paper page (University of Pennsylvania): distributed real-world evaluation of generalist robot policies.
Frontier AI Risk Management Framework in Practice (Shanghai AI Lab)
Creator not supplied; see original source.
arXiv paper (2025) by Shanghai AI Laboratory applying its Frontier AI Risk Management Framework to analyse model risks.
TaxPraBen: Chinese real-world tax practice
Creator not supplied; see original source.
ACL 2026 paper introducing TaxPraBen, a benchmark of LLMs on Chinese real-world tax practice tasks.
VBench-2.0 paper
Creator not supplied; see original source.
arXiv HTML version of the VBench-2.0 paper (2025) evaluating video generation on physics, commonsense and faithfulness.
When Rubrics Fail
Creator not supplied; see original source.
arXiv PDF (2026) "When Rubrics Fail", a critique of rubric-based evaluation in medical LLM benchmarks.
Who Validates the Validators? (Shankar et al.)
Creator not supplied; see original source.
arXiv paper (2024) by Shankar et al. on aligning LLM-assisted evaluators with human preferences and criteria drift.