Evaluation & benchmarks for advanced learners
452 resources · page 14 of 15
Agentic Benchmark Checklist (ABC)
arXiv paper (2025) proposing the Agentic Benchmark Checklist, guidelines for building valid agentic benchmarks.
ARC Prize 2025 Technical Report
by François Chollet et al. (ARC Prize Foundation)
arXiv paper (2026) ARC Prize 2025 technical report on ARC-AGI-2 results and winning approaches.
Arize Phoenix docs: Evaluating multi-agent systems
by Arize AI
Arize Phoenix documentation on evaluating multi-agent systems using traces of each agent's steps.
Benchmark-based assessment of Indian foundation models
by Avinash Agarwal and Vridhi Jain
arXiv paper (2026) comparing publicly benchmarked Indian foundation models across benchmarks.
C-Eval paper
by Yuzhen Huang et al. (Shanghai Jiao Tong University)
arXiv paper (2023) introducing C-Eval, a multi-level, multi-discipline Chinese evaluation suite for foundation models.
CLARIN-PT-LDB paper
by João Silva et al. (University of Lisbon)
arXiv paper (2026) introducing CLARIN-PT-LDB, an open LLM leaderboard for European Portuguese.
Coding-agent leaderboard resolution audit
by Fengshuo Liu et al. (Imperial College London)
arXiv paper (2026) arguing that coding agents have converged so the SWE-bench leaderboard can no longer separate them.
Cross-Context Verification: Hierarchical Detection of Benchmark Contamination
by Tae-Eun Song
arXiv paper (2026) introducing Cross-Context Verification, a hierarchical method for detecting benchmark contamination.
DuetBench (Decagon)
by Decagon
Decagon post introducing DuetBench, a benchmark for self-improving customer-service agents.
Fairlearn
Fairlearn: open-source toolkit and guides for assessing and mitigating fairness issues in ML models.
FlagEval Findings Report (LRMs)
arXiv paper (2025) FlagEval findings report: BAAI's preliminary evaluation of large reasoning models.
FlagEvalMM
arXiv paper (2025) introducing FlagEvalMM, BAAI's framework for evaluating multimodal models.
HAE-RAE Evaluation Toolkit / HRET
arXiv paper (2025) introducing the HAE-RAE Evaluation Toolkit (HRET), a unified framework for evaluating Korean LLMs.
HLE paper
by Long Phan et al. (Center for AI Safety and Scale AI)
arXiv paper (2025) introducing Humanity's Last Exam, a benchmark of expert-written questions across academic fields.
IFEval
by Jeffrey Zhou et al. (Yale University)
arXiv paper (2023) introducing IFEval, instruction-following evaluation with automatically verifiable instructions.
Inspect India Evals
by Abhishek Kumar Singh et al.
arXiv paper (2026) introducing Inspect India Evals, an open framework of Indian benchmarks with a fairness index.
J-AISI annual report 2025
Japan AI Safety Institute 2025 annual activity report (PDF, Japanese).
Korea AISI 5th AI Safety Forecast Report
by Korea AISI
Korea AI Safety Institute 5th AI Safety Forecast Report covering May–July 2026.
La Leaderboard paper
by María Grandury et al. (Universidad Politécnica de Madrid)
arXiv paper (2025) introducing La Leaderboard, an LLM leaderboard for Spanish varieties and languages of Spain and Latin America.
LabSafety Bench
by Yujun Zhou et al. (University of Notre Dame)
Nature Machine Intelligence paper introducing LabSafety Bench for LLM safety risks in scientific labs.
METR experienced-developer productivity experiment
by METR
METR study of how early-2025 AI tools affected experienced open-source developers' productivity.
MLflow documentation
by MLflow project (Databricks)
MLflow documentation: experiment tracking, model registry, GenAI tracing, evaluation and deployment.
MME
by Chaoyou Fu et al. (Nanjing University)
arXiv paper (2023) introducing MME, a benchmark for perception and cognition in multimodal LLMs.
Model Cards for Model Reporting
Google Research publication page for Model Cards for Model Reporting, a standard for documenting trained models.
MoleculeNet
by Zhenqin Wu et al. (Stanford University)
arXiv paper (2017) "MoleculeNet: A Benchmark for Molecular Machine Learning", introducing the MoleculeNet benchmark datasets for molecular property prediction.
Open Ko-LLM Leaderboard2 paper
by Hyeonwoo Kim et al. (Upstage)
arXiv paper (2024) describing Open Ko-LLM Leaderboard2 for evaluating Korean LLMs.
Introducing TRAIL: A Benchmark for Agentic Evaluation (Patronus AI)
by Patronus AI
Patronus AI post introducing TRAIL, a benchmark of annotated agent traces for detecting reasoning, planning and execution errors.
Quantifying infrastructure noise in agentic coding evals
by Anthropic
Anthropic engineering post quantifying how infrastructure noise affects agentic coding evaluation scores.
Safetywashing
by Richard Ren et al. (Center for AI Safety)
arXiv paper (2024) "Safetywashing", testing whether AI safety benchmarks measure safety or general capability.
SEA-HELM paper
by Yosephine Susanto et al. (AI Singapore)
arXiv paper (2025) introducing SEA-HELM, holistic evaluation of language models on Southeast Asian languages.
τ-Bench: Benchmarking AI agents for the real-world (Sierra)
by Sierra
Sierra post introducing τ-bench, a benchmark of tool-using agents talking to simulated users, with the pass^k reliability metric.
SUPERB
by Shu-wen Yang et al. (National Taiwan University)
arXiv paper (2021) "SUPERB: Speech processing Universal PERformance Benchmark", introducing SUPERB, a benchmark of speech representations across many speech tasks.