Benchmarks And Leaderboards: 512 curated resources
Curated material on benchmarks and leaderboards within evaluation.
512 resources · page 15 of 16
METR time-horizon paper
Creator not supplied; see original source.
arXiv paper (2025) by METR measuring AI ability to complete long software tasks as a time horizon.
The Open Arabic LLM Leaderboard 2
Creator not supplied; see original source.
Hugging Face post launching the Open Arabic LLM Leaderboard 2 with native Arabic benchmarks.
OECD AI Incidents Monitor
Creator not supplied; see original source.
OECD AI Incidents Monitor: database of AI incidents and hazards compiled from news reports.
promptfoo docs: Evaluate coding agents
Creator not supplied; see original source.
Promptfoo guide to evaluating coding agents with its open-source eval harness.
Sarvam-M multilingual language model (Sarvam AI)
Creator not supplied; see original source.
Sarvam AI technical post on Sarvam-M, a hybrid reasoning model for Indian languages, with post-training and inference details.
SWE-bench paper
Creator not supplied; see original source.
arXiv paper (2023) introducing SWE-bench, evaluating language models on resolving real GitHub issues.
The SWE-Bench Illusion
Creator not supplied; see original source.
arXiv paper (2025) "The SWE-Bench Illusion", testing whether LLMs memorise SWE-bench instead of reasoning.
W&B Weave documentation
Creator not supplied; see original source.
W&B Weave documentation, now hosted by CoreWeave: tracing, testing and evaluating LLM applications.
What We've Learned From A Year of Building with LLMs
Creator not supplied; see original source.
Applied LLMs guide by Yan, Bischof, Frye, Husain, Liu and Shankar: tactical, operational and strategic lessons from building LLM products.
Wikipedia: AI safety institutes
Creator not supplied; see original source.
Wikipedia article on AI safety institutes: state-backed bodies that evaluate AI safety, listed by country.
Accuracy incentives and hallucinations
Creator not supplied; see original source.
Nature paper (2026) showing that accuracy-only evaluation of LLMs incentivises hallucination.
Agentic Benchmark Checklist (ABC)
Creator not supplied; see original source.
arXiv paper (2025) proposing the Agentic Benchmark Checklist, guidelines for building valid agentic benchmarks.
ARC Prize 2025 Technical Report
Creator not supplied; see original source.
arXiv paper (2026) ARC Prize 2025 technical report on ARC-AGI-2 results and winning approaches.
Arize Phoenix docs: Evaluating multi-agent systems
Creator not supplied; see original source.
Arize Phoenix documentation on evaluating multi-agent systems using traces of each agent's steps.
Benchmark-based assessment of Indian foundation models
Creator not supplied; see original source.
arXiv paper (2026) comparing publicly benchmarked Indian foundation models across benchmarks.
C-Eval paper
Creator not supplied; see original source.
arXiv paper (2023) introducing C-Eval, a multi-level, multi-discipline Chinese evaluation suite for foundation models.
CLARIN-PT-LDB paper
Creator not supplied; see original source.
arXiv paper (2026) introducing CLARIN-PT-LDB, an open LLM leaderboard for European Portuguese.
Coding-agent leaderboard resolution audit
Creator not supplied; see original source.
arXiv paper (2026) arguing that coding agents have converged so the SWE-bench leaderboard can no longer separate them.
Cross-Context Verification: Hierarchical Detection of Benchmark Contamination
Creator not supplied; see original source.
arXiv paper (2026) introducing Cross-Context Verification, a hierarchical method for detecting benchmark contamination.
DuetBench (Decagon)
Creator not supplied; see original source.
Decagon post introducing DuetBench, a benchmark for self-improving customer-service agents.
Fairlearn
Creator not supplied; see original source.
Fairlearn: open-source toolkit and guides for assessing and mitigating fairness issues in ML models.
FlagEval Findings Report (LRMs)
Creator not supplied; see original source.
arXiv paper (2025) FlagEval findings report: BAAI's preliminary evaluation of large reasoning models.
FlagEvalMM
Creator not supplied; see original source.
arXiv paper (2025) introducing FlagEvalMM, BAAI's framework for evaluating multimodal models.
HAE-RAE Evaluation Toolkit / HRET
Creator not supplied; see original source.
arXiv paper (2025) introducing the HAE-RAE Evaluation Toolkit (HRET), a unified framework for evaluating Korean LLMs.
HLE paper
Creator not supplied; see original source.
arXiv paper (2025) introducing Humanity's Last Exam, a benchmark of expert-written questions across academic fields.
IFEval
Creator not supplied; see original source.
arXiv paper (2023) introducing IFEval, instruction-following evaluation with automatically verifiable instructions.
Inspect India Evals
Creator not supplied; see original source.
arXiv paper (2026) introducing Inspect India Evals, an open framework of Indian benchmarks with a fairness index.
J-AISI annual report 2025
Creator not supplied; see original source.
Japan AI Safety Institute 2025 annual activity report (PDF, Japanese).
Korea AISI 5th AI Safety Forecast Report
Creator not supplied; see original source.
Korea AI Safety Institute 5th AI Safety Forecast Report covering May–July 2026.
La Leaderboard paper
Creator not supplied; see original source.
arXiv paper (2025) introducing La Leaderboard, an LLM leaderboard for Spanish varieties and languages of Spain and Latin America.
LabSafety Bench
Creator not supplied; see original source.
Nature Machine Intelligence paper introducing LabSafety Bench for LLM safety risks in scientific labs.
METR experienced-developer productivity experiment
Creator not supplied; see original source.
METR study of how early-2025 AI tools affected experienced open-source developers' productivity.