Evaluation Benchmarks
513 resources · page 15 of 17
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
arXiv paper (2025) by METR reporting a randomized trial of early-2025 AI tools' effect on experienced open-source developers' speed.
METR time-horizon paper
arXiv paper (2025) by METR measuring AI ability to complete long software tasks as a time horizon.
The Open Arabic LLM Leaderboard 2
by Hugging Face
Hugging Face post launching the Open Arabic LLM Leaderboard 2 with native Arabic benchmarks.
OECD AI Incidents Monitor
by OECD
OECD AI Incidents Monitor: database of AI incidents and hazards compiled from news reports.
promptfoo docs: Evaluate coding agents
by promptfoo
Promptfoo guide to evaluating coding agents with its open-source eval harness.
Sarvam-M multilingual language model (Sarvam AI)
by Sarvam AI
Sarvam AI technical post on Sarvam-M, a hybrid reasoning model for Indian languages, with post-training and inference details.
SWE-bench paper
by Carlos E. Jimenez et al. (Princeton University)
arXiv paper (2023) introducing SWE-bench, evaluating language models on resolving real GitHub issues.
The SWE-Bench Illusion
by Shanchao Liang et al. (Purdue University)
arXiv paper (2025) "The SWE-Bench Illusion", testing whether LLMs memorise SWE-bench instead of reasoning.
W&B Weave documentation
by CoreWeave (Weights & Biases)
W&B Weave documentation, now hosted by CoreWeave: tracing, testing and evaluating LLM applications.
What We've Learned From A Year of Building with LLMs
by Eugene Yan, Bryan Bischof, Charles Frye, Hamel Husain, Jason Liu and Shreya Shankar
Applied LLMs guide by Yan, Bischof, Frye, Husain, Liu and Shankar: tactical, operational and strategic lessons from building LLM products.
Wikipedia: AI safety institutes
by Wikipedia (Wikimedia Foundation)
Wikipedia article on AI safety institutes: state-backed bodies that evaluate AI safety, listed by country.
Accuracy incentives and hallucinations
by Adam Tauman Kalai et al. (OpenAI)
Nature paper (2026) showing that accuracy-only evaluation of LLMs incentivises hallucination.
Agentic Benchmark Checklist (ABC)
arXiv paper (2025) proposing the Agentic Benchmark Checklist, guidelines for building valid agentic benchmarks.
ARC Prize 2025 Technical Report
by François Chollet et al. (ARC Prize Foundation)
arXiv paper (2026) ARC Prize 2025 technical report on ARC-AGI-2 results and winning approaches.
Arize Phoenix docs: Evaluating multi-agent systems
by Arize AI
Arize Phoenix documentation on evaluating multi-agent systems using traces of each agent's steps.
Benchmark-based assessment of Indian foundation models
by Avinash Agarwal and Vridhi Jain
arXiv paper (2026) comparing publicly benchmarked Indian foundation models across benchmarks.
C-Eval paper
by Yuzhen Huang et al. (Shanghai Jiao Tong University)
arXiv paper (2023) introducing C-Eval, a multi-level, multi-discipline Chinese evaluation suite for foundation models.
CLARIN-PT-LDB paper
by João Silva et al. (University of Lisbon)
arXiv paper (2026) introducing CLARIN-PT-LDB, an open LLM leaderboard for European Portuguese.
Coding-agent leaderboard resolution audit
by Fengshuo Liu et al. (Imperial College London)
arXiv paper (2026) arguing that coding agents have converged so the SWE-bench leaderboard can no longer separate them.
Cross-Context Verification: Hierarchical Detection of Benchmark Contamination
by Tae-Eun Song
arXiv paper (2026) introducing Cross-Context Verification, a hierarchical method for detecting benchmark contamination.
DuetBench (Decagon)
by Decagon
Decagon post introducing DuetBench, a benchmark for self-improving customer-service agents.
Fairlearn
Fairlearn: open-source toolkit and guides for assessing and mitigating fairness issues in ML models.
FlagEval Findings Report (LRMs)
arXiv paper (2025) FlagEval findings report: BAAI's preliminary evaluation of large reasoning models.
FlagEvalMM
arXiv paper (2025) introducing FlagEvalMM, BAAI's framework for evaluating multimodal models.
HAE-RAE Evaluation Toolkit / HRET
arXiv paper (2025) introducing the HAE-RAE Evaluation Toolkit (HRET), a unified framework for evaluating Korean LLMs.
HLE paper
by Long Phan et al. (Center for AI Safety and Scale AI)
arXiv paper (2025) introducing Humanity's Last Exam, a benchmark of expert-written questions across academic fields.
IFEval
by Jeffrey Zhou et al. (Yale University)
arXiv paper (2023) introducing IFEval, instruction-following evaluation with automatically verifiable instructions.
Inspect India Evals
by Abhishek Kumar Singh et al.
arXiv paper (2026) introducing Inspect India Evals, an open framework of Indian benchmarks with a fairness index.
J-AISI annual report 2025
Japan AI Safety Institute 2025 annual activity report (PDF, Japanese).
Korea AISI 5th AI Safety Forecast Report
by Korea AISI
Korea AI Safety Institute 5th AI Safety Forecast Report covering May–July 2026.
La Leaderboard paper
by María Grandury et al. (Universidad Politécnica de Madrid)
arXiv paper (2025) introducing La Leaderboard, an LLM leaderboard for Spanish varieties and languages of Spain and Latin America.
LabSafety Bench
by Yujun Zhou et al. (University of Notre Dame)
Nature Machine Intelligence paper introducing LabSafety Bench for LLM safety risks in scientific labs.