Evaluation
Measure model capabilities and application behavior with explicit criteria.
- Benchmarks And Leaderboards (512)
- Eval Harnesses (3)
- Application Evals (2)
- Llm As Judge (1)
- Agent Evals (1)
519 resources · page 16 of 17
J-AISI annual report 2025
Creator not supplied; see original source.
Japan AI Safety Institute 2025 annual activity report (PDF, Japanese).
Korea AISI 5th AI Safety Forecast Report
Creator not supplied; see original source.
Korea AI Safety Institute 5th AI Safety Forecast Report covering May–July 2026.
La Leaderboard paper
Creator not supplied; see original source.
arXiv paper (2025) introducing La Leaderboard, an LLM leaderboard for Spanish varieties and languages of Spain and Latin America.
LabSafety Bench
Creator not supplied; see original source.
Nature Machine Intelligence paper introducing LabSafety Bench for LLM safety risks in scientific labs.
METR experienced-developer productivity experiment
Creator not supplied; see original source.
METR study of how early-2025 AI tools affected experienced open-source developers' productivity.
MLflow documentation
Creator not supplied; see original source.
MLflow documentation: experiment tracking, model registry, GenAI tracing, evaluation and deployment.
MME
Creator not supplied; see original source.
arXiv paper (2023) introducing MME, a benchmark for perception and cognition in multimodal LLMs.
Model Cards for Model Reporting
Creator not supplied; see original source.
Google Research publication page for Model Cards for Model Reporting, a standard for documenting trained models.
MoleculeNet
Creator not supplied; see original source.
arXiv paper (2017) "MoleculeNet: A Benchmark for Molecular Machine Learning", introducing the MoleculeNet benchmark datasets for molecular property prediction.
Open Ko-LLM Leaderboard2 paper
Creator not supplied; see original source.
arXiv paper (2024) describing Open Ko-LLM Leaderboard2 for evaluating Korean LLMs.
simple-evals
by OpenAI
Read reference evaluation code with an explicit maintenance caveat.
Introducing TRAIL: A Benchmark for Agentic Evaluation (Patronus AI)
Creator not supplied; see original source.
Patronus AI post introducing TRAIL, a benchmark of annotated agent traces for detecting reasoning, planning and execution errors.
Quantifying infrastructure noise in agentic coding evals
Creator not supplied; see original source.
Anthropic engineering post quantifying how infrastructure noise affects agentic coding evaluation scores.
Safetywashing
Creator not supplied; see original source.
arXiv paper (2024) "Safetywashing", testing whether AI safety benchmarks measure safety or general capability.
SEA-HELM paper
Creator not supplied; see original source.
arXiv paper (2025) introducing SEA-HELM, holistic evaluation of language models on Southeast Asian languages.
τ-Bench: Benchmarking AI agents for the real-world (Sierra)
Creator not supplied; see original source.
Sierra post introducing τ-bench, a benchmark of tool-using agents talking to simulated users, with the pass^k reliability metric.
SUPERB
Creator not supplied; see original source.
arXiv paper (2021) "SUPERB: Speech processing Universal PERformance Benchmark", introducing SUPERB, a benchmark of speech representations across many speech tasks.
SWE-Bench Pro Verified
Creator not supplied; see original source.
arXiv paper (2026) introducing SWE-Bench Pro Verified, an audited software-engineering benchmark that removes leakage and flawed tasks.
The Leaderboard Illusion
Creator not supplied; see original source.
arXiv paper (2025) "The Leaderboard Illusion", auditing how private testing and data access distort Chatbot Arena rankings.
Tutor CoPilot: a human-AI approach to real-time expertise
Creator not supplied; see original source.
arXiv paper (2024) reporting a randomized field study of Tutor CoPilot, an AI assistant for human tutors.
Inspect
by UK AI Security Institute and Meridian Labs
Compose evaluation datasets, agents, tools and scorers.
UNLP 2026 Ukrainian NLP workshop paper
Creator not supplied; see original source.
UNLP 2026 workshop paper (PDF) on Ukrainian NLP evaluation.
3D Arena paper
Creator not supplied; see original source.
arXiv paper (2025) introducing 3D Arena, an open pairwise-preference platform for evaluating generative 3D models.
BEHAVIOR 2025 1st-place report
Creator not supplied; see original source.
arXiv PDF (2025) of the first-place report from the NeurIPS 2025 BEHAVIOR household robotics challenge.
A framework for evaluating the chemical knowledge and reasoning abilities of large language models (ChemBench)
Creator not supplied; see original source.
ChemBench paper (Nature Chemistry 2025, PMC full text): framework evaluating chemical knowledge and reasoning of LLMs against chemists.
EvalScope τ-bench docs (ModelScope/Alibaba)
Creator not supplied; see original source.
EvalScope documentation on τ-bench: running the conversational agent benchmark with Alibaba's ModelScope evaluation framework.
FinanceBench: A New Benchmark for Financial Question Answering
Creator not supplied; see original source.
arXiv paper (2023) introducing FinanceBench, a benchmark for financial question answering over company filings.
FrontierMath paper
Creator not supplied; see original source.
arXiv paper (2024) introducing FrontierMath, a benchmark of unpublished advanced mathematics problems.
MASK paper
Creator not supplied; see original source.
arXiv PDF (2025) of the MASK benchmark paper on measuring honesty separately from accuracy in LLMs.
Medical VLM contamination audit
Creator not supplied; see original source.
arXiv paper (2026) auditing pretraining contamination of public medical vision-language benchmarks.
MMMU-Pro paper
Creator not supplied; see original source.
arXiv HTML version of the MMMU-Pro paper (2024), a harder multimodal multi-discipline benchmark.
Music Arena paper
Creator not supplied; see original source.
arXiv paper (2025) introducing Music Arena, live human-preference evaluation for text-to-music models.