Evaluation for advanced learners
455 resources · page 14 of 15
Agentic Benchmark Checklist (ABC)
Creator not supplied; see original source.
arXiv paper (2025) proposing the Agentic Benchmark Checklist, guidelines for building valid agentic benchmarks.
ARC Prize 2025 Technical Report
Creator not supplied; see original source.
arXiv paper (2026) ARC Prize 2025 technical report on ARC-AGI-2 results and winning approaches.
Arize Phoenix docs: Evaluating multi-agent systems
Creator not supplied; see original source.
Arize Phoenix documentation on evaluating multi-agent systems using traces of each agent's steps.
Benchmark-based assessment of Indian foundation models
Creator not supplied; see original source.
arXiv paper (2026) comparing publicly benchmarked Indian foundation models across benchmarks.
C-Eval paper
Creator not supplied; see original source.
arXiv paper (2023) introducing C-Eval, a multi-level, multi-discipline Chinese evaluation suite for foundation models.
CLARIN-PT-LDB paper
Creator not supplied; see original source.
arXiv paper (2026) introducing CLARIN-PT-LDB, an open LLM leaderboard for European Portuguese.
Coding-agent leaderboard resolution audit
Creator not supplied; see original source.
arXiv paper (2026) arguing that coding agents have converged so the SWE-bench leaderboard can no longer separate them.
Cross-Context Verification: Hierarchical Detection of Benchmark Contamination
Creator not supplied; see original source.
arXiv paper (2026) introducing Cross-Context Verification, a hierarchical method for detecting benchmark contamination.
DuetBench (Decagon)
Creator not supplied; see original source.
Decagon post introducing DuetBench, a benchmark for self-improving customer-service agents.
Evaluating the Effectiveness of LLM-Evaluators (LLM-as-Judge)
Compare model-based graders against the judgments they must approximate.
Fairlearn
Creator not supplied; see original source.
Fairlearn: open-source toolkit and guides for assessing and mitigating fairness issues in ML models.
FlagEval Findings Report (LRMs)
Creator not supplied; see original source.
arXiv paper (2025) FlagEval findings report: BAAI's preliminary evaluation of large reasoning models.
FlagEvalMM
Creator not supplied; see original source.
arXiv paper (2025) introducing FlagEvalMM, BAAI's framework for evaluating multimodal models.
HAE-RAE Evaluation Toolkit / HRET
Creator not supplied; see original source.
arXiv paper (2025) introducing the HAE-RAE Evaluation Toolkit (HRET), a unified framework for evaluating Korean LLMs.
HLE paper
Creator not supplied; see original source.
arXiv paper (2025) introducing Humanity's Last Exam, a benchmark of expert-written questions across academic fields.
IFEval
Creator not supplied; see original source.
arXiv paper (2023) introducing IFEval, instruction-following evaluation with automatically verifiable instructions.
Inspect India Evals
Creator not supplied; see original source.
arXiv paper (2026) introducing Inspect India Evals, an open framework of Indian benchmarks with a fairness index.
J-AISI annual report 2025
Creator not supplied; see original source.
Japan AI Safety Institute 2025 annual activity report (PDF, Japanese).
Korea AISI 5th AI Safety Forecast Report
Creator not supplied; see original source.
Korea AI Safety Institute 5th AI Safety Forecast Report covering May–July 2026.
La Leaderboard paper
Creator not supplied; see original source.
arXiv paper (2025) introducing La Leaderboard, an LLM leaderboard for Spanish varieties and languages of Spain and Latin America.
LabSafety Bench
Creator not supplied; see original source.
Nature Machine Intelligence paper introducing LabSafety Bench for LLM safety risks in scientific labs.
METR experienced-developer productivity experiment
Creator not supplied; see original source.
METR study of how early-2025 AI tools affected experienced open-source developers' productivity.
MLflow documentation
Creator not supplied; see original source.
MLflow documentation: experiment tracking, model registry, GenAI tracing, evaluation and deployment.
MME
Creator not supplied; see original source.
arXiv paper (2023) introducing MME, a benchmark for perception and cognition in multimodal LLMs.
Model Cards for Model Reporting
Creator not supplied; see original source.
Google Research publication page for Model Cards for Model Reporting, a standard for documenting trained models.
MoleculeNet
Creator not supplied; see original source.
arXiv paper (2017) "MoleculeNet: A Benchmark for Molecular Machine Learning", introducing the MoleculeNet benchmark datasets for molecular property prediction.
Open Ko-LLM Leaderboard2 paper
Creator not supplied; see original source.
arXiv paper (2024) describing Open Ko-LLM Leaderboard2 for evaluating Korean LLMs.
simple-evals
by OpenAI
Read reference evaluation code with an explicit maintenance caveat.
Introducing TRAIL: A Benchmark for Agentic Evaluation (Patronus AI)
Creator not supplied; see original source.
Patronus AI post introducing TRAIL, a benchmark of annotated agent traces for detecting reasoning, planning and execution errors.
Quantifying infrastructure noise in agentic coding evals
Creator not supplied; see original source.
Anthropic engineering post quantifying how infrastructure noise affects agentic coding evaluation scores.
Safetywashing
Creator not supplied; see original source.
arXiv paper (2024) "Safetywashing", testing whether AI safety benchmarks measure safety or general capability.
SEA-HELM paper
Creator not supplied; see original source.
arXiv paper (2025) introducing SEA-HELM, holistic evaluation of language models on Southeast Asian languages.