Evaluation
Measure model capabilities and application behavior with explicit criteria.
- Benchmarks And Leaderboards (512)
- Eval Harnesses (3)
- Application Evals (2)
- Llm As Judge (1)
- Agent Evals (1)
519 resources · page 14 of 17
Langfuse joins ClickHouse
Creator not supplied; see original source.
Langfuse post announcing the company joined ClickHouse and describing plans for the open-source project.
Latent Space: SWE-bench dead
Creator not supplied; see original source.
Latent Space episode with OpenAI's Mia Glaese and Olivia Watkins on retiring SWE-bench Verified and next frontier evals.
MMLU (classic)
Creator not supplied; see original source.
arXiv paper (2020) "Measuring Massive Multitask Language Understanding", introducing MMLU, a 57-subject multitask knowledge benchmark.
BEHAVIOR Challenge Charts the Way Forward for Domestic Robotics (Stanford HAI)
Creator not supplied; see original source.
Stanford HAI article on the BEHAVIOR Challenge, a household-robotics competition in simulation held at NeurIPS 2025.
AI Agents That Matter
Creator not supplied; see original source.
arXiv paper (2024) "AI Agents That Matter", on cost-aware and reproducible evaluation of AI agents.
March 2026: Arena Updates
Creator not supplied; see original source.
Arena blog post on March 2026 updates: new document and video-edit arenas and leaderboard changes.
Arize Phoenix docs: LLM-as-a-judge
Creator not supplied; see original source.
Arize Phoenix documentation on LLM-as-a-judge evaluation: designing judge prompts and criteria.
Berget AI: AI in Swedish
Creator not supplied; see original source.
Berget AI post on choosing models for Swedish-language applications, pointing to EuroEval and Scandinavian embedding benchmarks.
Braintrust documentation
Creator not supplied; see original source.
Braintrust documentation for running experiments, scoring outputs and collecting production feedback on AI apps.
Braintrust docs: Evaluate systematically
Creator not supplied; see original source.
Braintrust documentation on systematic evaluation: datasets, scorers and regression detection for AI applications.
Can We Trust AI Benchmarks?
Creator not supplied; see original source.
arXiv paper (2025) reviewing about 100 studies on the trustworthiness and limitations of AI benchmarks.
Chatbot Arena paper
Creator not supplied; see original source.
arXiv paper (2024) describing Chatbot Arena, its crowdsourced pairwise voting and Bradley-Terry ranking method.
How Some of China's Top AI Thinkers Built Their Own AI Safety Institute (Carnegie)
Creator not supplied; see original source.
Carnegie Endowment analysis of the China AI Safety and Development Association and its role in Chinese AI governance.
Demystifying evals for AI agents
Creator not supplied; see original source.
Anthropic post on designing evals for AI agents, combining automated tests, graders and human review.
SimpleQA Verified: Benchmark review (Epoch AI)
Creator not supplied; see original source.
Epoch AI review of the SimpleQA Verified benchmark, checking a sample of questions for correctness.
Evaluating the Effectiveness of LLM-Evaluators (Eugene Yan)
Creator not supplied; see original source.
Eugene Yan's review of LLM-evaluator (LLM-as-judge) research: use cases, techniques, alignment and critiques.
GAIA paper
Creator not supplied; see original source.
arXiv paper (2023) introducing GAIA, a benchmark of real-world questions for general AI assistants.
Announcing Agent Control (Galileo)
Creator not supplied; see original source.
Galileo post announcing Agent Control, an open-source control plane for runtime guardrails on AI agents.
Launching Agent Leaderboard v2 (Galileo)
Creator not supplied; see original source.
Galileo post introducing Agent Leaderboard v2, which evaluates LLMs as enterprise customer-support agents.
Best Practices to Navigate the Complexities of Evaluating AI Agents (Galileo)
Creator not supplied; see original source.
Galileo guide to evaluating AI agents, covering accuracy metrics, action evaluation and production monitoring.
SberDevices blog on Habr
Creator not supplied; see original source.
SberDevices company blog on Habr with Russian-language technical posts on GigaChat models and evaluation.
AI Evals FAQ
by Hamel Husain and Shreya Shankar
Practical answers about error analysis, annotation and reliable evaluators.
Your AI Product Needs Evals
by Hamel Husain
Build a feedback loop from real failures, not general benchmark scores.
Using LLM-as-a-Judge For Evaluation: A Complete Guide (Hamel Husain)
Creator not supplied; see original source.
Hamel Husain's guide to building an LLM judge and validating it against domain-expert labels.
HF blog: Hebrew leaderboard
Creator not supplied; see original source.
Hugging Face blog source file announcing the Hebrew LLM leaderboard and its evaluation tasks.
AI Evals: Everything You Need to Know (Hamel Husain)
Creator not supplied; see original source.
Hamel Husain and Shreya Shankar's AI evals FAQ for engineers and PMs: testing AI systems, error analysis, RAG and agent evals.
Announcing Evals and Releases for Fin (Intercom)
Creator not supplied; see original source.
Intercom post announcing Evals and Releases for its Fin support agent, for testing before, during and after launch.
Judging LLM-as-a-Judge (MT-Bench)
Creator not supplied; see original source.
arXiv paper (2023) on using strong LLMs as judges, with MT-Bench, Chatbot Arena and analysis of judge biases.
The Most Common Questions About Agent Evals, Answered (LangChain)
Creator not supplied; see original source.
LangChain post answering common questions on agent evals: offline vs online, evaluator types, LLM judges and multi-turn tests.
Langfuse blog: Evaluating AI agent skills
Creator not supplied; see original source.
Langfuse blog post on how to evaluate AI agent skills (SKILL.md files) with traces and evals.
LangSmith docs: Evaluation concepts
Creator not supplied; see original source.
LangSmith documentation source on evaluation concepts: datasets, evaluators, offline and online evals.
Legal Technology Insider: Vals report missing the market leaders
Creator not supplied; see original source.
Legal Technology Insider article on the Vals AI legal research benchmark report, noting which major vendors did not take part.