Evaluation

Measure model capabilities and application behavior with explicit criteria.

• Benchmarks And Leaderboards (512)
• Eval Harnesses (3)
• Application Evals (2)
• Llm As Judge (1)
• Agent Evals (1)

519 resources · page 7 of 17

• LIBERO-Plus

  Creator not supplied; see original source.

  LIBERO-Plus paper: a robustness analysis of vision-language-action models using systematically perturbed LIBERO manipulation tasks.

  Not applicable (entity) · unknown · unknown resource · en

• LIBERO-PRO

  Creator not supplied; see original source.

  LIBERO-PRO paper testing vision-language-action models under perturbed objects, instructions and environments to check if LIBERO scores generalise.

  Not applicable (entity) · unknown · unknown resource · en

• LiveBench

  Creator not supplied; see original source.

  LiveBench 2.0: regularly refreshed, contamination-limited benchmark of long-horizon real-work tasks.

  Not applicable (entity) · unknown · unknown resource · en

• LiveCodeBench

  Creator not supplied; see original source.

  LiveCodeBench leaderboard: code generation results on problems collected from recent programming contests.

  Not applicable (entity) · unknown · unknown resource · en

• LiveMedBench

  Creator not supplied; see original source.

  LiveMedBench paper page: a continuously refreshed medical LLM benchmark built from real clinical cases with automated rubric scoring.

  Not applicable (entity) · unknown · unknown resource · en

• LLM-jp Eval

  Creator not supplied; see original source.

  llm-jp-eval GitHub repo: tool for automatically evaluating Japanese LLMs across multiple datasets.

  Not applicable (entity) · unknown · unknown resource · ja

• Arena text-to-image leaderboard

  Creator not supplied; see original source.

  Arena (formerly LMArena) text-to-image leaderboard: human-preference rankings of image generation models.

  Not applicable (entity) · unknown · unknown resource · en

• Arena text-to-video leaderboard

  Creator not supplied; see original source.

  Arena (formerly LMArena) text-to-video leaderboard: human-preference rankings of video generation models.

  Not applicable (entity) · unknown · unknown resource · en

• Arena Leaderboard (formerly LMArena)

  Creator not supplied; see original source.

  Arena (formerly LMArena) leaderboards: human-preference rankings across text, webdev, vision, image, video and search arenas.

  Not applicable (entity) · unknown · unknown resource · en

• LMMS-Eval

  Creator not supplied; see original source.

  LMMS-Eval GitHub repo: evaluation toolkit for multimodal models across text, image, video and audio tasks.

  Not applicable (entity) · unknown · unknown resource · en

• LongBench

  Creator not supplied; see original source.

  LongBench GitHub repo from THUDM: bilingual long-context benchmarks LongBench v1 and v2 with datasets and evaluation code.

  Not applicable (entity) · unknown · unknown resource · zh

• Probing the limitations of multimodal language models for chemistry and materials research (MaCBench)

  Creator not supplied; see original source.

  MaCBench paper probing multimodal language models on chemistry and materials-science tasks such as reading spectra and lab images.

  Not applicable (entity) · unknown · unknown resource · en

• ManiSkill

  Creator not supplied; see original source.

  ManiSkill site: GPU-accelerated simulation environments for training and evaluating robot manipulation policies at scale.

  Not applicable (entity) · unknown · unknown resource · en

• MASK (Scale Labs leaderboard)

  Creator not supplied; see original source.

  Scale Labs MASK leaderboard: benchmark by Scale AI and CAIS measuring honesty separately from accuracy.

  Not applicable (entity) · unknown · unknown resource · en

• Matbench

  Creator not supplied; see original source.

  Materials Project Matbench GitHub repo: benchmark tasks for materials property prediction.

  Not applicable (entity) · unknown · unknown resource · en

• Matbench Discovery

  Creator not supplied; see original source.

  Matbench Discovery GitHub repo: evaluation framework for ML models in materials stability discovery.

  Not applicable (entity) · unknown · unknown resource · en

• MATH

  Creator not supplied; see original source.

  MATH dataset GitHub repo: competition mathematics problems with step-by-step solutions for evaluating reasoning.

  Not applicable (entity) · unknown · unknown resource · en

• MathArena

  Creator not supplied; see original source.

  MathArena: evaluation of LLMs on newly released math competitions and arXiv-derived problems to avoid contamination.

  Not applicable (entity) · unknown · unknown resource · en

• MathVision

  Creator not supplied; see original source.

  MATH-Vision project page: competition math problems with visual contexts for evaluating multimodal reasoning.

  Not applicable (entity) · unknown · unknown resource · en

• MathVista

  Creator not supplied; see original source.

  MathVista project page: benchmark of math reasoning in visual contexts such as charts, diagrams and figures.

  Not applicable (entity) · unknown · unknown resource · en

• MedBench

  Creator not supplied; see original source.

  OpenCompass MedBench portal: Chinese medical benchmark platform for evaluating LLMs on clinical knowledge and tasks.

  Not applicable (entity) · unknown · unknown resource · zh

• MedHELM

  Creator not supplied; see original source.

  Stanford CRFM MedHELM: evaluation of language models on clinician-defined medical tasks beyond exam-style questions.

  Not applicable (entity) · unknown · unknown resource · en

• MedHELM (Stanford CRFM)

  Creator not supplied; see original source.

  Latest MedHELM results page from Stanford CRFM, scoring language models on a taxonomy of real clinical tasks.

  Not applicable (entity) · unknown · unknown resource · en

• MedMNIST

  Creator not supplied; see original source.

  MedMNIST site: standardised 2D and 3D biomedical image classification datasets for lightweight benchmarking.

  Not applicable (entity) · unknown · unknown resource · en

• MedQA

  Creator not supplied; see original source.

  MedQA GitHub repo: data and baselines for open-domain medical exam question answering.

  Not applicable (entity) · unknown · unknown resource · en

• Melting Pot

  Creator not supplied; see original source.

  DeepMind Melting Pot GitHub repo: test scenarios for generalisation in multi-agent reinforcement learning.

  Not applicable (entity) · unknown · unknown resource · en

• MERA-Evaluation/MERA

  Creator not supplied; see original source.

  MERA GitHub repo: Russian-language instruction benchmark with open task definitions and an evaluation harness.

  Not applicable (entity) · unknown · unknown resource · ru

• MERA (Alliance in AI)

  Creator not supplied; see original source.

  MERA site in Russian: open benchmark evaluating foundation models on Russian-language tasks, with leaderboards.

  Not applicable (entity) · unknown · unknown resource · ru

• MERA Code

  Creator not supplied; see original source.

  MERA Code GitHub repo: a benchmark for evaluating code generation by LLMs on tasks described in Russian.

  Not applicable (entity) · unknown · unknown resource · ru

• MERA Code / SWE-MERA

  Creator not supplied; see original source.

  MERA Code and SWE-MERA: benchmarks of code generation and software-engineering agents for Russian-language tasks.

  Not applicable (entity) · unknown · unknown resource · ru

• MERA Industrial

  Creator not supplied; see original source.

  MERA Industrial: benchmark of foundation models on industry-specific Russian-language tasks.

  Not applicable (entity) · unknown · unknown resource · ru

• MERA leaderboard and ecosystem

  Creator not supplied; see original source.

  MERA portal: Russian-language benchmark leaderboard for foundation models, with methodology and task categories.

  Not applicable (entity) · unknown · unknown resource · ru

Plain-text Markdown copy