Evaluation Benchmarks
513 resources · page 7 of 17
LIBERO-Plus
by Senyu Fei et al. (Tongji University)
LIBERO-Plus paper: a robustness analysis of vision-language-action models using systematically perturbed LIBERO manipulation tasks.
LIBERO-PRO
by Xueyang Zhou et al. (Huazhong University of Science and Technology)
LIBERO-PRO paper testing vision-language-action models under perturbed objects, instructions and environments to check if LIBERO scores generalise.
LiveBench
by LiveBench team (Abacus.AI, NYU, NVIDIA and others)
LiveBench 2.0: regularly refreshed, contamination-limited benchmark of long-horizon real-work tasks.
LiveCodeBench
LiveCodeBench leaderboard: code generation results on problems collected from recent programming contests.
LiveMedBench
by alphaXiv
LiveMedBench paper page: a continuously refreshed medical LLM benchmark built from real clinical cases with automated rubric scoring.
LLM-jp Eval
by llm-jp
llm-jp-eval GitHub repo: tool for automatically evaluating Japanese LLMs across multiple datasets.
Arena text-to-image leaderboard
Arena (formerly LMArena) text-to-image leaderboard: human-preference rankings of image generation models.
Arena text-to-video leaderboard
Arena (formerly LMArena) text-to-video leaderboard: human-preference rankings of video generation models.
Arena Leaderboard (formerly LMArena)
Arena (formerly LMArena) leaderboards: human-preference rankings across text, webdev, vision, image, video and search arenas.
LMMS-Eval
by LMMs-Lab
LMMS-Eval GitHub repo: evaluation toolkit for multimodal models across text, image, video and audio tasks.
LongBench
by THUKEG
LongBench GitHub repo from THUDM: bilingual long-context benchmarks LongBench v1 and v2 with datasets and evaluation code.
Probing the limitations of multimodal language models for chemistry and materials research (MaCBench)
by Nawaf Alampara et al. (Friedrich Schiller University Jena)
MaCBench paper probing multimodal language models on chemistry and materials-science tasks such as reading spectra and lab images.
ManiSkill
by Hillbot and UC San Diego (ManiSkill)
ManiSkill site: GPU-accelerated simulation environments for training and evaluating robot manipulation policies at scale.
MASK (Scale Labs leaderboard)
by Scale AI
Scale Labs MASK leaderboard: benchmark by Scale AI and CAIS measuring honesty separately from accuracy.
Matbench
Materials Project Matbench GitHub repo: benchmark tasks for materials property prediction.
Matbench Discovery
Matbench Discovery GitHub repo: evaluation framework for ML models in materials stability discovery.
MATH
MATH dataset GitHub repo: competition mathematics problems with step-by-step solutions for evaluating reasoning.
MathArena
by MathArena team (SRI Lab, ETH Zurich)
MathArena: evaluation of LLMs on newly released math competitions and arXiv-derived problems to avoid contamination.
MathVision
MATH-Vision project page: competition math problems with visual contexts for evaluating multimodal reasoning.
MathVista
by mathvista
MathVista project page: benchmark of math reasoning in visual contexts such as charts, diagrams and figures.
MedBench
by Shanghai AI Laboratory (OpenCompass)
OpenCompass MedBench portal: Chinese medical benchmark platform for evaluating LLMs on clinical knowledge and tasks.
MedHELM
Stanford CRFM MedHELM: evaluation of language models on clinician-defined medical tasks beyond exam-style questions.
MedHELM (Stanford CRFM)
Latest MedHELM results page from Stanford CRFM, scoring language models on a taxonomy of real clinical tasks.
MedMNIST
by MedMNIST team (Shanghai Jiao Tong University)
MedMNIST site: standardised 2D and 3D biomedical image classification datasets for lightweight benchmarking.
MedQA
by Di Jin
MedQA GitHub repo: data and baselines for open-domain medical exam question answering.
Melting Pot
DeepMind Melting Pot GitHub repo: test scenarios for generalisation in multi-agent reinforcement learning.
MERA-Evaluation/MERA
MERA GitHub repo: Russian-language instruction benchmark with open task definitions and an evaluation harness.
MERA (Alliance in AI)
MERA site in Russian: open benchmark evaluating foundation models on Russian-language tasks, with leaderboards.
MERA Code
MERA Code GitHub repo: a benchmark for evaluating code generation by LLMs on tasks described in Russian.
MERA Code / SWE-MERA
MERA Code and SWE-MERA: benchmarks of code generation and software-engineering agents for Russian-language tasks.
MERA Industrial
MERA Industrial: benchmark of foundation models on industry-specific Russian-language tasks.
MERA leaderboard and ecosystem
MERA portal: Russian-language benchmark leaderboard for foundation models, with methodology and task categories.