modality evals (image/video/speech): 87 curated resources
modality evals (image/video/speech)
87 resources · page 2 of 3
ImgEdit
ImgEdit GitHub repo from PKU: image-editing dataset and benchmark for evaluating instruction-based editing models.
KITTI Vision Benchmark Suite
by Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI)
KITTI Vision Benchmark Suite: real driving data and leaderboards for stereo, optical flow, odometry and object detection.
Arena text-to-image leaderboard
Arena (formerly LMArena) text-to-image leaderboard: human-preference rankings of image generation models.
Arena text-to-video leaderboard
Arena (formerly LMArena) text-to-video leaderboard: human-preference rankings of video generation models.
LMMS-Eval
by LMMs-Lab
LMMS-Eval GitHub repo: evaluation toolkit for multimodal models across text, image, video and audio tasks.
Probing the limitations of multimodal language models for chemistry and materials research (MaCBench)
by Nawaf Alampara et al. (Friedrich Schiller University Jena)
MaCBench paper probing multimodal language models on chemistry and materials-science tasks such as reading spectra and lab images.
MathVision
MATH-Vision project page: competition math problems with visual contexts for evaluating multimodal reasoning.
MathVista
by mathvista
MathVista project page: benchmark of math reasoning in visual contexts such as charts, diagrams and figures.
MERA Multimodal
MERA Multimodal GitHub repo: open benchmark for evaluating multimodal models in Russian, with task definitions and code.
MMAU
by mmaubench
MMAU project page: audio understanding and reasoning benchmark over speech, music and environmental sounds.
MMAU-Pro
by Sonal Kumar
MMAU-Pro GitHub repo: benchmark for audio understanding and reasoning in audio-language models.
MMBench
by OpenCompass
OpenCompass MMBench GitHub repo: multimodal evaluation benchmark in multiple languages.
MMMU
by MMMU
MMMU project site: college-level multimodal reasoning benchmark across disciplines, including MMMU-Pro and leaderboards.
MTEB Leaderboard
by Massive Text Embedding Benchmark
MTEB leaderboard Space: embedding model scores across many retrieval, clustering and classification tasks.
MuChoMusic
by MuLaB
MuChoMusic GitHub repo: benchmark for music understanding in audio language models.
MusTBench
by Sony
Sony MusTBench GitHub repo: benchmark for temporal grounding of music in large audio-language models.
MVBench (OpenGVLab/Ask-Anything)
by OpenGVLab
OpenGVLab Ask-Anything GitHub repo: VideoChat models and the MVBench video understanding benchmark.
nuScenes
nuScenes site: autonomous-driving dataset with camera, lidar and radar data plus detection, tracking and prediction benchmarks.
ObjectNet
ObjectNet site: real-world object recognition test set with controlled backgrounds, rotations and viewpoints.
OCRBench
by Yuliang Liu
MultimodalOCR GitHub repo: OCRBench evaluation of text recognition in large multimodal models.
OmniDocBench
by OpenDataLab
OmniDocBench GitHub repo: benchmark for document parsing covering layout, tables, formulas and OCR across document types.
Open ASR Leaderboard
Hugging Face Open ASR Leaderboard: word error rate and speed of speech recognition models on public datasets.
Open VLM Leaderboard (OpenCompass)
by OpenCompass
OpenCompass Open VLM Leaderboard Space: vision-language model results aggregated from VLMEvalKit benchmarks.
OpenCompass
by OpenCompass
OpenCompass GitHub repo: LLM evaluation platform with configs for many datasets, inference, judging and leaderboards.
OpenCompass CompassRank: Multimodal Leaderboard
by Shanghai AI Laboratory (OpenCompass)
OpenCompass CompassRank multimodal leaderboard: scores of vision-language models across benchmarks.
P3D-Bench
by SpatiaOS
P3D-Bench GitHub repo: benchmark of MLLMs on parametric 3D generation and structural reasoning.
PhysBench
by PhysBench
PhysBench project page: benchmark of physical-world questions testing vision-language models on objects, motion and interactions.
POPE
by Yifan Li
POPE GitHub repo (EMNLP 2023): polling-based evaluation of object hallucination in vision-language models.
Rapidata Image-to-3D
by Rapidata
benchmark.ai Image-to-3D leaderboard from Rapidata ranking 3D generation models by pairwise human preference votes.
ScanNet benchmark
by ScanNet
ScanNet GitHub repo: RGB-D video dataset of 1,500+ indoor scans with 3D annotations and benchmark.
SEED-TTS-EVAL
ByteDance seed-tts-eval GitHub repo: objective test set and scripts for evaluating TTS models.
SongBench
by Tencent
Tencent SongBench GitHub repo: benchmark evaluating AI-generated songs across multiple musical dimensions.