Modality Evals Image Video Speech

87 resources · page 2 of 3

• ImgEdit

  Creator not supplied; see original source.

  ImgEdit GitHub repo from PKU: image-editing dataset and benchmark for evaluating instruction-based editing models.

  Not applicable (entity) · unknown · unknown resource · en

• KITTI Vision Benchmark Suite

  Creator not supplied; see original source.

  KITTI Vision Benchmark Suite: real driving data and leaderboards for stereo, optical flow, odometry and object detection.

  Not applicable (entity) · unknown · unknown resource · en

• Arena text-to-image leaderboard

  Creator not supplied; see original source.

  Arena (formerly LMArena) text-to-image leaderboard: human-preference rankings of image generation models.

  Not applicable (entity) · unknown · unknown resource · en

• Arena text-to-video leaderboard

  Creator not supplied; see original source.

  Arena (formerly LMArena) text-to-video leaderboard: human-preference rankings of video generation models.

  Not applicable (entity) · unknown · unknown resource · en

• LMMS-Eval

  Creator not supplied; see original source.

  LMMS-Eval GitHub repo: evaluation toolkit for multimodal models across text, image, video and audio tasks.

  Not applicable (entity) · unknown · unknown resource · en

• Probing the limitations of multimodal language models for chemistry and materials research (MaCBench)

  Creator not supplied; see original source.

  MaCBench paper probing multimodal language models on chemistry and materials-science tasks such as reading spectra and lab images.

  Not applicable (entity) · unknown · unknown resource · en

• MathVision

  Creator not supplied; see original source.

  MATH-Vision project page: competition math problems with visual contexts for evaluating multimodal reasoning.

  Not applicable (entity) · unknown · unknown resource · en

• MathVista

  Creator not supplied; see original source.

  MathVista project page: benchmark of math reasoning in visual contexts such as charts, diagrams and figures.

  Not applicable (entity) · unknown · unknown resource · en

• MERA Multimodal

  Creator not supplied; see original source.

  MERA Multimodal GitHub repo: open benchmark for evaluating multimodal models in Russian, with task definitions and code.

  Not applicable (entity) · unknown · unknown resource · ru

• MMAU

  Creator not supplied; see original source.

  MMAU project page: audio understanding and reasoning benchmark over speech, music and environmental sounds.

  Not applicable (entity) · unknown · unknown resource · en

• MMAU-Pro

  Creator not supplied; see original source.

  MMAU-Pro GitHub repo: benchmark for audio understanding and reasoning in audio-language models.

  Not applicable (entity) · unknown · unknown resource · en

• MMBench

  Creator not supplied; see original source.

  OpenCompass MMBench GitHub repo: multimodal evaluation benchmark in multiple languages.

  Not applicable (entity) · unknown · unknown resource · zh

• MMMU

  Creator not supplied; see original source.

  MMMU project site: college-level multimodal reasoning benchmark across disciplines, including MMMU-Pro and leaderboards.

  Not applicable (entity) · unknown · unknown resource · en

• MTEB Leaderboard

  Creator not supplied; see original source.

  MTEB leaderboard Space: embedding model scores across many retrieval, clustering and classification tasks.

  Not applicable (entity) · unknown · unknown resource · mul

• MuChoMusic

  Creator not supplied; see original source.

  MuChoMusic GitHub repo: benchmark for music understanding in audio language models.

  Not applicable (entity) · unknown · unknown resource · en

• MusTBench

  Creator not supplied; see original source.

  Sony MusTBench GitHub repo: benchmark for temporal grounding of music in large audio-language models.

  Not applicable (entity) · unknown · unknown resource · en

• MVBench (OpenGVLab/Ask-Anything)

  Creator not supplied; see original source.

  OpenGVLab Ask-Anything GitHub repo: VideoChat models and the MVBench video understanding benchmark.

  Not applicable (entity) · unknown · unknown resource · en

• nuScenes

  Creator not supplied; see original source.

  nuScenes site: autonomous-driving dataset with camera, lidar and radar data plus detection, tracking and prediction benchmarks.

  Not applicable (entity) · unknown · unknown resource · en

• ObjectNet

  Creator not supplied; see original source.

  ObjectNet site: real-world object recognition test set with controlled backgrounds, rotations and viewpoints.

  Not applicable (entity) · unknown · unknown resource · en

• OCRBench

  Creator not supplied; see original source.

  MultimodalOCR GitHub repo: OCRBench evaluation of text recognition in large multimodal models.

  Not applicable (entity) · unknown · unknown resource · en

• OmniDocBench

  Creator not supplied; see original source.

  OmniDocBench GitHub repo: benchmark for document parsing covering layout, tables, formulas and OCR across document types.

  Not applicable (entity) · unknown · unknown resource · zh

• Open ASR Leaderboard

  Creator not supplied; see original source.

  Hugging Face Open ASR Leaderboard: word error rate and speed of speech recognition models on public datasets.

  Not applicable (entity) · unknown · unknown resource · en|mul

• Open VLM Leaderboard (OpenCompass)

  Creator not supplied; see original source.

  OpenCompass Open VLM Leaderboard Space: vision-language model results aggregated from VLMEvalKit benchmarks.

  Not applicable (entity) · unknown · unknown resource · en|zh

• OpenCompass

  Creator not supplied; see original source.

  OpenCompass GitHub repo: LLM evaluation platform with configs for many datasets, inference, judging and leaderboards.

  Not applicable (entity) · unknown · unknown resource · zh

• OpenCompass CompassRank: Multimodal Leaderboard

  Creator not supplied; see original source.

  OpenCompass CompassRank multimodal leaderboard: scores of vision-language models across benchmarks.

  Not applicable (entity) · unknown · unknown resource · zh

• P3D-Bench

  Creator not supplied; see original source.

  P3D-Bench GitHub repo: benchmark of MLLMs on parametric 3D generation and structural reasoning.

  Not applicable (entity) · unknown · unknown resource · en

• PhysBench

  Creator not supplied; see original source.

  PhysBench project page: benchmark of physical-world questions testing vision-language models on objects, motion and interactions.

  Not applicable (entity) · unknown · unknown resource · en

• POPE

  Creator not supplied; see original source.

  POPE GitHub repo (EMNLP 2023): polling-based evaluation of object hallucination in vision-language models.

  Not applicable (entity) · unknown · unknown resource · en

• Rapidata Image-to-3D

  Creator not supplied; see original source.

  benchmark.ai Image-to-3D leaderboard from Rapidata ranking 3D generation models by pairwise human preference votes.

  Not applicable (entity) · unknown · unknown resource · en

• ScanNet benchmark

  Creator not supplied; see original source.

  ScanNet GitHub repo: RGB-D video dataset of 1,500+ indoor scans with 3D annotations and benchmark.

  Not applicable (entity) · unknown · unknown resource · en

• SEED-TTS-EVAL

  Creator not supplied; see original source.

  ByteDance seed-tts-eval GitHub repo: objective test set and scripts for evaluating TTS models.

  Not applicable (entity) · unknown · unknown resource · en

• SongBench

  Creator not supplied; see original source.

  Tencent SongBench GitHub repo: benchmark evaluating AI-generated songs across multiple musical dimensions.

  Not applicable (entity) · unknown · unknown resource · en

Plain-text Markdown copy