Evaluation & benchmarks for intermediate learners
454 resources · page 12 of 15
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
by Xiang Deng et al. (Scale AI)
SWE-Bench Pro paper: long-horizon software-engineering tasks, including a held-out commercial split, for evaluating coding agents.
SWE-Lancer
by OpenAI
OpenAI SWE-Lancer GitHub repo: dataset and code for real freelance software engineering tasks.
SWE-MERA
SWE-MERA page: continuously refreshed software-engineering tasks built as a contamination-resistant alternative to SWE-bench.
SWE-rebench
SWE-rebench: continuously updated, decontaminated software-engineering benchmark with leaderboard.
T-Tech Ru Dialogue Benchmarks
by T-Tech
T-Tech collection on Hugging Face of benchmarks for evaluating Russian-language dialogue models.
T2I-CompBench
by Karine_H
T2I-CompBench GitHub repo: benchmark for compositional text-to-image generation such as attribute binding and spatial relations.
T3Bench
by T3Bench team
T3Bench site: benchmark for text-to-3D generation measuring quality and alignment with the prompt.
TabArena leaderboard
by TabArena
TabArena leaderboard Hugging Face Space: living benchmark of machine-learning models on tabular datasets.
τ-bench
τ-bench site: benchmarks of AI agents conversing with users, calling tools and following policies across domains.
tau2-bench
tau2-bench GitHub repo (Sierra): benchmark of tool-using agents interacting with users under domain policies.
TaxCalcBench (Column Tax)
by Column Tax
Column Tax post introducing TaxCalcBench, a benchmark of tax-return calculation cases with an analysis of where models fail.
TC260 national cybersecurity standards committee
by National Information Security Standardization Technical Committee (TC260)
TC260 official site: China's national cybersecurity standardisation committee, which issues AI security standards.
Temporal Graph Benchmark
by Complex Data Lab (McGill University and Mila)
Temporal Graph Benchmark documentation: datasets and tasks for evaluating machine learning on dynamic graphs.
Terminal-Bench
by Terminal-Bench team (Stanford University and Laude Institute)
Terminal-Bench: benchmark of AI agents completing tasks in sandboxed terminal environments.
Terminal-Bench 2.0 (Snorkel)
by Snorkel AI
Snorkel-hosted Terminal-Bench 2.0 leaderboard: 89 human-checked terminal tasks scored by resolution rate.
Thai LLM Leaderboard
Thai LLM Leaderboard Hugging Face Space: evaluation results of models across Thai benchmarks.
The Agent Company
by TheAgentCompany team (Carnegie Mellon University)
TheAgentCompany site: benchmark of simulated software-company tasks for evaluating LLM agents across business tools.
Therapeutics Data Commons
by Therapeutics Data Commons team (Harvard University)
Therapeutics Data Commons: datasets, tasks and leaderboards for machine learning across drug discovery and development.
TII Arabic LLM benchmark index
by Technology Innovation Institute
TII Arabic-LLM-Benchmarks GitHub repo: list of benchmarks for evaluating Arabic LLMs.
TMMLU+ (Taiwan)
by iKala
iKala TMMLU+ dataset: Traditional Chinese multiple-choice questions across 66 subjects with a Taiwan-specific context.
ToolBench
by OpenBMB
OpenBMB ToolBench GitHub repo (ICLR 2024): platform and benchmark for training and evaluating LLM tool use.
Top 3D AI Arena
by Top 3D AI
Top 3D AI leaderboard ranking 3D generation tools by Elo from blind community votes.
Российский энтузиаст создал бесплатный ИИ-бенчмарк (Tproger)
by Tproger
Tproger report in Russian on LLM Arena, a free blind pairwise benchmark for Russian-language models.
TruthfulQA
by sylinrl
TruthfulQA GitHub repo: benchmark testing whether language models repeat common human misconceptions, with data and evaluation code.
TTS Arena V2 (Hugging Face Space)
by TTS AGI
TTS Arena V2 Space: community arena voting between anonymous text-to-speech models.
UK AI Security Institute
UK AI Security Institute: government body publishing frontier AI evaluations and the Inspect evaluation framework.
UK AISI Inspect Evals
by Department of Business, Energy and Industrial Strategy
UK AISI Inspect Evals: registry of evaluations implemented in Inspect, including cybersecurity, safety and agent benchmarks.
Announcing Inspect Evals (UK AISI)
UK AISI announcement of Inspect Evals, an open-source collection of LLM evaluations.
Ukrainian LLM Leaderboard (lang-uk)
by Lang UK
lang-uk Ukrainian LLM Leaderboard Hugging Face Space: dashboard of model results on Ukrainian tasks.
Vals.ai
by Vals AI
Vals AI benchmarks: independently run leaderboards on coding, finance, legal and science tasks with evaluation costs.
Vals AI MMMU Pro Leaderboard
by Vals AI
Vals AI MMMU-Pro page: archived multimodal reasoning results and methodology; no longer run on new models.
Vals AI's Latest Benchmark Finds Legal and General AI Now Outperform Lawyers in Legal Research (LawNext)
by LawSites
LawNext article on Vals AI's legal research benchmark comparing legal AI tools and general LLMs with a lawyer baseline.