datasets: 36 curated resources
datasets
36 resources · page 1 of 2
AI9Stars model and dataset registry
by AI9Stars
AI9Stars Hugging Face organisation: compact G9 language models and datasets.
Argilla
by Argilla
Argilla GitHub repo: collaboration tool for building datasets with human feedback and annotation.
Common Crawl
by Common Crawl
Common Crawl: nonprofit maintaining an open repository of web crawl data, a common source of model training data.
DatologyAI
by DatologyAI
DatologyAI: company offering automated curation of model training data.
Docling (IBM)
IBM Docling GitHub repo: converts PDFs and other documents into structured formats for generative AI.
FathomNet
by FathomNet
FathomNet: labelled database of underwater images with ML tools for analysing ocean visual data.
FineWeb
by FineData
Hugging Face FineWeb dataset: about 15 trillion tokens of filtered English web text for LLM pretraining.
Global Fishing Watch
Global Fishing Watch: uses machine learning on satellite and vessel data to map human activity at sea.
HF datasets
by Hugging Face
Hugging Face Datasets hub: searchable catalogue of public machine-learning datasets.
LightOn HF
by LightOn AI
LightOn Hugging Face organisation: retrieval, OCR and language models.
Marker (Datalab)
by Datalab
Datalab Marker GitHub repo: converts PDF documents to Markdown and JSON.
MinerU (OpenDataLab)
by OpenDataLab
OpenDataLab MinerU GitHub repo: converts PDFs and Office documents into Markdown and JSON.
Multimodal Universe
by Multimodal Universe Collaboration
Multimodal Universe GitHub repo: large dataset of astronomical images, spectra and light curves for training ML models.
MVTec Anomaly Detection dataset
MVTec AD dataset page: over 5,000 high-resolution images for industrial anomaly detection and localisation.
Vera C. Rubin Observatory
Vera C. Rubin Observatory: NSF–DOE telescope running the LSST survey, whose alert stream requires ML triage.
xView2 / xBD
by Defense Innovation Unit (xView2)
xView2 challenge site: xBD satellite imagery dataset for assessing building damage after natural disasters.
Yambda dataset
by Yandex
Yandex Yambda dataset on Hugging Face: large music-streaming interaction dataset for recommender-system research.
How to Shape a Strong AI Testing and Evaluation Ecosystem (Scale AI)
by Scale AI
Scale AI policy post on building an AI testing and evaluation ecosystem for the United States.
Screen annotators with Exams for higher-quality data (Toloka)
by Toloka
Toloka post on screening data annotators with training and exams to improve label quality.
Argilla documentation
Argilla documentation: open-source tool for collaborative data annotation and dataset curation for LLMs.
Hugging Face Hub documentation
by Hugging Face
Hugging Face Hub documentation: repositories, model cards, datasets, Spaces and gated access.
Nomic blog
by Nomic
Nomic blog with posts on its embedding models, data mapping tools and company updates.
Scale AI blog
by Scale AI
Scale AI blog with company updates and articles on data labelling, evaluation and AI research.
Get started with Copilot in Excel
by Microsoft
Microsoft Support guide to Copilot in Excel for analysing data, generating formulas and creating charts.
Benchmarks are broken (Surge AI)
by Surge AI
Surge AI post arguing academic benchmarks poorly reflect model quality, with examples of benchmark flaws.
HANDBOOK.md Benchmark (Surge AI)
by Surge AI
Surge AI post on HANDBOOK.md, a benchmark testing whether agents follow long company policy handbooks.
HellaSwag or HellaBad? (Surge AI)
by Surge AI
Surge AI audit reporting errors in 36% of HellaSwag benchmark rows, with examples.
How Shopify and Toloka built a ground-truth flywheel
by Toloka
Toloka case study on building a human-labelled ground-truth loop with Shopify to keep skill agents accurate.
Ai2 model cards and datasets
by Ai2
Ai2's Hugging Face organisation page listing its model checkpoints, model cards and datasets, including OLMo.
LLM-jp corpora (GitLab)
by LLM-jp (National Institute of Informatics)
LLM-jp datasets on the NII GitLab: open Japanese pretraining corpora.
RL environments for LLM agents: Design, rewards, and validation (Snorkel)
by Snorkel AI
Snorkel AI post on designing RL environments for LLM agents: state, actions, rewards and validation.
Partial Credit in RL: When It Opposes Completion (Snorkel)
by Snorkel AI
Snorkel AI post on how partial-credit rewards in reinforcement learning can work against task completion.