Data & datasets for expert learners
31 resources · page 1 of 1
AI9Stars model and dataset registry
by AI9Stars
AI9Stars Hugging Face organisation: compact G9 language models and datasets.
AMLSim
by International Business Machines
IBM AMLSim GitHub repo: multi-agent simulator generating synthetic bank transaction data for anti-money-laundering research.
Argilla
by Argilla
Argilla GitHub repo: collaboration tool for building datasets with human feedback and annotation.
ColBERT
Stanford ColBERT GitHub repo: late-interaction neural retrieval model and library.
Common Crawl
by Common Crawl
Common Crawl: nonprofit maintaining an open repository of web crawl data, a common source of model training data.
data engineering / RAG pipelines
by InfiniFlow
InfiniFlow RAGFlow GitHub repo: open-source retrieval-augmented generation engine with document parsing.
datatrove
by Hugging Face
Hugging Face datatrove GitHub repo: large-scale text data processing library, used to build FineWeb.
DatologyAI
by DatologyAI
DatologyAI: company offering automated curation of model training data.
Docling (IBM)
IBM Docling GitHub repo: converts PDFs and other documents into structured formats for generative AI.
FathomNet
by FathomNet
FathomNet: labelled database of underwater images with ML tools for analysing ocean visual data.
FineWeb
by FineData
Hugging Face FineWeb dataset: about 15 trillion tokens of filtered English web text for LLM pretraining.
Global Fishing Watch
Global Fishing Watch: uses machine learning on satellite and vessel data to map human activity at sea.
HF datasets
by Hugging Face
Hugging Face Datasets hub: searchable catalogue of public machine-learning datasets.
LightOn HF
by LightOn AI
LightOn Hugging Face organisation: retrieval, OCR and language models.
OtterTune wiki: An End-To-End Tuning Example
OtterTune wiki page on GitHub (CMU): end-to-end example of machine-learning database configuration tuning.
Marker (Datalab)
by Datalab
Datalab Marker GitHub repo: converts PDF documents to Markdown and JSON.
MinerU (OpenDataLab)
by OpenDataLab
OpenDataLab MinerU GitHub repo: converts PDFs and Office documents into Markdown and JSON.
Multimodal Universe
by Multimodal Universe Collaboration
Multimodal Universe GitHub repo: large dataset of astronomical images, spectra and light curves for training ML models.
MVTec Anomaly Detection dataset
MVTec AD dataset page: over 5,000 high-resolution images for industrial anomaly detection and localisation.
Qwen-Agent
by Qwen
QwenLM Qwen-Agent GitHub repo: agent framework with function calling, MCP and code interpreter for Qwen models.
RAG_Techniques
by Nir Diamant
RAG_Techniques GitHub repo (Nir Diamant): notebooks demonstrating many retrieval-augmented generation techniques.
circuit_training: Ariane RISC-V example
Ariane documentation in Google's circuit_training repo: RL chip floorplanning on the Ariane RISC-V design.
Vera C. Rubin Observatory
Vera C. Rubin Observatory: NSF–DOE telescope running the LSST survey, whose alert stream requires ML triage.
xView2 / xBD
by Defense Innovation Unit (xView2)
xView2 challenge site: xBD satellite imagery dataset for assessing building damage after natural disasters.
Yambda dataset
by Yandex
Yandex Yambda dataset on Hugging Face: large music-streaming interaction dataset for recommender-system research.
Synthetic Data in Investment Management (CFA Institute)
by CFA Institute Research and Policy Center
CFA Institute Research and Policy Center report (2025) on synthetic data in investment management.
DataComp / DCLM
by DataComp team (University of Washington and others)
DataComp site: data-centric benchmarks for building better training datasets for CLIP models and language models (DCLM).
DCLM paper
by Jeffrey Li et al. (University of Washington)
arXiv paper (2024) introducing DataComp-LM (DCLM), a testbed for curating language model training sets.
Microsoft GraphRAG
by Microsoft
Microsoft GraphRAG documentation: RAG approach that builds an LLM-generated knowledge graph for global summarisation queries.
Do people click on links in Google AI summaries? (Pew Research Center)
Pew Research short read on Google users clicking fewer links when an AI summary appears.
Yandex ШАД (School of Data Analysis)
by Yandex School of Data Analysis
Yandex School of Data Analysis: free two-year Russian programme in ML, data analysis and engineering.