Data & datasets
Data & datasets
- data engineering & RAG (52)
- datasets (36)
88 resources · page 1 of 3
AI9Stars model and dataset registry
by AI9Stars
AI9Stars Hugging Face organisation: compact G9 language models and datasets.
AMLSim
by International Business Machines
IBM AMLSim GitHub repo: multi-agent simulator generating synthetic bank transaction data for anti-money-laundering research.
Argilla
by Argilla
Argilla GitHub repo: collaboration tool for building datasets with human feedback and annotation.
ColBERT
Stanford ColBERT GitHub repo: late-interaction neural retrieval model and library.
Common Crawl
by Common Crawl
Common Crawl: nonprofit maintaining an open repository of web crawl data, a common source of model training data.
data engineering / RAG pipelines
by InfiniFlow
InfiniFlow RAGFlow GitHub repo: open-source retrieval-augmented generation engine with document parsing.
datatrove
by Hugging Face
Hugging Face datatrove GitHub repo: large-scale text data processing library, used to build FineWeb.
DatologyAI
by DatologyAI
DatologyAI: company offering automated curation of model training data.
Docling (IBM)
IBM Docling GitHub repo: converts PDFs and other documents into structured formats for generative AI.
FathomNet
by FathomNet
FathomNet: labelled database of underwater images with ML tools for analysing ocean visual data.
FineWeb
by FineData
Hugging Face FineWeb dataset: about 15 trillion tokens of filtered English web text for LLM pretraining.
Global Fishing Watch
Global Fishing Watch: uses machine learning on satellite and vessel data to map human activity at sea.
HF datasets
by Hugging Face
Hugging Face Datasets hub: searchable catalogue of public machine-learning datasets.
LightOn HF
by LightOn AI
LightOn Hugging Face organisation: retrieval, OCR and language models.
OtterTune wiki: An End-To-End Tuning Example
OtterTune wiki page on GitHub (CMU): end-to-end example of machine-learning database configuration tuning.
Marker (Datalab)
by Datalab
Datalab Marker GitHub repo: converts PDF documents to Markdown and JSON.
MinerU (OpenDataLab)
by OpenDataLab
OpenDataLab MinerU GitHub repo: converts PDFs and Office documents into Markdown and JSON.
Multimodal Universe
by Multimodal Universe Collaboration
Multimodal Universe GitHub repo: large dataset of astronomical images, spectra and light curves for training ML models.
MVTec Anomaly Detection dataset
MVTec AD dataset page: over 5,000 high-resolution images for industrial anomaly detection and localisation.
Qwen-Agent
by Qwen
QwenLM Qwen-Agent GitHub repo: agent framework with function calling, MCP and code interpreter for Qwen models.
RAG_Techniques
by Nir Diamant
RAG_Techniques GitHub repo (Nir Diamant): notebooks demonstrating many retrieval-augmented generation techniques.
circuit_training: Ariane RISC-V example
Ariane documentation in Google's circuit_training repo: RL chip floorplanning on the Ariane RISC-V design.
Vera C. Rubin Observatory
Vera C. Rubin Observatory: NSF–DOE telescope running the LSST survey, whose alert stream requires ML triage.
xView2 / xBD
by Defense Innovation Unit (xView2)
xView2 challenge site: xBD satellite imagery dataset for assessing building damage after natural disasters.
Yambda dataset
by Yandex
Yandex Yambda dataset on Hugging Face: large music-streaming interaction dataset for recommender-system research.
Postman: Write scripts to test API response data
by Postman
Postman documentation on writing test scripts that assert API response data.
DataTalksClub LLM Zoomcamp
DataTalksClub LLM Zoomcamp GitHub repo: free course on building LLM and RAG applications.
Summarize incidents with Microsoft Copilot in Microsoft Defender
by Microsoft
Microsoft Learn page on generating incident summaries with Microsoft Copilot in Microsoft Defender.
Getting started with Pulumi Neo
by Pulumi
Pulumi documentation on setting up Pulumi Neo, an AI agent for infrastructure-as-code tasks.
How to Shape a Strong AI Testing and Evaluation Ecosystem (Scale AI)
by Scale AI
Scale AI policy post on building an AI testing and evaluation ecosystem for the United States.
Screen annotators with Exams for higher-quality data (Toloka)
by Toloka
Toloka post on screening data annotators with training and exams to improve label quality.
Using GitHub Copilot code review
by GitHub
GitHub documentation on requesting pull-request code reviews from GitHub Copilot.