Datasets

36 resources · page 1 of 2

• AI9Stars model and dataset registry

  Creator not supplied; see original source.

  AI9Stars Hugging Face organisation: compact G9 language models and datasets.

  Not applicable (entity) · unknown · unknown resource · en

• Argilla

  Creator not supplied; see original source.

  Argilla GitHub repo: collaboration tool for building datasets with human feedback and annotation.

  Not applicable (entity) · unknown · unknown resource · en

• Common Crawl

  Creator not supplied; see original source.

  Common Crawl: nonprofit maintaining an open repository of web crawl data, a common source of model training data.

  Not applicable (entity) · unknown · unknown resource · en

• DatologyAI

  Creator not supplied; see original source.

  DatologyAI: company offering automated curation of model training data.

  Not applicable (entity) · unknown · unknown resource · en

• Docling (IBM)

  Creator not supplied; see original source.

  IBM Docling GitHub repo: converts PDFs and other documents into structured formats for generative AI.

  Not applicable (entity) · unknown · unknown resource · en

• FathomNet

  Creator not supplied; see original source.

  FathomNet: labelled database of underwater images with ML tools for analysing ocean visual data.

  Not applicable (entity) · unknown · unknown resource · en

• FineWeb

  Creator not supplied; see original source.

  Hugging Face FineWeb dataset: about 15 trillion tokens of filtered English web text for LLM pretraining.

  Not applicable (entity) · unknown · unknown resource · en

• Global Fishing Watch

  Creator not supplied; see original source.

  Global Fishing Watch: uses machine learning on satellite and vessel data to map human activity at sea.

  Not applicable (entity) · unknown · unknown resource · en|es

• HF datasets

  Creator not supplied; see original source.

  Hugging Face Datasets hub: searchable catalogue of public machine-learning datasets.

  Not applicable (entity) · unknown · unknown resource · und

• LightOn HF

  Creator not supplied; see original source.

  LightOn Hugging Face organisation: retrieval, OCR and language models.

  Not applicable (entity) · unknown · unknown resource · und

• Marker (Datalab)

  Creator not supplied; see original source.

  Datalab Marker GitHub repo: converts PDF documents to Markdown and JSON.

  Not applicable (entity) · unknown · unknown resource · en

• MinerU (OpenDataLab)

  Creator not supplied; see original source.

  OpenDataLab MinerU GitHub repo: converts PDFs and Office documents into Markdown and JSON.

  Not applicable (entity) · unknown · unknown resource · en

• Multimodal Universe

  Creator not supplied; see original source.

  Multimodal Universe GitHub repo: large dataset of astronomical images, spectra and light curves for training ML models.

  Not applicable (entity) · unknown · unknown resource · en

• MVTec Anomaly Detection dataset

  Creator not supplied; see original source.

  MVTec AD dataset page: over 5,000 high-resolution images for industrial anomaly detection and localisation.

  Not applicable (entity) · unknown · unknown resource · en

• Vera C. Rubin Observatory

  Creator not supplied; see original source.

  Vera C. Rubin Observatory: NSF–DOE telescope running the LSST survey, whose alert stream requires ML triage.

  Not applicable (entity) · unknown · unknown resource · en|es

• xView2 / xBD

  Creator not supplied; see original source.

  xView2 challenge site: xBD satellite imagery dataset for assessing building damage after natural disasters.

  Not applicable (entity) · unknown · unknown resource · en

• Yambda dataset

  Creator not supplied; see original source.

  Yandex Yambda dataset on Hugging Face: large music-streaming interaction dataset for recommender-system research.

  Not applicable (entity) · unknown · unknown resource · en

• How to Shape a Strong AI Testing and Evaluation Ecosystem (Scale AI)

  Creator not supplied; see original source.

  Scale AI policy post on building an AI testing and evaluation ecosystem for the United States.

  Beginner · unknown · unknown resource · en

• Screen annotators with Exams for higher-quality data (Toloka)

  Creator not supplied; see original source.

  Toloka post on screening data annotators with training and exams to improve label quality.

  Beginner · unknown · unknown resource · en

• Argilla documentation

  Creator not supplied; see original source.

  Argilla documentation: open-source tool for collaborative data annotation and dataset curation for LLMs.

  Intermediate · unknown · unknown resource · en

• Hugging Face Hub documentation

  Creator not supplied; see original source.

  Hugging Face Hub documentation: repositories, model cards, datasets, Spaces and gated access.

  Intermediate · unknown · unknown resource · en

• Nomic blog

  Creator not supplied; see original source.

  Nomic blog with posts on its embedding models, data mapping tools and company updates.

  Intermediate · unknown · unknown resource · en

• Scale AI blog

  Creator not supplied; see original source.

  Scale AI blog with company updates and articles on data labelling, evaluation and AI research.

  Intermediate · unknown · unknown resource · en

• Get started with Copilot in Excel

  Creator not supplied; see original source.

  Microsoft Support guide to Copilot in Excel for analysing data, generating formulas and creating charts.

  Intermediate–Advanced · unknown · unknown resource · en

• Benchmarks are broken (Surge AI)

  Creator not supplied; see original source.

  Surge AI post arguing academic benchmarks poorly reflect model quality, with examples of benchmark flaws.

  Intermediate · unknown · unknown resource · en

• HANDBOOK.md Benchmark (Surge AI)

  Creator not supplied; see original source.

  Surge AI post on HANDBOOK.md, a benchmark testing whether agents follow long company policy handbooks.

  Intermediate · unknown · unknown resource · en

• HellaSwag or HellaBad? (Surge AI)

  Creator not supplied; see original source.

  Surge AI audit reporting errors in 36% of HellaSwag benchmark rows, with examples.

  Intermediate · unknown · unknown resource · en

• How Shopify and Toloka built a ground-truth flywheel

  Creator not supplied; see original source.

  Toloka case study on building a human-labelled ground-truth loop with Shopify to keep skill agents accurate.

  Intermediate · unknown · unknown resource · en

• Ai2 model cards and datasets

  Creator not supplied; see original source.

  Ai2's Hugging Face organisation page listing its model checkpoints, model cards and datasets, including OLMo.

  Advanced · unknown · unknown resource · en

• LLM-jp corpora (GitLab)

  Creator not supplied; see original source.

  LLM-jp datasets on the NII GitLab: open Japanese pretraining corpora.

  Advanced · unknown · unknown resource · ja

• RL environments for LLM agents: Design, rewards, and validation (Snorkel)

  Creator not supplied; see original source.

  Snorkel AI post on designing RL environments for LLM agents: state, actions, rewards and validation.

  Advanced · unknown · unknown resource · en

• Partial Credit in RL: When It Opposes Completion (Snorkel)

  Creator not supplied; see original source.

  Snorkel AI post on how partial-credit rewards in reinforcement learning can work against task completion.

  Advanced · unknown · unknown resource · en

Plain-text Markdown copy