--- id: "/data-datasets/datasets/" title: "datasets: 36 curated resources" url: "https://skillissue.dev/data-datasets/datasets/" --- # datasets: 36 curated resources datasets [Beginner](/data-datasets/beginner/index.md) [Intermediate](/data-datasets/intermediate/index.md) [Advanced](/data-datasets/advanced/index.md) [Expert](/data-datasets/expert/index.md) 36 resources · page 1 of 2 - ### [AI9Stars model and dataset registry](/context-engineering/retrieval-and-rag/ai9stars-model-and-dataset-registry/index.md) by [AI9Stars](https://huggingface.co/ai9stars) AI9Stars Hugging Face organisation: compact G9 language models and datasets. entity org International; domicile unverified en - ### [Argilla](/context-engineering/retrieval-and-rag/argilla-repo/index.md) by [Argilla](https://argilla.io) Argilla GitHub repo: collaboration tool for building datasets with human feedback and annotation. entity repo Spain/EU en - ### [Common Crawl](/context-engineering/retrieval-and-rag/common-crawl/index.md) by [Common Crawl](https://commoncrawl.org/) Common Crawl: nonprofit maintaining an open repository of web crawl data, a common source of model training data. entity lab US en - ### [DatologyAI](/context-engineering/retrieval-and-rag/datologyai/index.md) by [DatologyAI](https://www.datologyai.com/) DatologyAI: company offering automated curation of model training data. entity lab US en - ### [Docling (IBM)](/context-engineering/retrieval-and-rag/docling-ibm/index.md) by [Docling Project](https://github.com/docling-project) IBM Docling GitHub repo: converts PDFs and other documents into structured formats for generative AI. entity repo US en - ### [FathomNet](/context-engineering/retrieval-and-rag/fathomnet/index.md) by [FathomNet](https://fathomnet.org/) FathomNet: labelled database of underwater images with ML tools for analysing ocean visual data. entity dataset US (MBARI) en - ### [FineWeb](/context-engineering/retrieval-and-rag/fineweb/index.md) by [FineData](https://huggingface.co/HuggingFaceFW) Hugging Face FineWeb dataset: about 15 trillion tokens of filtered English web text for LLM pretraining. entity dataset US en - ### [Global Fishing Watch](/context-engineering/retrieval-and-rag/global-fishing-watch/index.md) by [Global Fishing Watch](https://globalfishingwatch.org/) Global Fishing Watch: uses machine learning on satellite and vessel data to map human activity at sea. entity org Global en|es - ### [HF datasets](/context-engineering/retrieval-and-rag/hf-datasets/index.md) by [Hugging Face](https://huggingface.co/) Hugging Face Datasets hub: searchable catalogue of public machine-learning datasets. entity lab US en - ### [LightOn HF](/context-engineering/retrieval-and-rag/lighton-hf/index.md) by [LightOn AI](https://huggingface.co/lightonai) LightOn Hugging Face organisation: retrieval, OCR and language models. entity lab EU-FR en - ### [Marker (Datalab)](/context-engineering/retrieval-and-rag/marker-datalab/index.md) by [Datalab](https://www.datalab.to) Datalab Marker GitHub repo: converts PDF documents to Markdown and JSON. entity repo US en - ### [MinerU (OpenDataLab)](/context-engineering/retrieval-and-rag/mineru-opendatalab/index.md) by [OpenDataLab](https://opendatalab.org.cn) OpenDataLab MinerU GitHub repo: converts PDFs and Office documents into Markdown and JSON. entity repo CN en - ### [Multimodal Universe](/context-engineering/retrieval-and-rag/multimodal-universe/index.md) by [Multimodal Universe Collaboration](https://github.com/MultimodalUniverse) Multimodal Universe GitHub repo: large dataset of astronomical images, spectra and light curves for training ML models. entity dataset Global en - ### [MVTec Anomaly Detection dataset](/context-engineering/retrieval-and-rag/mvtec-anomaly-detection-dataset/index.md) by [MVTec Software](https://www.mvtec.com/) MVTec AD dataset page: over 5,000 high-resolution images for industrial anomaly detection and localisation. entity dataset EU (DE) en - ### [Vera C. Rubin Observatory](/context-engineering/retrieval-and-rag/vera-c-rubin-observatory/index.md) by [Vera C. Rubin Observatory](https://rubinobservatory.org/) Vera C. Rubin Observatory: NSF–DOE telescope running the LSST survey, whose alert stream requires ML triage. entity org US/CL en|es - ### [xView2 / xBD](/context-engineering/retrieval-and-rag/xview2-xbd/index.md) by [Defense Innovation Unit (xView2)](https://www.diu.mil/) xView2 challenge site: xBD satellite imagery dataset for assessing building damage after natural disasters. entity benchmark US (DIU) en - ### [Yambda dataset](/context-engineering/retrieval-and-rag/yambda-dataset/index.md) by [Yandex](https://huggingface.co/yandex) Yandex Yambda dataset on Hugging Face: large music-streaming interaction dataset for recommender-system research. entity dataset Russia en - ### [How to Shape a Strong AI Testing and Evaluation Ecosystem (Scale AI)](/context-engineering/retrieval-and-rag/scale-ai-building-trust-at-the-frontier/index.md) by [Scale AI](https://scale.com/) Scale AI policy post on building an AI testing and evaluation ecosystem for the United States. Beginner learning_resource blog US en - ### [Screen annotators with Exams for higher-quality data (Toloka)](/context-engineering/retrieval-and-rag/toloka-screen-annotators-with-exams/index.md) by [Toloka](https://toloka.ai/) Toloka post on screening data annotators with training and exams to improve label quality. Beginner learning_resource guide Netherlands (RU-origin) en - ### [Argilla documentation](/context-engineering/retrieval-and-rag/argilla-documentation/index.md) by [Argilla (Hugging Face)](https://argilla.io/) Argilla documentation: open-source tool for collaborative data annotation and dataset curation for LLMs. Intermediate learning_resource docs Spain/global en - ### [Hugging Face Hub documentation](/context-engineering/retrieval-and-rag/hugging-face-hub-documentation/index.md) by [Hugging Face](https://huggingface.co/) Hugging Face Hub documentation: repositories, model cards, datasets, Spaces and gated access. Intermediate learning_resource docs US/France en - ### [Nomic blog](/context-engineering/retrieval-and-rag/nomic-blog/index.md) by [Nomic](https://nomic.ai/) Nomic blog with posts on its embedding models, data mapping tools and company updates. Intermediate learning_resource blog US en - ### [Scale AI blog](/context-engineering/retrieval-and-rag/scale-ai-blog/index.md) by [Scale AI](https://scale.com/) Scale AI blog with company updates and articles on data labelling, evaluation and AI research. Intermediate learning_resource blog US en - ### [Get started with Copilot in Excel](/context-engineering/retrieval-and-rag/spreadsheets-data-analysis/index.md) by [Microsoft](https://www.microsoft.com/) Microsoft Support guide to Copilot in Excel for analysing data, generating formulas and creating charts. Intermediate–Advanced learning_resource docs US en - ### [Benchmarks are broken (Surge AI)](/context-engineering/retrieval-and-rag/surge-ai-benchmarks-are-broken/index.md) by [Surge AI](https://surgehq.ai/) Surge AI post arguing academic benchmarks poorly reflect model quality, with examples of benchmark flaws. Intermediate learning_resource blog US en - ### [HANDBOOK.md Benchmark (Surge AI)](/context-engineering/retrieval-and-rag/surge-ai-handbook-md-benchmark/index.md) by [Surge AI](https://surgehq.ai/) Surge AI post on HANDBOOK.md, a benchmark testing whether agents follow long company policy handbooks. Intermediate learning_resource blog US en - ### [HellaSwag or HellaBad? (Surge AI)](/context-engineering/retrieval-and-rag/surge-ai-hellaswag-or-hellabad-36-contains-errors/index.md) by [Surge AI](https://surgehq.ai/) Surge AI audit reporting errors in 36% of HellaSwag benchmark rows, with examples. Intermediate learning_resource blog US en - ### [How Shopify and Toloka built a ground-truth flywheel](/context-engineering/retrieval-and-rag/toloka-shopify-toloka-ground-truth-flywheel/index.md) by [Toloka](https://toloka.ai/) Toloka case study on building a human-labelled ground-truth loop with Shopify to keep skill agents accurate. Intermediate learning_resource guide Netherlands (RU-origin) en - ### [Ai2 model cards and datasets](/context-engineering/retrieval-and-rag/ai2-model-cards-and-datasets/index.md) by [Ai2](https://huggingface.co/allenai) Ai2's Hugging Face organisation page listing its model checkpoints, model cards and datasets, including OLMo. Advanced learning_resource docs US en - ### [LLM-jp corpora (GitLab)](/context-engineering/retrieval-and-rag/llm-jp-corpora-gitlab/index.md) by [LLM-jp (National Institute of Informatics)](https://llm-jp.nii.ac.jp/) LLM-jp datasets on the NII GitLab: open Japanese pretraining corpora. Advanced learning_resource docs Japan ja - ### [RL environments for LLM agents: Design, rewards, and validation (Snorkel)](/context-engineering/retrieval-and-rag/snorkel-rl-environments-for-llm-agents-design-rewards-validation/index.md) by [Snorkel AI](https://snorkel.ai/) Snorkel AI post on designing RL environments for LLM agents: state, actions, rewards and validation. Advanced learning_resource blog US en - ### [Partial Credit in RL: When It Opposes Completion (Snorkel)](/context-engineering/retrieval-and-rag/snorkel-when-partial-credit-opposes-completion/index.md) by [Snorkel AI](https://snorkel.ai/) Snorkel AI post on how partial-credit rewards in reinforcement learning can work against task completion. Advanced learning_resource blog US en [Next page](/data-datasets/datasets/page/2/index.md) [Plain-text Markdown copy](/data-datasets/datasets/index.md)