--- id: "/data-datasets/advanced/" title: "Data & datasets for advanced learners" url: "https://skillissue.dev/data-datasets/advanced/" --- # Data & datasets for advanced learners [Beginner](/data-datasets/beginner/index.md) [Intermediate](/data-datasets/intermediate/index.md) [Advanced](/data-datasets/advanced/index.md) [Expert](/data-datasets/expert/index.md) 45 resources · page 1 of 2 - ### [AI9Stars model and dataset registry](/context-engineering/retrieval-and-rag/ai9stars-model-and-dataset-registry/index.md) by [AI9Stars](https://huggingface.co/ai9stars) AI9Stars Hugging Face organisation: compact G9 language models and datasets. entity org International; domicile unverified en - ### [AMLSim](/context-engineering/retrieval-and-rag/amlsim/index.md) by [International Business Machines](https://www.ibm.com/opensource/) IBM AMLSim GitHub repo: multi-agent simulator generating synthetic bank transaction data for anti-money-laundering research. entity repo US/global en - ### [Argilla](/context-engineering/retrieval-and-rag/argilla-repo/index.md) by [Argilla](https://argilla.io) Argilla GitHub repo: collaboration tool for building datasets with human feedback and annotation. entity repo Spain/EU en - ### [ColBERT](/context-engineering/retrieval-and-rag/colbert/index.md) by [Future Data Systems](https://github.com/stanford-futuredata) Stanford ColBERT GitHub repo: late-interaction neural retrieval model and library. entity repo US en - ### [Common Crawl](/context-engineering/retrieval-and-rag/common-crawl/index.md) by [Common Crawl](https://commoncrawl.org/) Common Crawl: nonprofit maintaining an open repository of web crawl data, a common source of model training data. entity lab US en - ### [data engineering / RAG pipelines](/context-engineering/retrieval-and-rag/data-engineering-rag-pipelines/index.md) by [InfiniFlow](https://ragflow.io/) InfiniFlow RAGFlow GitHub repo: open-source retrieval-augmented generation engine with document parsing. entity repo US en - ### [datatrove](/context-engineering/retrieval-and-rag/datatrove/index.md) by [Hugging Face](https://huggingface.co/) Hugging Face datatrove GitHub repo: large-scale text data processing library, used to build FineWeb. entity repo France en - ### [DatologyAI](/context-engineering/retrieval-and-rag/datologyai/index.md) by [DatologyAI](https://www.datologyai.com/) DatologyAI: company offering automated curation of model training data. entity lab US en - ### [Docling (IBM)](/context-engineering/retrieval-and-rag/docling-ibm/index.md) by [Docling Project](https://github.com/docling-project) IBM Docling GitHub repo: converts PDFs and other documents into structured formats for generative AI. entity repo US en - ### [FathomNet](/context-engineering/retrieval-and-rag/fathomnet/index.md) by [FathomNet](https://fathomnet.org/) FathomNet: labelled database of underwater images with ML tools for analysing ocean visual data. entity dataset US (MBARI) en - ### [FineWeb](/context-engineering/retrieval-and-rag/fineweb/index.md) by [FineData](https://huggingface.co/HuggingFaceFW) Hugging Face FineWeb dataset: about 15 trillion tokens of filtered English web text for LLM pretraining. entity dataset US en - ### [Global Fishing Watch](/context-engineering/retrieval-and-rag/global-fishing-watch/index.md) by [Global Fishing Watch](https://globalfishingwatch.org/) Global Fishing Watch: uses machine learning on satellite and vessel data to map human activity at sea. entity org Global en|es - ### [HF datasets](/context-engineering/retrieval-and-rag/hf-datasets/index.md) by [Hugging Face](https://huggingface.co/) Hugging Face Datasets hub: searchable catalogue of public machine-learning datasets. entity lab US en - ### [LightOn HF](/context-engineering/retrieval-and-rag/lighton-hf/index.md) by [LightOn AI](https://huggingface.co/lightonai) LightOn Hugging Face organisation: retrieval, OCR and language models. entity lab EU-FR en - ### [OtterTune wiki: An End-To-End Tuning Example](/context-engineering/retrieval-and-rag/machine-learning-database-configuration-tuning-an-end-to-end-tuning-example-veri/index.md) by [CMU Database Group](https://db.cs.cmu.edu) OtterTune wiki page on GitHub (CMU): end-to-end example of machine-learning database configuration tuning. entity repo US en - ### [Marker (Datalab)](/context-engineering/retrieval-and-rag/marker-datalab/index.md) by [Datalab](https://www.datalab.to) Datalab Marker GitHub repo: converts PDF documents to Markdown and JSON. entity repo US en - ### [MinerU (OpenDataLab)](/context-engineering/retrieval-and-rag/mineru-opendatalab/index.md) by [OpenDataLab](https://opendatalab.org.cn) OpenDataLab MinerU GitHub repo: converts PDFs and Office documents into Markdown and JSON. entity repo CN en - ### [Multimodal Universe](/context-engineering/retrieval-and-rag/multimodal-universe/index.md) by [Multimodal Universe Collaboration](https://github.com/MultimodalUniverse) Multimodal Universe GitHub repo: large dataset of astronomical images, spectra and light curves for training ML models. entity dataset Global en - ### [MVTec Anomaly Detection dataset](/context-engineering/retrieval-and-rag/mvtec-anomaly-detection-dataset/index.md) by [MVTec Software](https://www.mvtec.com/) MVTec AD dataset page: over 5,000 high-resolution images for industrial anomaly detection and localisation. entity dataset EU (DE) en - ### [Qwen-Agent](/context-engineering/retrieval-and-rag/qwen-agent/index.md) by [Qwen](https://qwen.ai/) QwenLM Qwen-Agent GitHub repo: agent framework with function calling, MCP and code interpreter for Qwen models. entity repo China zh - ### [RAG_Techniques](/context-engineering/retrieval-and-rag/rag-techniques/index.md) by [Nir Diamant](https://newsletter.diamant-ai.com) RAG_Techniques GitHub repo (Nir Diamant): notebooks demonstrating many retrieval-augmented generation techniques. entity repo Israel en - ### [circuit_training: Ariane RISC-V example](/context-engineering/retrieval-and-rag/reinforcement-learning-chip-floorplanning-circuit-training-at-scale-with-ariane-/index.md) by [Google Research](https://research.google) Ariane documentation in Google's circuit_training repo: RL chip floorplanning on the Ariane RISC-V design. entity repo US en - ### [Vera C. Rubin Observatory](/context-engineering/retrieval-and-rag/vera-c-rubin-observatory/index.md) by [Vera C. Rubin Observatory](https://rubinobservatory.org/) Vera C. Rubin Observatory: NSF–DOE telescope running the LSST survey, whose alert stream requires ML triage. entity org US/CL en|es - ### [xView2 / xBD](/context-engineering/retrieval-and-rag/xview2-xbd/index.md) by [Defense Innovation Unit (xView2)](https://www.diu.mil/) xView2 challenge site: xBD satellite imagery dataset for assessing building damage after natural disasters. entity benchmark US (DIU) en - ### [Yambda dataset](/context-engineering/retrieval-and-rag/yambda-dataset/index.md) by [Yandex](https://huggingface.co/yandex) Yandex Yambda dataset on Hugging Face: large music-streaming interaction dataset for recommender-system research. entity dataset Russia en - ### [Databricks: Add AI-generated comments to Unity Catalog objects](/context-engineering/retrieval-and-rag/data-catalog-metadata-and-column-description-enrichment-add-ai-generated-comment/index.md) by [Databricks](https://www.databricks.com/) Databricks documentation on AI-generated comments for Unity Catalog tables and columns. Intermediate–Advanced learning_resource docs en - ### [SDV: CTGANSynthesizer](/context-engineering/retrieval-and-rag/gan-based-synthetic-tabular-data-generation-ctgansynthesizer-verified-co/index.md) by [DataCebo (Synthetic Data Vault)](https://datacebo.com/) Synthetic Data Vault documentation for CTGANSynthesizer, a GAN-based generator of synthetic tabular data. Intermediate–Advanced learning_resource docs en - ### [Upgrading Java versions with Amazon Q Developer](/context-engineering/retrieval-and-rag/java-language-and-dependency-modernization-upgrading-java-versions-with-amazon-q/index.md) by [Amazon Web Services](https://aws.amazon.com/) AWS documentation on upgrading Java application versions with Amazon Q Developer code transformation. Intermediate–Advanced learning_resource docs US en - ### [AI-Network-Troubleshooting-PoC (Cisco Code Exchange)](/context-engineering/retrieval-and-rag/network-routing-fault-diagnosis-with-telemetry-and-agents-ai-network-troubleshoo/index.md) by [Cisco](https://www.cisco.com/) Cisco Code Exchange entry for a proof-of-concept that uses AI agents and telemetry to troubleshoot network faults. Intermediate–Advanced learning_resource docs en - ### [aider docs: Usage](/context-engineering/retrieval-and-rag/repository-aware-code-generation-and-editing-usage-aider-verified-local-models-s/index.md) by [Aider](https://aider.chat/) Aider usage documentation: how to add files, chat and edit code in a git repo, including with local models. Intermediate–Advanced learning_resource docs West en - ### [Get started with Copilot in Excel (Excel support)](/context-engineering/retrieval-and-rag/spreadsheet-formula-and-workbook-automation-get-started-with-copilot-in-excel/index.md) by [Microsoft](https://www.microsoft.com/) Microsoft Support getting-started guide for Copilot in Excel: data analysis, formulas and charts. Intermediate–Advanced learning_resource docs US en - ### [Get started with Copilot in Excel](/context-engineering/retrieval-and-rag/spreadsheets-data-analysis/index.md) by [Microsoft](https://www.microsoft.com/) Microsoft Support guide to Copilot in Excel for analysing data, generating formulas and creating charts. Intermediate–Advanced learning_resource docs US en [Next page](/data-datasets/advanced/page/2/index.md) [Plain-text Markdown copy](/data-datasets/advanced/index.md)