--- id: "/data-datasets/data-engineering-rag/" title: "data engineering & RAG: 52 curated resources" url: "https://skillissue.dev/data-datasets/data-engineering-rag/" --- # data engineering & RAG: 52 curated resources data engineering & RAG [Beginner](/data-datasets/beginner/index.md) [Intermediate](/data-datasets/intermediate/index.md) [Advanced](/data-datasets/advanced/index.md) [Expert](/data-datasets/expert/index.md) 52 resources · page 1 of 2 - ### [AMLSim](/context-engineering/retrieval-and-rag/amlsim/index.md) by [International Business Machines](https://www.ibm.com/opensource/) IBM AMLSim GitHub repo: multi-agent simulator generating synthetic bank transaction data for anti-money-laundering research. entity repo US/global en - ### [ColBERT](/context-engineering/retrieval-and-rag/colbert/index.md) by [Future Data Systems](https://github.com/stanford-futuredata) Stanford ColBERT GitHub repo: late-interaction neural retrieval model and library. entity repo US en - ### [data engineering / RAG pipelines](/context-engineering/retrieval-and-rag/data-engineering-rag-pipelines/index.md) by [InfiniFlow](https://ragflow.io/) InfiniFlow RAGFlow GitHub repo: open-source retrieval-augmented generation engine with document parsing. entity repo US en - ### [datatrove](/context-engineering/retrieval-and-rag/datatrove/index.md) by [Hugging Face](https://huggingface.co/) Hugging Face datatrove GitHub repo: large-scale text data processing library, used to build FineWeb. entity repo France en - ### [OtterTune wiki: An End-To-End Tuning Example](/context-engineering/retrieval-and-rag/machine-learning-database-configuration-tuning-an-end-to-end-tuning-example-veri/index.md) by [CMU Database Group](https://db.cs.cmu.edu) OtterTune wiki page on GitHub (CMU): end-to-end example of machine-learning database configuration tuning. entity repo US en - ### [Qwen-Agent](/context-engineering/retrieval-and-rag/qwen-agent/index.md) by [Qwen](https://qwen.ai/) QwenLM Qwen-Agent GitHub repo: agent framework with function calling, MCP and code interpreter for Qwen models. entity repo China zh - ### [RAG_Techniques](/context-engineering/retrieval-and-rag/rag-techniques/index.md) by [Nir Diamant](https://newsletter.diamant-ai.com) RAG_Techniques GitHub repo (Nir Diamant): notebooks demonstrating many retrieval-augmented generation techniques. entity repo Israel en - ### [circuit_training: Ariane RISC-V example](/context-engineering/retrieval-and-rag/reinforcement-learning-chip-floorplanning-circuit-training-at-scale-with-ariane-/index.md) by [Google Research](https://research.google) Ariane documentation in Google's circuit_training repo: RL chip floorplanning on the Ariane RISC-V design. entity repo US en - ### [Postman: Write scripts to test API response data](/context-engineering/retrieval-and-rag/api-response-assertion-and-contract-test-authoring-write-scripts-to-test-api-res/index.md) by [Postman](https://www.postman.com/) Postman documentation on writing test scripts that assert API response data. Beginner–Intermediate learning_resource course en - ### [DataTalksClub LLM Zoomcamp](/context-engineering/retrieval-and-rag/datatalksclub-llm-zoomcamp/index.md) by [DataTalksClub](https://datatalks.club) DataTalksClub LLM Zoomcamp GitHub repo: free course on building LLM and RAG applications. Beginner–Intermediate learning_resource course Germany/Global en - ### [Summarize incidents with Microsoft Copilot in Microsoft Defender](/context-engineering/retrieval-and-rag/defensive-soc-incident-summarization-and-triage-summarize-incidents-with-microso/index.md) by [Microsoft](https://www.microsoft.com/) Microsoft Learn page on generating incident summaries with Microsoft Copilot in Microsoft Defender. Beginner–Intermediate learning_resource course US en - ### [Getting started with Pulumi Neo](/context-engineering/retrieval-and-rag/infrastructure-as-code-maintenance-and-upgrade-planning-getting-started-with-pul/index.md) by [Pulumi](https://www.pulumi.com/) Pulumi documentation on setting up Pulumi Neo, an AI agent for infrastructure-as-code tasks. Beginner learning_resource docs en - ### [Using GitHub Copilot code review](/context-engineering/retrieval-and-rag/ai-assisted-pull-request-code-review-using-github-copilot-code-review/index.md) by [GitHub](https://github.com/) GitHub documentation on requesting pull-request code reviews from GitHub Copilot. Intermediate learning_resource guide US en - ### [Resolving code scanning alerts (GitHub Docs)](/context-engineering/retrieval-and-rag/ai-assisted-remediation-of-static-analysis-vulnerabilities-resolving-code-scanni/index.md) by [GitHub](https://github.com/) GitHub documentation on viewing, fixing and dismissing code scanning alerts, including Copilot Autofix. Intermediate learning_resource guide US en - ### [Introducing Contextual Retrieval (Anthropic)](/context-engineering/retrieval-and-rag/anthropic-contextual-retrieval/index.md) by [Anthropic](https://anthropic.com/) Anthropic announcement of Contextual Retrieval, combining contextual embeddings, BM25 and reranking, with measured retrieval gains. Intermediate learning_resource guide US en - ### [Copilot in Power BI tutorial: Explore data and gain insights](/context-engineering/retrieval-and-rag/bi-report-exploration-and-narrative-insight-generation-copilot-in-power-bi-tutor/index.md) by [Microsoft](https://www.microsoft.com/) Microsoft tutorial on using Copilot in Power BI to explore report data and create summaries. Intermediate learning_resource guide US en - ### [Diagnosing CI test failures (GitHub Copilot Chat Cookbook)](/context-engineering/retrieval-and-rag/ci-failure-diagnosis-from-workflow-logs-and-local-code-diagnosing-ci-test-failur/index.md) by [GitHub](https://github.com/) GitHub Copilot cookbook guide on using Copilot CLI to pull CI logs, match failures to local code and fix them. Intermediate learning_resource guide US en - ### [Grafana Assistant Investigations](/context-engineering/retrieval-and-rag/cross-signal-production-incident-investigation-grafana-assistant-investigations/index.md) by [Grafana Labs](https://grafana.com/) Grafana Cloud documentation on Grafana Assistant investigations for diagnosing production incidents across signals. Intermediate learning_resource guide en - ### [Databricks: Add AI-generated comments to Unity Catalog objects](/context-engineering/retrieval-and-rag/data-catalog-metadata-and-column-description-enrichment-add-ai-generated-comment/index.md) by [Databricks](https://www.databricks.com/) Databricks documentation on AI-generated comments for Unity Catalog tables and columns. Intermediate–Advanced learning_resource docs en - ### [Datasette LLM documentation](/context-engineering/retrieval-and-rag/datasette-llm-documentation/index.md) by [Simon Willison](https://simonwillison.net/) Documentation for Simon Willison's LLM, a CLI tool and Python library for prompting many models with plugins and logged conversations. Intermediate learning_resource docs UK/US/global en - ### [Cleanlab: Classification with Tabular Data using Scikit-Learn and Cleanlab](/context-engineering/retrieval-and-rag/detecting-mislabeled-records-in-tabular-datasets-classification-with-tabular-dat/index.md) by [Cleanlab](https://cleanlab.ai/) Cleanlab tutorial on finding label errors in tabular classification data with scikit-learn models. Intermediate learning_resource guide US en - ### [DVC documentation](/context-engineering/retrieval-and-rag/dvc-documentation/index.md) by [Iterative (DVC)](https://dvc.org/) DVC documentation: open-source tool for versioning data, models and ML pipelines alongside Git. Intermediate learning_resource docs US/global en - ### [LiteRT: Get started with microcontrollers](/context-engineering/retrieval-and-rag/embedded-microcontroller-neural-inference-get-started-with-microcontrollers-veri/index.md) by [Google](https://developers.google.com/) Google AI Edge guide to running LiteRT neural-network inference on microcontrollers. Intermediate learning_resource guide US en - ### [FastGPT docs: dataset engine](/context-engineering/retrieval-and-rag/fastgpt-dataset-engine/index.md) by [labring](https://sealos.io) FastGPT documentation in Chinese on its knowledge-base engine: how documents are indexed and retrieved for RAG. Intermediate learning_resource docs China zh - ### [SDV: CTGANSynthesizer](/context-engineering/retrieval-and-rag/gan-based-synthetic-tabular-data-generation-ctgansynthesizer-verified-co/index.md) by [DataCebo (Synthetic Data Vault)](https://datacebo.com/) Synthetic Data Vault documentation for CTGANSynthesizer, a GAN-based generator of synthetic tabular data. Intermediate–Advanced learning_resource docs en - ### [Haystack by deepset](/context-engineering/retrieval-and-rag/haystack-by-deepset/index.md) by [deepset](https://www.deepset.ai/) Haystack documentation: deepset's open-source framework for building RAG pipelines and AI agents. Intermediate learning_resource docs Germany en - ### [Jason Liu RAG writing](/context-engineering/retrieval-and-rag/jason-liu-rag-writing/index.md) by [Jason Liu](https://jxnl.co/) Jason Liu's essays on RAG systems: evaluation, failure modes and lessons from coding-agent teams. Intermediate learning_resource guide US en - ### [Upgrading Java versions with Amazon Q Developer](/context-engineering/retrieval-and-rag/java-language-and-dependency-modernization-upgrading-java-versions-with-amazon-q/index.md) by [Amazon Web Services](https://aws.amazon.com/) AWS documentation on upgrading Java application versions with Amazon Q Developer code transformation. Intermediate–Advanced learning_resource docs US en - ### [Label Studio documentation](/context-engineering/retrieval-and-rag/label-studio-documentation/index.md) by [HumanSignal (Label Studio)](https://humansignal.com/) Label Studio documentation: open-source tool for labelling text, image, audio and video data for ML. Intermediate learning_resource docs US/global en - ### [LlamaIndex documentation](/context-engineering/retrieval-and-rag/llamaindex-documentation/index.md) by [LlamaIndex](https://www.llamaindex.ai/) LlamaIndex documentation: open-source framework for data ingestion, retrieval and document agents, plus LlamaParse. Intermediate learning_resource docs US en - ### [Microsoft Presidio](/context-engineering/retrieval-and-rag/microsoft-presidio/index.md) by [Microsoft (Presidio)](https://microsoft.github.io/presidio/) Microsoft Presidio documentation: open-source PII detection and anonymisation for text, images and structured data. Intermediate learning_resource docs US/global en - ### [Snowflake Cortex Analyst tutorial: time-series revenue data](/context-engineering/retrieval-and-rag/natural-language-to-sql-over-governed-database-schemas-tutorial-answer-questions/index.md) by [Snowflake](https://www.snowflake.com/) Snowflake tutorial using Cortex Analyst to answer natural-language questions over time-series revenue data with SQL. Intermediate learning_resource guide en [Next page](/data-datasets/data-engineering-rag/page/2/index.md) [Plain-text Markdown copy](/data-datasets/data-engineering-rag/index.md)