data engineering & RAG: 52 curated resources
data engineering & RAG
52 resources · page 1 of 2
AMLSim
by International Business Machines
IBM AMLSim GitHub repo: multi-agent simulator generating synthetic bank transaction data for anti-money-laundering research.
ColBERT
Stanford ColBERT GitHub repo: late-interaction neural retrieval model and library.
data engineering / RAG pipelines
by InfiniFlow
InfiniFlow RAGFlow GitHub repo: open-source retrieval-augmented generation engine with document parsing.
datatrove
by Hugging Face
Hugging Face datatrove GitHub repo: large-scale text data processing library, used to build FineWeb.
OtterTune wiki: An End-To-End Tuning Example
OtterTune wiki page on GitHub (CMU): end-to-end example of machine-learning database configuration tuning.
Qwen-Agent
by Qwen
QwenLM Qwen-Agent GitHub repo: agent framework with function calling, MCP and code interpreter for Qwen models.
RAG_Techniques
by Nir Diamant
RAG_Techniques GitHub repo (Nir Diamant): notebooks demonstrating many retrieval-augmented generation techniques.
circuit_training: Ariane RISC-V example
Ariane documentation in Google's circuit_training repo: RL chip floorplanning on the Ariane RISC-V design.
Postman: Write scripts to test API response data
by Postman
Postman documentation on writing test scripts that assert API response data.
DataTalksClub LLM Zoomcamp
DataTalksClub LLM Zoomcamp GitHub repo: free course on building LLM and RAG applications.
Summarize incidents with Microsoft Copilot in Microsoft Defender
by Microsoft
Microsoft Learn page on generating incident summaries with Microsoft Copilot in Microsoft Defender.
Getting started with Pulumi Neo
by Pulumi
Pulumi documentation on setting up Pulumi Neo, an AI agent for infrastructure-as-code tasks.
Using GitHub Copilot code review
by GitHub
GitHub documentation on requesting pull-request code reviews from GitHub Copilot.
Resolving code scanning alerts (GitHub Docs)
by GitHub
GitHub documentation on viewing, fixing and dismissing code scanning alerts, including Copilot Autofix.
Introducing Contextual Retrieval (Anthropic)
by Anthropic
Anthropic announcement of Contextual Retrieval, combining contextual embeddings, BM25 and reranking, with measured retrieval gains.
Copilot in Power BI tutorial: Explore data and gain insights
by Microsoft
Microsoft tutorial on using Copilot in Power BI to explore report data and create summaries.
Diagnosing CI test failures (GitHub Copilot Chat Cookbook)
by GitHub
GitHub Copilot cookbook guide on using Copilot CLI to pull CI logs, match failures to local code and fix them.
Grafana Assistant Investigations
by Grafana Labs
Grafana Cloud documentation on Grafana Assistant investigations for diagnosing production incidents across signals.
Databricks: Add AI-generated comments to Unity Catalog objects
by Databricks
Databricks documentation on AI-generated comments for Unity Catalog tables and columns.
Datasette LLM documentation
Documentation for Simon Willison's LLM, a CLI tool and Python library for prompting many models with plugins and logged conversations.
Cleanlab: Classification with Tabular Data using Scikit-Learn and Cleanlab
by Cleanlab
Cleanlab tutorial on finding label errors in tabular classification data with scikit-learn models.
DVC documentation
DVC documentation: open-source tool for versioning data, models and ML pipelines alongside Git.
LiteRT: Get started with microcontrollers
by Google
Google AI Edge guide to running LiteRT neural-network inference on microcontrollers.
FastGPT docs: dataset engine
by labring
FastGPT documentation in Chinese on its knowledge-base engine: how documents are indexed and retrieved for RAG.
SDV: CTGANSynthesizer
by DataCebo (Synthetic Data Vault)
Synthetic Data Vault documentation for CTGANSynthesizer, a GAN-based generator of synthetic tabular data.
Haystack by deepset
by deepset
Haystack documentation: deepset's open-source framework for building RAG pipelines and AI agents.
Jason Liu RAG writing
by Jason Liu
Jason Liu's essays on RAG systems: evaluation, failure modes and lessons from coding-agent teams.
Upgrading Java versions with Amazon Q Developer
AWS documentation on upgrading Java application versions with Amazon Q Developer code transformation.
Label Studio documentation
Label Studio documentation: open-source tool for labelling text, image, audio and video data for ML.
LlamaIndex documentation
by LlamaIndex
LlamaIndex documentation: open-source framework for data ingestion, retrieval and document agents, plus LlamaParse.
Microsoft Presidio
Microsoft Presidio documentation: open-source PII detection and anonymisation for text, images and structured data.
Snowflake Cortex Analyst tutorial: time-series revenue data
by Snowflake
Snowflake tutorial using Cortex Analyst to answer natural-language questions over time-series revenue data with SQL.