Training Infrastructure: 123 curated resources
Curated material on training infrastructure within model training.
123 resources · page 4 of 4
Building an efficient harness for advanced enterprise work (Glean)
Creator not supplied; see original source.
Glean post on its coding-agent harness using code execution, tool-output truncation and progressive disclosure to cut tokens.
Grouped-Query Attention
Creator not supplied; see original source.
arXiv paper (2023) introducing grouped-query attention (GQA) and uptraining multi-head models to it.
MindSpore 中文 API 文档
Creator not supplied; see original source.
MindSpore API documentation in Chinese for Huawei's deep learning training and inference framework.
Modal guides
Creator not supplied; see original source.
Modal guides: deploying model serving, GPU jobs and scalable applications on Modal's serverless platform.
Scaling to 1 million concurrent sandboxes in seconds (Modal)
Creator not supplied; see original source.
Modal engineering post on building a scheduler that runs one million concurrent code sandboxes per workspace.
Moore Threads docs center
Creator not supplied; see original source.
Moore Threads documentation centre for its Chinese GPUs and the MUSA SDK (in Chinese).
PEFT documentation
Creator not supplied; see original source.
Hugging Face PEFT documentation: parameter-efficient fine-tuning methods such as LoRA, with recipes.
Qiskit Machine Learning
Creator not supplied; see original source.
Qiskit Machine Learning documentation: quantum kernels and quantum neural networks for ML experiments.
QLoRA
Creator not supplied; see original source.
arXiv paper (2023) introducing QLoRA, fine-tuning 4-bit quantized LLMs with low-rank adapters.
Ray Serve documentation
Creator not supplied; see original source.
Ray Serve documentation: scalable library for building online inference APIs and composing model deployments.
SGLang docs: HiCache best practices
Creator not supplied; see original source.
SGLang HiCache best practices: configuring hierarchical KV caching across GPU, CPU and storage.
Tinker documentation
Creator not supplied; see original source.
Tinker documentation from Thinking Machines Lab: training API for fine-tuning open-weight LLMs, with SDK reference and recipes.
AdapTive-LeArning Speculator System (ATLAS) (Together AI)
Creator not supplied; see original source.
Together AI post on ATLAS, a speculative decoding system whose speculators keep learning from the live workload.
Cache-aware prefill-decode disaggregation (Together AI)
Creator not supplied; see original source.
Together AI post on cache-aware prefill-decode disaggregation for faster long-context inference serving.
Together AI documentation
Creator not supplied; see original source.
Together AI documentation for running, fine-tuning and serving open-source models.
vLLM docs: Automatic prefix caching
Creator not supplied; see original source.
vLLM design document on automatic prefix caching and how shared prompt prefixes reuse the KV cache.
vLLM docs
Creator not supplied; see original source.
vLLM latest documentation: installation, serving, supported models and quantisation for the LLM inference engine.
CMU 11-868 LLM Systems
Creator not supplied; see original source.
CMU 11-868 LLM Systems course site: systems for training and serving large language models, with schedule and readings.
CUDA C++ Programming Guide
Creator not supplied; see original source.
NVIDIA CUDA C++ Programming Guide: reference for the CUDA programming model, memory hierarchy and interfaces.
EAGLE
Creator not supplied; see original source.
arXiv paper (2024) introducing EAGLE, speculative sampling with feature-level draft heads.
Fast Inference from Transformers via Speculative Decoding
Creator not supplied; see original source.
arXiv paper (2022) introducing speculative decoding to speed up transformer inference without changing outputs.
FlashAttention
Creator not supplied; see original source.
arXiv paper (2022) introducing FlashAttention, an IO-aware exact attention algorithm for GPUs.
How to Scale Your Model (JAX scaling book)
Creator not supplied; see original source.
How to Scale Your Model: free online book on TPU/GPU rooflines, transformer math, parallelism and serving for LLMs.
The Ultra-Scale Playbook
Understand how parallelism changes memory use and training throughput.
PagedAttention / vLLM paper
Creator not supplied; see original source.
arXiv paper (2023) introducing PagedAttention and the vLLM serving system for efficient KV-cache memory management.
SGLang docs
Creator not supplied; see original source.
SGLang documentation: LLM serving framework with RadixAttention prefix caching and fast structured outputs.
How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance (Simon Boehm)
Creator not supplied; see original source.
Simon Boehm's worklog optimising a CUDA matrix-multiplication kernel step by step towards cuBLAS performance.