Model Training
Study the data, algorithms and infrastructure used to change model weights.
- Pretraining (3)
- Training Data (1)
- Post Training Rl (5)
- Fine Tuning (4)
- Training Infrastructure (123)
136 resources · page 5 of 5
How to Scale Your Model (JAX scaling book)
Creator not supplied; see original source.
How to Scale Your Model: free online book on TPU/GPU rooflines, transformer math, parallelism and serving for LLMs.
The Smol Training Playbook
A training account that includes failed experiments and engineering decisions.
The Ultra-Scale Playbook
Understand how parallelism changes memory use and training throughput.
The FineWeb Datasets
by Guilherme Penedo and coauthors, Hugging Face
Study how filtering and deduplication affect pretraining data.
PagedAttention / vLLM paper
Creator not supplied; see original source.
arXiv paper (2023) introducing PagedAttention and the vLLM serving system for efficient KV-cache memory management.
SGLang docs
Creator not supplied; see original source.
SGLang documentation: LLM serving framework with RadixAttention prefix caching and fast structured outputs.
How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance (Simon Boehm)
Creator not supplied; see original source.
Simon Boehm's worklog optimising a CUDA matrix-multiplication kernel step by step towards cuBLAS performance.
verl (HybridFlow)
by verl project (originated at ByteDance Seed)
Explore how an RL training framework coordinates generation and optimization.