Infrastructure, chips & efficiency
Infrastructure, chips & efficiency
- chips & hardware (37)
- efficiency & quantization (8)
- inference & serving (58)
- local/on-device (6)
- training systems (13)
122 resources · page 4 of 4
Building an efficient harness for advanced enterprise work (Glean)
by Glean
Glean post on its coding-agent harness using code execution, tool-output truncation and progressive disclosure to cut tokens.
Grouped-Query Attention
by Joshua Ainslie et al. (Google)
arXiv paper (2023) introducing grouped-query attention (GQA) and uptraining multi-head models to it.
MindSpore 中文 API 文档
MindSpore API documentation in Chinese for Huawei's deep learning training and inference framework.
Modal guides
by Modal
Modal guides: deploying model serving, GPU jobs and scalable applications on Modal's serverless platform.
Scaling to 1 million concurrent sandboxes in seconds (Modal)
by Modal
Modal engineering post on building a scheduler that runs one million concurrent code sandboxes per workspace.
Moore Threads docs center
Moore Threads documentation centre for its Chinese GPUs and the MUSA SDK (in Chinese).
PEFT documentation
by Hugging Face
Hugging Face PEFT documentation: parameter-efficient fine-tuning methods such as LoRA, with recipes.
Qiskit Machine Learning
Qiskit Machine Learning documentation: quantum kernels and quantum neural networks for ML experiments.
QLoRA
by Tim Dettmers et al. (University of Washington)
arXiv paper (2023) introducing QLoRA, fine-tuning 4-bit quantized LLMs with low-rank adapters.
Ray Serve documentation
Ray Serve documentation: scalable library for building online inference APIs and composing model deployments.
SGLang docs: HiCache best practices
by sgl-project
SGLang HiCache best practices: configuring hierarchical KV caching across GPU, CPU and storage.
Tinker documentation
Tinker documentation from Thinking Machines Lab: training API for fine-tuning open-weight LLMs, with SDK reference and recipes.
AdapTive-LeArning Speculator System (ATLAS) (Together AI)
by Together AI
Together AI post on ATLAS, a speculative decoding system whose speculators keep learning from the live workload.
Cache-aware prefill-decode disaggregation (Together AI)
by Together AI
Together AI post on cache-aware prefill-decode disaggregation for faster long-context inference serving.
Together AI documentation
by Together AI
Together AI documentation for running, fine-tuning and serving open-source models.
vLLM docs: Automatic prefix caching
by vLLM
vLLM design document on automatic prefix caching and how shared prompt prefixes reuse the KV cache.
vLLM docs
by vLLM project
vLLM latest documentation: installation, serving, supported models and quantisation for the LLM inference engine.
CMU 11-868 LLM Systems
by llmsystem
CMU 11-868 LLM Systems course site: systems for training and serving large language models, with schedule and readings.
CUDA C++ Programming Guide
by NVIDIA
NVIDIA CUDA C++ Programming Guide: reference for the CUDA programming model, memory hierarchy and interfaces.
EAGLE
arXiv paper (2024) introducing EAGLE, speculative sampling with feature-level draft heads.
Fast Inference from Transformers via Speculative Decoding
by Yaniv Leviathan et al. (Google)
arXiv paper (2022) introducing speculative decoding to speed up transformer inference without changing outputs.
FlashAttention
by Tri Dao et al. (Stanford University)
arXiv paper (2022) introducing FlashAttention, an IO-aware exact attention algorithm for GPUs.
How to Scale Your Model (JAX scaling book)
by jax-ml
How to Scale Your Model: free online book on TPU/GPU rooflines, transformer math, parallelism and serving for LLMs.
PagedAttention / vLLM paper
by Woosuk Kwon et al. (UC Berkeley)
arXiv paper (2023) introducing PagedAttention and the vLLM serving system for efficient KV-cache memory management.
SGLang docs
SGLang documentation: LLM serving framework with RadixAttention prefix caching and fast structured outputs.
How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance (Simon Boehm)
by Simon Boehm
Simon Boehm's worklog optimising a CUDA matrix-multiplication kernel step by step towards cuBLAS performance.