inference & serving: 58 curated resources
inference & serving
58 resources · page 2 of 2
Qwen 中文文档
Qwen documentation in Chinese: usage, deployment and fine-tuning of Qwen language and multimodal models.
Transformers documentation
by Hugging Face
Hugging Face Transformers documentation: reference for model architectures, loading, training and generation.
Agentic inference optimization (Baseten)
by Baseten
Baseten post on optimising inference engines for agent workloads, reporting decode-speed and latency gains over vLLM.
Baseten documentation
by Baseten
Baseten documentation for running hosted models, deploying custom models and training on GPU infrastructure.
BentoML documentation
by BentoML
BentoML documentation: open-source inference platform for packaging, deploying and scaling AI models.
Defeating Nondeterminism in LLM Inference
Thinking Machines Lab post on the causes of nondeterminism in LLM inference and batch-invariant kernels to fix it.
DoWhy documentation
by PyWhy
DoWhy documentation: Python library for causal inference with modelling, identification, estimation and refutation steps.
EconML
by PyWhy
EconML documentation: Python methods for estimating heterogeneous treatment effects with machine learning.
Feast documentation
Feast documentation: open-source feature store for serving consistent features to ML training and inference.
Fireworks AI documentation
by Fireworks AI
Fireworks AI documentation: serverless and dedicated inference, fine-tuning and deployment of open models.
Three Tests to Run Before You Switch from LoRA to FullFT (Fireworks AI)
by Fireworks AI
Fireworks AI post with controlled experiments on Qwen3.5-9B showing what to test before switching from LoRA to full fine-tuning.
Grouped-Query Attention
by Joshua Ainslie et al. (Google)
arXiv paper (2023) introducing grouped-query attention (GQA) and uptraining multi-head models to it.
Scaling to 1 million concurrent sandboxes in seconds (Modal)
by Modal
Modal engineering post on building a scheduler that runs one million concurrent code sandboxes per workspace.
Ray Serve documentation
Ray Serve documentation: scalable library for building online inference APIs and composing model deployments.
SGLang docs: HiCache best practices
by sgl-project
SGLang HiCache best practices: configuring hierarchical KV caching across GPU, CPU and storage.
AdapTive-LeArning Speculator System (ATLAS) (Together AI)
by Together AI
Together AI post on ATLAS, a speculative decoding system whose speculators keep learning from the live workload.
Cache-aware prefill-decode disaggregation (Together AI)
by Together AI
Together AI post on cache-aware prefill-decode disaggregation for faster long-context inference serving.
Together AI documentation
by Together AI
Together AI documentation for running, fine-tuning and serving open-source models.
vLLM docs: Automatic prefix caching
by vLLM
vLLM design document on automatic prefix caching and how shared prompt prefixes reuse the KV cache.
vLLM docs
by vLLM project
vLLM latest documentation: installation, serving, supported models and quantisation for the LLM inference engine.
CMU 11-868 LLM Systems
by llmsystem
CMU 11-868 LLM Systems course site: systems for training and serving large language models, with schedule and readings.
EAGLE
arXiv paper (2024) introducing EAGLE, speculative sampling with feature-level draft heads.
Fast Inference from Transformers via Speculative Decoding
by Yaniv Leviathan et al. (Google)
arXiv paper (2022) introducing speculative decoding to speed up transformer inference without changing outputs.
How to Scale Your Model (JAX scaling book)
by jax-ml
How to Scale Your Model: free online book on TPU/GPU rooflines, transformer math, parallelism and serving for LLMs.
PagedAttention / vLLM paper
by Woosuk Kwon et al. (UC Berkeley)
arXiv paper (2023) introducing PagedAttention and the vLLM serving system for efficient KV-cache memory management.
SGLang docs
SGLang documentation: LLM serving framework with RadixAttention prefix caching and fast structured outputs.