Inference Serving
58 resources · page 2 of 2
Qwen 中文文档
Creator not supplied; see original source.
Qwen documentation in Chinese: usage, deployment and fine-tuning of Qwen language and multimodal models.
Transformers documentation
Creator not supplied; see original source.
Hugging Face Transformers documentation: reference for model architectures, loading, training and generation.
Agentic inference optimization (Baseten)
Creator not supplied; see original source.
Baseten post on optimising inference engines for agent workloads, reporting decode-speed and latency gains over vLLM.
Baseten documentation
Creator not supplied; see original source.
Baseten documentation for running hosted models, deploying custom models and training on GPU infrastructure.
BentoML documentation
Creator not supplied; see original source.
BentoML documentation: open-source inference platform for packaging, deploying and scaling AI models.
Defeating Nondeterminism in LLM Inference
Creator not supplied; see original source.
Thinking Machines Lab post on the causes of nondeterminism in LLM inference and batch-invariant kernels to fix it.
DoWhy documentation
Creator not supplied; see original source.
DoWhy documentation: Python library for causal inference with modelling, identification, estimation and refutation steps.
EconML
Creator not supplied; see original source.
EconML documentation: Python methods for estimating heterogeneous treatment effects with machine learning.
Feast documentation
Creator not supplied; see original source.
Feast documentation: open-source feature store for serving consistent features to ML training and inference.
Fireworks AI documentation
Creator not supplied; see original source.
Fireworks AI documentation: serverless and dedicated inference, fine-tuning and deployment of open models.
Three Tests to Run Before You Switch from LoRA to FullFT (Fireworks AI)
Creator not supplied; see original source.
Fireworks AI post with controlled experiments on Qwen3.5-9B showing what to test before switching from LoRA to full fine-tuning.
Grouped-Query Attention
Creator not supplied; see original source.
arXiv paper (2023) introducing grouped-query attention (GQA) and uptraining multi-head models to it.
Scaling to 1 million concurrent sandboxes in seconds (Modal)
Creator not supplied; see original source.
Modal engineering post on building a scheduler that runs one million concurrent code sandboxes per workspace.
Ray Serve documentation
Creator not supplied; see original source.
Ray Serve documentation: scalable library for building online inference APIs and composing model deployments.
SGLang docs: HiCache best practices
Creator not supplied; see original source.
SGLang HiCache best practices: configuring hierarchical KV caching across GPU, CPU and storage.
AdapTive-LeArning Speculator System (ATLAS) (Together AI)
Creator not supplied; see original source.
Together AI post on ATLAS, a speculative decoding system whose speculators keep learning from the live workload.
Cache-aware prefill-decode disaggregation (Together AI)
Creator not supplied; see original source.
Together AI post on cache-aware prefill-decode disaggregation for faster long-context inference serving.
Together AI documentation
Creator not supplied; see original source.
Together AI documentation for running, fine-tuning and serving open-source models.
vLLM docs: Automatic prefix caching
Creator not supplied; see original source.
vLLM design document on automatic prefix caching and how shared prompt prefixes reuse the KV cache.
vLLM docs
Creator not supplied; see original source.
vLLM latest documentation: installation, serving, supported models and quantisation for the LLM inference engine.
CMU 11-868 LLM Systems
Creator not supplied; see original source.
CMU 11-868 LLM Systems course site: systems for training and serving large language models, with schedule and readings.
EAGLE
Creator not supplied; see original source.
arXiv paper (2024) introducing EAGLE, speculative sampling with feature-level draft heads.
Fast Inference from Transformers via Speculative Decoding
Creator not supplied; see original source.
arXiv paper (2022) introducing speculative decoding to speed up transformer inference without changing outputs.
How to Scale Your Model (JAX scaling book)
Creator not supplied; see original source.
How to Scale Your Model: free online book on TPU/GPU rooflines, transformer math, parallelism and serving for LLMs.
PagedAttention / vLLM paper
Creator not supplied; see original source.
arXiv paper (2023) introducing PagedAttention and the vLLM serving system for efficient KV-cache memory management.
SGLang docs
Creator not supplied; see original source.
SGLang documentation: LLM serving framework with RadixAttention prefix caching and fast structured outputs.