--- id: "/infrastructure-chips-efficiency/inference-serving/page/2/" title: "inference & serving: 58 curated resources · page 2" url: "https://skillissue.dev/infrastructure-chips-efficiency/inference-serving/page/2/" --- # inference & serving: 58 curated resources inference & serving [Beginner](/infrastructure-chips-efficiency/beginner/index.md) [Intermediate](/infrastructure-chips-efficiency/intermediate/index.md) [Advanced](/infrastructure-chips-efficiency/advanced/index.md) [Expert](/infrastructure-chips-efficiency/expert/index.md) 58 resources · page 2 of 2 - ### [Qwen 中文文档](/model-training/training-infrastructure/qwen/index.md) by [Qwen (Alibaba)](https://qwen.ai/) Qwen documentation in Chinese: usage, deployment and fine-tuning of Qwen language and multimodal models. Intermediate learning_resource docs China zh - ### [Transformers documentation](/model-training/training-infrastructure/transformers-documentation/index.md) by [Hugging Face](https://huggingface.co/) Hugging Face Transformers documentation: reference for model architectures, loading, training and generation. Intermediate learning_resource docs US; France; international en - ### [Agentic inference optimization (Baseten)](/model-training/training-infrastructure/baseten-agentic-inference-optimization/index.md) by [Baseten](https://baseten.co/) Baseten post on optimising inference engines for agent workloads, reporting decode-speed and latency gains over vLLM. Advanced learning_resource blog US en - ### [Baseten documentation](/model-training/training-infrastructure/baseten-documentation/index.md) by [Baseten](https://www.baseten.co/) Baseten documentation for running hosted models, deploying custom models and training on GPU infrastructure. Advanced learning_resource docs US en - ### [BentoML documentation](/model-training/training-infrastructure/bentoml-documentation/index.md) by [BentoML](https://www.bentoml.com/) BentoML documentation: open-source inference platform for packaging, deploying and scaling AI models. Advanced learning_resource docs US/global en - ### [Defeating Nondeterminism in LLM Inference](/model-training/training-infrastructure/defeating-nondeterminism-in-llm-inference/index.md) by [Thinking Machines Lab](https://thinkingmachines.ai/) Thinking Machines Lab post on the causes of nondeterminism in LLM inference and batch-invariant kernels to fix it. Advanced learning_resource paper US en - ### [DoWhy documentation](/model-training/training-infrastructure/dowhy-2/index.md) by [PyWhy](https://www.pywhy.org/) DoWhy documentation: Python library for causal inference with modelling, identification, estimation and refutation steps. Advanced learning_resource docs Global en - ### [EconML](/model-training/training-infrastructure/econml/index.md) by [PyWhy](https://www.pywhy.org/) EconML documentation: Python methods for estimating heterogeneous treatment effects with machine learning. Advanced learning_resource docs US; international en - ### [Feast documentation](/model-training/training-infrastructure/feast-documentation/index.md) by [Feast project](https://feast.dev/) Feast documentation: open-source feature store for serving consistent features to ML training and inference. Advanced learning_resource docs Global en - ### [Fireworks AI documentation](/model-training/training-infrastructure/fireworks-ai-documentation/index.md) by [Fireworks AI](https://fireworks.ai/) Fireworks AI documentation: serverless and dedicated inference, fine-tuning and deployment of open models. Advanced learning_resource docs US en - ### [Three Tests to Run Before You Switch from LoRA to FullFT (Fireworks AI)](/model-training/training-infrastructure/fireworks-three-tests-before-you-switch-from-lora-to-fullft/index.md) by [Fireworks AI](https://fireworks.ai/) Fireworks AI post with controlled experiments on Qwen3.5-9B showing what to test before switching from LoRA to full fine-tuning. Advanced learning_resource blog US en - ### [Grouped-Query Attention](/model-training/training-infrastructure/grouped-query-attention/index.md) by [Joshua Ainslie et al. (Google)](https://ai.google/) arXiv paper (2023) introducing grouped-query attention (GQA) and uptraining multi-head models to it. Advanced learning_resource paper US en - ### [Scaling to 1 million concurrent sandboxes in seconds (Modal)](/model-training/training-infrastructure/modal-scaling-to-1m-concurrent-sandboxes/index.md) by [Modal](https://modal.com/) Modal engineering post on building a scheduler that runs one million concurrent code sandboxes per workspace. Advanced learning_resource blog US en - ### [Ray Serve documentation](/model-training/training-infrastructure/ray-serve-documentation/index.md) by [Anyscale (Ray)](https://www.anyscale.com/) Ray Serve documentation: scalable library for building online inference APIs and composing model deployments. Advanced learning_resource docs US/global en - ### [SGLang docs: HiCache best practices](/model-training/training-infrastructure/sglang-hicache-best-practices/index.md) by [sgl-project](https://github.com/sgl-project) SGLang HiCache best practices: configuring hierarchical KV caching across GPU, CPU and storage. Advanced learning_resource guide US/China en - ### [AdapTive-LeArning Speculator System (ATLAS) (Together AI)](/model-training/training-infrastructure/together-ai-atlas-runtime-learning-speculators/index.md) by [Together AI](https://together.ai/) Together AI post on ATLAS, a speculative decoding system whose speculators keep learning from the live workload. Advanced learning_resource blog US en - ### [Cache-aware prefill-decode disaggregation (Together AI)](/model-training/training-infrastructure/together-ai-cache-aware-prefill-decode-disaggregation/index.md) by [Together AI](https://together.ai/) Together AI post on cache-aware prefill-decode disaggregation for faster long-context inference serving. Advanced learning_resource blog US en - ### [Together AI documentation](/model-training/training-infrastructure/together-ai-documentation/index.md) by [Together AI](https://www.together.ai/) Together AI documentation for running, fine-tuning and serving open-source models. Advanced learning_resource docs US en - ### [vLLM docs: Automatic prefix caching](/model-training/training-infrastructure/vllm-automatic-prefix-caching-design/index.md) by [vLLM](https://github.com/vllm-project) vLLM design document on automatic prefix caching and how shared prompt prefixes reuse the KV cache. Advanced learning_resource docs Global en - ### [vLLM docs](/model-training/training-infrastructure/vllm-docs/index.md) by [vLLM project](https://vllm.ai/) vLLM latest documentation: installation, serving, supported models and quantisation for the LLM inference engine. Advanced learning_resource docs US/Global en - ### [CMU 11-868 LLM Systems](/model-training/training-infrastructure/cmu-11-868-llm-systems/index.md) by [llmsystem](https://github.com/llmsystem) CMU 11-868 LLM Systems course site: systems for training and serving large language models, with schedule and readings. Expert learning_resource course US en - ### [EAGLE](/model-training/training-infrastructure/eagle/index.md) by [Yuhui Li et al.](https://orcid.org/0000-0003-3375-0273) arXiv paper (2024) introducing EAGLE, speculative sampling with feature-level draft heads. Expert learning_resource paper China/Canada en - ### [Fast Inference from Transformers via Speculative Decoding](/model-training/training-infrastructure/fast-inference-from-transformers-via-speculative-decoding/index.md) by [Yaniv Leviathan et al. (Google)](https://ai.google/) arXiv paper (2022) introducing speculative decoding to speed up transformer inference without changing outputs. Expert learning_resource paper Israel; US en - ### [How to Scale Your Model (JAX scaling book)](/model-training/training-infrastructure/how-to-scale-your-model-jax-scaling-book/index.md) by [jax-ml](https://docs.jax.dev) How to Scale Your Model: free online book on TPU/GPU rooflines, transformer math, parallelism and serving for LLMs. Expert learning_resource book US en - ### [PagedAttention / vLLM paper](/model-training/training-infrastructure/pagedattention-vllm-paper/index.md) by [Woosuk Kwon et al. (UC Berkeley)](https://www.berkeley.edu/) arXiv paper (2023) introducing PagedAttention and the vLLM serving system for efficient KV-cache memory management. Expert learning_resource paper US; international en - ### [SGLang docs](/model-training/training-infrastructure/sglang-docs/index.md) by [SGLang project (LMSYS)](https://lmsys.org/) SGLang documentation: LLM serving framework with RadixAttention prefix caching and fast structured outputs. Expert learning_resource docs US/China en [Previous page](/infrastructure-chips-efficiency/inference-serving/index.md) [Plain-text Markdown copy](/infrastructure-chips-efficiency/inference-serving/page/2/index.md)