--- id: "/infrastructure-chips-efficiency/inference-serving/" title: "inference & serving: 58 curated resources" url: "https://skillissue.dev/infrastructure-chips-efficiency/inference-serving/" --- # inference & serving: 58 curated resources inference & serving [Beginner](/infrastructure-chips-efficiency/beginner/index.md) [Intermediate](/infrastructure-chips-efficiency/intermediate/index.md) [Advanced](/infrastructure-chips-efficiency/advanced/index.md) [Expert](/infrastructure-chips-efficiency/expert/index.md) 58 resources · page 1 of 2 - ### [OpenInference](/model-training/training-infrastructure/arize-openinference/index.md) by [Arize AI](https://www.arize.com) Arize OpenInference GitHub repo: OpenTelemetry instrumentation and semantic conventions for LLM tracing. entity repo US en - ### [Truss](/model-training/training-infrastructure/baseten-truss/index.md) by [Baseten](https://baseten.co) Baseten Truss GitHub repo: packaging and serving ML models in production. entity repo US en - ### [checkpoint-engine](/model-training/training-infrastructure/checkpoint-engine/index.md) by [Moonshot AI](https://moonshot.ai) Moonshot checkpoint-engine GitHub repo: middleware for updating model weights in running LLM inference engines. entity repo China en - ### [DeepSeek Open Infra Index](/model-training/training-infrastructure/deepseek-open-infra-index/index.md) by [DeepSeek](https://www.deepseek.com/) DeepSeek Open Infra Index GitHub repo: index of released infrastructure tools such as FlashMLA, DeepEP, DeepGEMM and 3FS. entity leaderboard China en - ### [py-why/dowhy](/model-training/training-infrastructure/dowhy/index.md) by [PyWhy](https://pywhy.org) PyWhy DoWhy GitHub repo: Python library for causal inference with explicit assumptions and refutation tests. entity repo International en - ### [FastDeploy](/model-training/training-infrastructure/fastdeploy/index.md) by [PaddlePaddle](https://www.paddlepaddle.org.cn/) PaddlePaddle FastDeploy GitHub repo: inference and deployment toolkit for LLMs and VLMs. entity repo China en|zh - ### [Fireworks cookbook](/model-training/training-infrastructure/fireworks-cookbook/index.md) by [Fireworks AI](https://fireworks.ai) Fireworks cookbook GitHub repo: recipes for fine-tuning, function calling and building with Fireworks AI. entity repo US en - ### [LMDeploy](/model-training/training-infrastructure/lmdeploy/index.md) by [InternLM](https://chat.intern-ai.org.cn/) InternLM LMDeploy GitHub repo: toolkit for compressing, deploying and serving LLMs. entity repo China en|zh - ### [MLX / mlx-lm](/model-training/training-infrastructure/mlx-mlx-lm/index.md) by [ml-explore](https://github.com/ml-explore) Apple mlx-lm GitHub repo: Python package for running and fine-tuning LLMs with MLX. entity repo US en - ### [Mooncake](/model-training/training-infrastructure/mooncake/index.md) by [kvcache.ai](https://madsys.cs.tsinghua.edu.cn/) Mooncake GitHub repo: KV-cache-centric disaggregated serving platform used for Kimi. entity repo China en - ### [Motif Technologies](/model-training/training-infrastructure/motif-technologies/index.md) by [Motif Technologies](https://github.com/MotifTechnologies) Motif Technologies GitHub organisation: training and serving code for its Korean models. entity org South Korea en - ### [Multiverse Computing model registry](/model-training/training-infrastructure/multiverse-computing-model-registry/index.md) by [Multiverse Computing](https://huggingface.co/MultiverseComputingCAI) Multiverse Computing Hugging Face organisation: compressed model cards such as HyperNova. entity org Spain / International en - ### [NVIDIA Dynamo](/model-training/training-infrastructure/nvidia-dynamo/index.md) by [Dynamo](https://developer.nvidia.com/dynamo) ai-dynamo Dynamo GitHub repo: NVIDIA's datacenter-scale distributed, disaggregated inference serving framework. entity repo US en - ### [NVIDIA NeMo Speech](/model-training/training-infrastructure/nvidia-nemo-speech/index.md) by [NVIDIA-NeMo](https://nvidia.com/) NVIDIA NeMo Speech GitHub repo: framework for speech recognition and TTS model training and inference. entity repo US en - ### [NVIDIA/TensorRT-LLM](/model-training/training-infrastructure/nvidia-tensorrt-llm/index.md) by [NVIDIA Corporation](https://nvidia.com) NVIDIA TensorRT-LLM GitHub repo: Python API and kernels for optimised LLM inference on NVIDIA GPUs. entity repo US en - ### [Ollama](/model-training/training-infrastructure/ollama-repo/index.md) by [Ollama](https://ollama.com) Ollama GitHub repo: runtime for running open LLMs locally, with API and cloud options. entity repo US en - ### [OLMo training and inference](/model-training/training-infrastructure/olmo-training-and-inference/index.md) by [Ai2](https://allenai.org/) Ai2 OLMo GitHub repo: modelling, training, evaluation and inference code for OLMo open language models. entity repo US en - ### [onediff](/model-training/training-infrastructure/siliconflow-onediff/index.md) by [SiliconFlow](https://siliconflow.com) SiliconFlow onediff GitHub repo: acceleration library for diffusion model inference. entity repo China en|zh - ### [Together cookbook](/model-training/training-infrastructure/together-cookbook/index.md) by [Together](https://github.com/togethercomputer) Together cookbook GitHub repo: notebooks with recipes for building with Together AI models. entity repo US en - ### [Alibaba Cloud Model Studio: Prompt engineering guide](/model-training/training-infrastructure/alibaba-cloud-bailian-prompt-engineering-guide/index.md) by [Alibaba Cloud](https://www.alibabacloud.com/) Alibaba Cloud Model Studio prompt engineering guide in Chinese with techniques for designing and optimising prompts. Beginner learning_resource guide China zh - ### [Cloud.ru Foundation Models documentation](/model-training/training-infrastructure/cloud-ru-sber-foundation-models/index.md) by [Cloud.ru](https://cloud.ru/) Cloud.ru Foundation Models documentation: Sber subsidiary's unified API for hosted language models (in Russian). Beginner learning_resource docs Russia ru - ### [fal Docs (fal.ai/docs)](/model-training/training-infrastructure/fal-docs-2/index.md) by [fal](https://fal.ai/) fal documentation for deploying and calling generative media models on fal's serving platform. Beginner learning_resource docs US en - ### [GroqDocs: Prompt Basics](/model-training/training-infrastructure/groq-prompt-basics/index.md) by [Groq](https://groq.com/) Groq prompting guide covering roles, instructions, context, input and output formats, and sampling parameters. Beginner learning_resource docs US en - ### [LM Studio docs](/model-training/training-infrastructure/lm-studio-docs/index.md) by [LM Studio](https://lmstudio.ai/) LM Studio app documentation: running Llama, DeepSeek, Qwen and other LLMs locally on a desktop. Beginner learning_resource docs US en - ### [GigaChat docs: Prompt engineering basics](/model-training/training-infrastructure/sber-gigachat/index.md) by [Sber](https://www.sber.ru/) Sber GigaChat prompt engineering basics in Russian: how to write effective prompts for GigaChat. Beginner learning_resource guide Russia ru - ### [SiliconFlow docs: Platform introduction](/model-training/training-infrastructure/siliconflow/index.md) by [SiliconFlow](https://siliconflow.cn/) SiliconFlow platform introduction in Chinese: inference service offering API access to many open models. Beginner learning_resource docs China zh - ### [Volcano Engine Ark (方舟) docs](/model-training/training-infrastructure/volcano-engine-ark-docs/index.md) by [Volcano Engine (ByteDance)](https://www.volcengine.com/) Volcano Engine Ark documentation entry: ByteDance's model platform docs, redirecting to docs.volcengine.com. Beginner learning_resource docs China zh - ### [Alibaba Cloud Model Studio: Create a knowledge base](/model-training/training-infrastructure/alibaba-cloud-bailian-knowledge-base-rag/index.md) by [Alibaba Cloud](https://www.alibabacloud.com/) Alibaba Cloud Model Studio documentation on creating knowledge bases for RAG, with parsing and chunking settings (in Chinese). Intermediate learning_resource docs China zh - ### [Cloudflare Workers AI](/model-training/training-infrastructure/cloudflare-workers-ai/index.md) by [Cloudflare](https://developers.cloudflare.com/) Cloudflare Workers AI documentation for running ML models on serverless GPUs across Cloudflare's network. Intermediate learning_resource docs US en - ### [Groq documentation](/model-training/training-infrastructure/groq-documentation/index.md) by [Groq](https://groq.com/) Groq documentation: OpenAI-compatible API, available models and rate limits for its LPU inference service. Intermediate learning_resource docs US en - ### [vLLM documentation](/model-training/training-infrastructure/inference-serving-quantization/index.md) by [vLLM project](https://vllm.ai/) vLLM documentation: open-source engine for high-throughput LLM inference and serving. Intermediate–Advanced learning_resource docs US en - ### [OpenRouter docs: Prompt Caching](/model-training/training-infrastructure/openrouter-prompt-caching/index.md) by [OpenRouter](https://openrouter.ai/) OpenRouter guide to prompt caching across model providers, with cost implications. Intermediate learning_resource docs US en [Next page](/infrastructure-chips-efficiency/inference-serving/page/2/index.md) [Plain-text Markdown copy](/infrastructure-chips-efficiency/inference-serving/index.md)