Inference Serving

58 resources · page 1 of 2

• OpenInference

  Creator not supplied; see original source.

  Arize OpenInference GitHub repo: OpenTelemetry instrumentation and semantic conventions for LLM tracing.

  Not applicable (entity) · unknown · unknown resource · en

• Truss

  Creator not supplied; see original source.

  Baseten Truss GitHub repo: packaging and serving ML models in production.

  Not applicable (entity) · unknown · unknown resource · en

• checkpoint-engine

  Creator not supplied; see original source.

  Moonshot checkpoint-engine GitHub repo: middleware for updating model weights in running LLM inference engines.

  Not applicable (entity) · unknown · unknown resource · en

• DeepSeek Open Infra Index

  Creator not supplied; see original source.

  DeepSeek Open Infra Index GitHub repo: index of released infrastructure tools such as FlashMLA, DeepEP, DeepGEMM and 3FS.

  Not applicable (entity) · unknown · unknown resource · en

• py-why/dowhy

  Creator not supplied; see original source.

  PyWhy DoWhy GitHub repo: Python library for causal inference with explicit assumptions and refutation tests.

  Not applicable (entity) · unknown · unknown resource · en

• FastDeploy

  Creator not supplied; see original source.

  PaddlePaddle FastDeploy GitHub repo: inference and deployment toolkit for LLMs and VLMs.

  Not applicable (entity) · unknown · unknown resource · en|zh

• Fireworks cookbook

  Creator not supplied; see original source.

  Fireworks cookbook GitHub repo: recipes for fine-tuning, function calling and building with Fireworks AI.

  Not applicable (entity) · unknown · unknown resource · en

• LMDeploy

  Creator not supplied; see original source.

  InternLM LMDeploy GitHub repo: toolkit for compressing, deploying and serving LLMs.

  Not applicable (entity) · unknown · unknown resource · en|zh

• MLX / mlx-lm

  Creator not supplied; see original source.

  Apple mlx-lm GitHub repo: Python package for running and fine-tuning LLMs with MLX.

  Not applicable (entity) · unknown · unknown resource · en

• Mooncake

  Creator not supplied; see original source.

  Mooncake GitHub repo: KV-cache-centric disaggregated serving platform used for Kimi.

  Not applicable (entity) · unknown · unknown resource · en

• Motif Technologies

  Creator not supplied; see original source.

  Motif Technologies GitHub organisation: training and serving code for its Korean models.

  Not applicable (entity) · unknown · unknown resource · en

• Multiverse Computing model registry

  Creator not supplied; see original source.

  Multiverse Computing Hugging Face organisation: compressed model cards such as HyperNova.

  Not applicable (entity) · unknown · unknown resource · en

• NVIDIA Dynamo

  Creator not supplied; see original source.

  ai-dynamo Dynamo GitHub repo: NVIDIA's datacenter-scale distributed, disaggregated inference serving framework.

  Not applicable (entity) · unknown · unknown resource · en

• NVIDIA NeMo Speech

  Creator not supplied; see original source.

  NVIDIA NeMo Speech GitHub repo: framework for speech recognition and TTS model training and inference.

  Not applicable (entity) · unknown · unknown resource · en

• NVIDIA/TensorRT-LLM

  Creator not supplied; see original source.

  NVIDIA TensorRT-LLM GitHub repo: Python API and kernels for optimised LLM inference on NVIDIA GPUs.

  Not applicable (entity) · unknown · unknown resource · en

• Ollama

  Creator not supplied; see original source.

  Ollama GitHub repo: runtime for running open LLMs locally, with API and cloud options.

  Not applicable (entity) · unknown · unknown resource · en

• OLMo training and inference

  Creator not supplied; see original source.

  Ai2 OLMo GitHub repo: modelling, training, evaluation and inference code for OLMo open language models.

  Not applicable (entity) · unknown · unknown resource · en

• onediff

  Creator not supplied; see original source.

  SiliconFlow onediff GitHub repo: acceleration library for diffusion model inference.

  Not applicable (entity) · unknown · unknown resource · en|zh

• Together cookbook

  Creator not supplied; see original source.

  Together cookbook GitHub repo: notebooks with recipes for building with Together AI models.

  Not applicable (entity) · unknown · unknown resource · en

• Alibaba Cloud Model Studio: Prompt engineering guide

  Creator not supplied; see original source.

  Alibaba Cloud Model Studio prompt engineering guide in Chinese with techniques for designing and optimising prompts.

  Beginner · unknown · unknown resource · zh

• Cloud.ru Foundation Models documentation

  Creator not supplied; see original source.

  Cloud.ru Foundation Models documentation: Sber subsidiary's unified API for hosted language models (in Russian).

  Beginner · unknown · unknown resource · ru

• fal Docs (fal.ai/docs)

  Creator not supplied; see original source.

  fal documentation for deploying and calling generative media models on fal's serving platform.

  Beginner · unknown · unknown resource · en

• GroqDocs: Prompt Basics

  Creator not supplied; see original source.

  Groq prompting guide covering roles, instructions, context, input and output formats, and sampling parameters.

  Beginner · unknown · unknown resource · en

• LM Studio docs

  Creator not supplied; see original source.

  LM Studio app documentation: running Llama, DeepSeek, Qwen and other LLMs locally on a desktop.

  Beginner · unknown · unknown resource · en

• GigaChat docs: Prompt engineering basics

  Creator not supplied; see original source.

  Sber GigaChat prompt engineering basics in Russian: how to write effective prompts for GigaChat.

  Beginner · unknown · unknown resource · ru

• SiliconFlow docs: Platform introduction

  Creator not supplied; see original source.

  SiliconFlow platform introduction in Chinese: inference service offering API access to many open models.

  Beginner · unknown · unknown resource · zh

• Volcano Engine Ark (方舟) docs

  Creator not supplied; see original source.

  Volcano Engine Ark documentation entry: ByteDance's model platform docs, redirecting to docs.volcengine.com.

  Beginner · unknown · unknown resource · zh

• Alibaba Cloud Model Studio: Create a knowledge base

  Creator not supplied; see original source.

  Alibaba Cloud Model Studio documentation on creating knowledge bases for RAG, with parsing and chunking settings (in Chinese).

  Intermediate · unknown · unknown resource · zh

• Cloudflare Workers AI

  Creator not supplied; see original source.

  Cloudflare Workers AI documentation for running ML models on serverless GPUs across Cloudflare's network.

  Intermediate · unknown · unknown resource · en

• Groq documentation

  Creator not supplied; see original source.

  Groq documentation: OpenAI-compatible API, available models and rate limits for its LPU inference service.

  Intermediate · unknown · unknown resource · en

• vLLM documentation

  Creator not supplied; see original source.

  vLLM documentation: open-source engine for high-throughput LLM inference and serving.

  Intermediate–Advanced · unknown · unknown resource · en

• OpenRouter docs: Prompt Caching

  Creator not supplied; see original source.

  OpenRouter guide to prompt caching across model providers, with cost implications.

  Intermediate · unknown · unknown resource · en

Plain-text Markdown copy