inference & serving: 58 curated resources

inference & serving

58 resources · page 1 of 2

• OpenInference

  by Arize AI

  Arize OpenInference GitHub repo: OpenTelemetry instrumentation and semantic conventions for LLM tracing.

  entity repo US en

• Truss

  by Baseten

  Baseten Truss GitHub repo: packaging and serving ML models in production.

  entity repo US en

• checkpoint-engine

  by Moonshot AI

  Moonshot checkpoint-engine GitHub repo: middleware for updating model weights in running LLM inference engines.

  entity repo China en

• DeepSeek Open Infra Index

  by DeepSeek

  DeepSeek Open Infra Index GitHub repo: index of released infrastructure tools such as FlashMLA, DeepEP, DeepGEMM and 3FS.

  entity leaderboard China en

• py-why/dowhy

  by PyWhy

  PyWhy DoWhy GitHub repo: Python library for causal inference with explicit assumptions and refutation tests.

  entity repo International en

• FastDeploy

  by PaddlePaddle

  PaddlePaddle FastDeploy GitHub repo: inference and deployment toolkit for LLMs and VLMs.

  entity repo China en|zh

• Fireworks cookbook

  by Fireworks AI

  Fireworks cookbook GitHub repo: recipes for fine-tuning, function calling and building with Fireworks AI.

  entity repo US en

• LMDeploy

  by InternLM

  InternLM LMDeploy GitHub repo: toolkit for compressing, deploying and serving LLMs.

  entity repo China en|zh

• MLX / mlx-lm

  by ml-explore

  Apple mlx-lm GitHub repo: Python package for running and fine-tuning LLMs with MLX.

  entity repo US en

• Mooncake

  by kvcache.ai

  Mooncake GitHub repo: KV-cache-centric disaggregated serving platform used for Kimi.

  entity repo China en

• Motif Technologies

  by Motif Technologies

  Motif Technologies GitHub organisation: training and serving code for its Korean models.

  entity org South Korea en

• Multiverse Computing model registry

  by Multiverse Computing

  Multiverse Computing Hugging Face organisation: compressed model cards such as HyperNova.

  entity org Spain / International en

• NVIDIA Dynamo

  by Dynamo

  ai-dynamo Dynamo GitHub repo: NVIDIA's datacenter-scale distributed, disaggregated inference serving framework.

  entity repo US en

• NVIDIA NeMo Speech

  by NVIDIA-NeMo

  NVIDIA NeMo Speech GitHub repo: framework for speech recognition and TTS model training and inference.

  entity repo US en

• NVIDIA/TensorRT-LLM

  by NVIDIA Corporation

  NVIDIA TensorRT-LLM GitHub repo: Python API and kernels for optimised LLM inference on NVIDIA GPUs.

  entity repo US en

• Ollama

  by Ollama

  Ollama GitHub repo: runtime for running open LLMs locally, with API and cloud options.

  entity repo US en

• OLMo training and inference

  by Ai2

  Ai2 OLMo GitHub repo: modelling, training, evaluation and inference code for OLMo open language models.

  entity repo US en

• onediff

  by SiliconFlow

  SiliconFlow onediff GitHub repo: acceleration library for diffusion model inference.

  entity repo China en|zh

• Together cookbook

  by Together

  Together cookbook GitHub repo: notebooks with recipes for building with Together AI models.

  entity repo US en

• Alibaba Cloud Model Studio: Prompt engineering guide

  by Alibaba Cloud

  Alibaba Cloud Model Studio prompt engineering guide in Chinese with techniques for designing and optimising prompts.

  Beginner learning_resource guide China zh

• Cloud.ru Foundation Models documentation

  by Cloud.ru

  Cloud.ru Foundation Models documentation: Sber subsidiary's unified API for hosted language models (in Russian).

  Beginner learning_resource docs Russia ru

• fal Docs (fal.ai/docs)

  by fal

  fal documentation for deploying and calling generative media models on fal's serving platform.

  Beginner learning_resource docs US en

• GroqDocs: Prompt Basics

  by Groq

  Groq prompting guide covering roles, instructions, context, input and output formats, and sampling parameters.

  Beginner learning_resource docs US en

• LM Studio docs

  by LM Studio

  LM Studio app documentation: running Llama, DeepSeek, Qwen and other LLMs locally on a desktop.

  Beginner learning_resource docs US en

• GigaChat docs: Prompt engineering basics

  by Sber

  Sber GigaChat prompt engineering basics in Russian: how to write effective prompts for GigaChat.

  Beginner learning_resource guide Russia ru

• SiliconFlow docs: Platform introduction

  by SiliconFlow

  SiliconFlow platform introduction in Chinese: inference service offering API access to many open models.

  Beginner learning_resource docs China zh

• Volcano Engine Ark (方舟) docs

  by Volcano Engine (ByteDance)

  Volcano Engine Ark documentation entry: ByteDance's model platform docs, redirecting to docs.volcengine.com.

  Beginner learning_resource docs China zh

• Alibaba Cloud Model Studio: Create a knowledge base

  by Alibaba Cloud

  Alibaba Cloud Model Studio documentation on creating knowledge bases for RAG, with parsing and chunking settings (in Chinese).

  Intermediate learning_resource docs China zh

• Cloudflare Workers AI

  by Cloudflare

  Cloudflare Workers AI documentation for running ML models on serverless GPUs across Cloudflare's network.

  Intermediate learning_resource docs US en

• Groq documentation

  by Groq

  Groq documentation: OpenAI-compatible API, available models and rate limits for its LPU inference service.

  Intermediate learning_resource docs US en

• vLLM documentation

  by vLLM project

  vLLM documentation: open-source engine for high-throughput LLM inference and serving.

  Intermediate–Advanced learning_resource docs US en

• OpenRouter docs: Prompt Caching

  by OpenRouter

  OpenRouter guide to prompt caching across model providers, with cost implications.

  Intermediate learning_resource docs US en

Plain-text Markdown copy