Infrastructure, chips & efficiency for advanced learners
86 resources · page 2 of 3
NVIDIA Dynamo
by Dynamo
ai-dynamo Dynamo GitHub repo: NVIDIA's datacenter-scale distributed, disaggregated inference serving framework.
NVIDIA NeMo Speech
by NVIDIA-NeMo
NVIDIA NeMo Speech GitHub repo: framework for speech recognition and TTS model training and inference.
NVIDIA/TensorRT-LLM
NVIDIA TensorRT-LLM GitHub repo: Python API and kernels for optimised LLM inference on NVIDIA GPUs.
NVIDIA VerilogEval
NVIDIA VerilogEval GitHub repo: benchmark for evaluating LLMs on generating Verilog hardware-description code.
Ollama
by Ollama
Ollama GitHub repo: runtime for running open LLMs locally, with API and cloud options.
OLMo training and inference
by Ai2
Ai2 OLMo GitHub repo: modelling, training, evaluation and inference code for OLMo open language models.
OpenROAD
OpenROAD GitHub repo: open RTL-to-GDS chip design flow, also used in ML-for-EDA research.
SageAttention
by TSAIL group
Tsinghua SageAttention GitHub repo: quantised low-bit attention kernels for faster inference.
onediff
by SiliconFlow
SiliconFlow onediff GitHub repo: acceleration library for diffusion model inference.
Tenstorrent
by Tenstorrent
Tenstorrent: company building RISC-V based AI accelerators and computers.
ThunderKittens
by HazyResearch
HazyResearch ThunderKittens GitHub repo: tile primitives framework for writing fast GPU kernels.
TileKernels
by DeepSeek
DeepSeek TileKernels GitHub repo: library of GPU kernels written in TileLang.
Together cookbook
by Together
Together cookbook GitHub repo: notebooks with recipes for building with Together AI models.
torchtitan
by pytorch
PyTorch torchtitan GitHub repo: PyTorch-native platform for large-scale generative model pretraining.
fairseq: NLLB branch
Meta fairseq nllb branch on GitHub: code for the No Language Left Behind translation models.
Triton-distributed
ByteDance Seed Triton-distributed GitHub repo: distributed compiler and parallel kernels overlapping compute and communication.
TurboDiffusion (thu-ml)
by TSAIL group
Tsinghua TurboDiffusion GitHub repo: acceleration framework for video diffusion models.
GPU Glossary (Modal)
by Modal
Modal GPU Glossary: reference explaining GPU hardware and software terms from CUDA cores to memory hierarchy.
AMD ROCm
by AMD
AMD ROCm developer hub with documentation, training and tools for GPU-accelerated computing on AMD hardware.
Google TPU docs
by Google Cloud
Google Cloud TPU documentation for running machine-learning training and inference on Tensor Processing Units.
GPU MODE
GPU MODE site: community for learning GPU programming through lectures, hackathons, kernel competitions and working groups.
GPU MODE lectures
by GPU MODE
GPU MODE lectures GitHub repo: slides and code from community lectures on GPU programming and CUDA kernels.
vLLM documentation
by vLLM project
vLLM documentation: open-source engine for high-throughput LLM inference and serving.
PyTorch FSDP tutorial
PyTorch tutorial on Fully Sharded Data Parallel (FSDP2) for training large models across GPUs.
SGLang documentation
SGLang documentation: serving framework for large language and multimodal models.
Triton tutorials
Triton tutorials: ordered examples from vector addition to fused softmax, matmul and fused attention GPU kernels.
A graph placement methodology for fast chip design
by Azalia Mirhoseini et al. (Google)
Nature paper (2021) by Mirhoseini et al. introducing the reinforcement learning chip floorplanning method later named AlphaChip.
Agentic inference optimization (Baseten)
by Baseten
Baseten post on optimising inference engines for agent workloads, reporting decode-speed and latency gains over vLLM.
Baseten documentation
by Baseten
Baseten documentation for running hosted models, deploying custom models and training on GPU infrastructure.
BentoML documentation
by BentoML
BentoML documentation: open-source inference platform for packaging, deploying and scaling AI models.
Code execution with MCP
by Anthropic
Anthropic post on code execution with MCP: agents write code to call tools, reducing token use and context load.
Defeating Nondeterminism in LLM Inference
Thinking Machines Lab post on the causes of nondeterminism in LLM inference and batch-invariant kernels to fix it.