Infrastructure, chips & efficiency for expert learners
59 resources · page 2 of 2
NVIDIA Dynamo
by Dynamo
ai-dynamo Dynamo GitHub repo: NVIDIA's datacenter-scale distributed, disaggregated inference serving framework.
NVIDIA NeMo Speech
by NVIDIA-NeMo
NVIDIA NeMo Speech GitHub repo: framework for speech recognition and TTS model training and inference.
NVIDIA/TensorRT-LLM
NVIDIA TensorRT-LLM GitHub repo: Python API and kernels for optimised LLM inference on NVIDIA GPUs.
NVIDIA VerilogEval
NVIDIA VerilogEval GitHub repo: benchmark for evaluating LLMs on generating Verilog hardware-description code.
Ollama
by Ollama
Ollama GitHub repo: runtime for running open LLMs locally, with API and cloud options.
OLMo training and inference
by Ai2
Ai2 OLMo GitHub repo: modelling, training, evaluation and inference code for OLMo open language models.
OpenROAD
OpenROAD GitHub repo: open RTL-to-GDS chip design flow, also used in ML-for-EDA research.
SageAttention
by TSAIL group
Tsinghua SageAttention GitHub repo: quantised low-bit attention kernels for faster inference.
onediff
by SiliconFlow
SiliconFlow onediff GitHub repo: acceleration library for diffusion model inference.
Tenstorrent
by Tenstorrent
Tenstorrent: company building RISC-V based AI accelerators and computers.
ThunderKittens
by HazyResearch
HazyResearch ThunderKittens GitHub repo: tile primitives framework for writing fast GPU kernels.
TileKernels
by DeepSeek
DeepSeek TileKernels GitHub repo: library of GPU kernels written in TileLang.
Together cookbook
by Together
Together cookbook GitHub repo: notebooks with recipes for building with Together AI models.
torchtitan
by pytorch
PyTorch torchtitan GitHub repo: PyTorch-native platform for large-scale generative model pretraining.
fairseq: NLLB branch
Meta fairseq nllb branch on GitHub: code for the No Language Left Behind translation models.
Triton-distributed
ByteDance Seed Triton-distributed GitHub repo: distributed compiler and parallel kernels overlapping compute and communication.
TurboDiffusion (thu-ml)
by TSAIL group
Tsinghua TurboDiffusion GitHub repo: acceleration framework for video diffusion models.
GPU Glossary (Modal)
by Modal
Modal GPU Glossary: reference explaining GPU hardware and software terms from CUDA cores to memory hierarchy.
CMU 11-868 LLM Systems
by llmsystem
CMU 11-868 LLM Systems course site: systems for training and serving large language models, with schedule and readings.
CUDA C++ Programming Guide
by NVIDIA
NVIDIA CUDA C++ Programming Guide: reference for the CUDA programming model, memory hierarchy and interfaces.
EAGLE
arXiv paper (2024) introducing EAGLE, speculative sampling with feature-level draft heads.
Fast Inference from Transformers via Speculative Decoding
by Yaniv Leviathan et al. (Google)
arXiv paper (2022) introducing speculative decoding to speed up transformer inference without changing outputs.
FlashAttention
by Tri Dao et al. (Stanford University)
arXiv paper (2022) introducing FlashAttention, an IO-aware exact attention algorithm for GPUs.
How to Scale Your Model (JAX scaling book)
by jax-ml
How to Scale Your Model: free online book on TPU/GPU rooflines, transformer math, parallelism and serving for LLMs.
PagedAttention / vLLM paper
by Woosuk Kwon et al. (UC Berkeley)
arXiv paper (2023) introducing PagedAttention and the vLLM serving system for efficient KV-cache memory management.
SGLang docs
SGLang documentation: LLM serving framework with RadixAttention prefix caching and fast structured outputs.
How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance (Simon Boehm)
by Simon Boehm
Simon Boehm's worklog optimising a CUDA matrix-multiplication kernel step by step towards cuBLAS performance.