training systems: 13 curated resources
training systems
13 resources · page 1 of 1
Colossal-AI
by HPC-AI Tech
HPC-AI Tech Colossal-AI GitHub repo: parallel training system for large AI models.
DeepGEMM-Ascend
by DeepSeek
DeepSeek DeepGEMM-Ascend GitHub repo: matrix multiplication kernels for Huawei Ascend hardware.
DeepSpeed
by deepspeedai
DeepSpeed GitHub repo: deep learning optimisation library for large-scale training and inference, origin of ZeRO.
FlashMLA
by DeepSeek
DeepSeek FlashMLA GitHub repo: multi-head latent attention decoding kernels for Hopper GPUs.
FlashQLA
by Qwen
QwenLM FlashQLA GitHub repo: linear attention kernel library built on TileLang.
Megatron-LM
NVIDIA Megatron-LM GitHub repo: research code for training transformer models at scale with tensor and pipeline parallelism.
TileKernels
by DeepSeek
DeepSeek TileKernels GitHub repo: library of GPU kernels written in TileLang.
torchtitan
by pytorch
PyTorch torchtitan GitHub repo: PyTorch-native platform for large-scale generative model pretraining.
Triton-distributed
ByteDance Seed Triton-distributed GitHub repo: distributed compiler and parallel kernels overlapping compute and communication.
TurboDiffusion (thu-ml)
by TSAIL group
Tsinghua TurboDiffusion GitHub repo: acceleration framework for video diffusion models.
PyTorch FSDP tutorial
PyTorch tutorial on Fully Sharded Data Parallel (FSDP2) for training large models across GPUs.
Tinker documentation
Tinker documentation from Thinking Machines Lab: training API for fine-tuning open-weight LLMs, with SDK reference and recipes.
FlashAttention
by Tri Dao et al. (Stanford University)
arXiv paper (2022) introducing FlashAttention, an IO-aware exact attention algorithm for GPUs.