--- id: "/infrastructure-chips-efficiency/training-systems/" title: "training systems: 13 curated resources" url: "https://skillissue.dev/infrastructure-chips-efficiency/training-systems/" --- # training systems: 13 curated resources training systems [Beginner](/infrastructure-chips-efficiency/beginner/index.md) [Intermediate](/infrastructure-chips-efficiency/intermediate/index.md) [Advanced](/infrastructure-chips-efficiency/advanced/index.md) [Expert](/infrastructure-chips-efficiency/expert/index.md) 13 resources · page 1 of 1 - ### [Colossal-AI](/model-training/training-infrastructure/colossal-ai/index.md) by [HPC-AI Tech](https://hpc-ai.com/) HPC-AI Tech Colossal-AI GitHub repo: parallel training system for large AI models. entity repo China/Singapore en|zh - ### [DeepGEMM-Ascend](/model-training/training-infrastructure/deepgemm-ascend/index.md) by [DeepSeek](https://www.deepseek.com/) DeepSeek DeepGEMM-Ascend GitHub repo: matrix multiplication kernels for Huawei Ascend hardware. entity repo China en - ### [DeepSpeed](/model-training/training-infrastructure/deepspeed/index.md) by [deepspeedai](https://github.com/deepspeedai) DeepSpeed GitHub repo: deep learning optimisation library for large-scale training and inference, origin of ZeRO. entity repo US en - ### [FlashMLA](/model-training/training-infrastructure/flashmla/index.md) by [DeepSeek](https://www.deepseek.com/) DeepSeek FlashMLA GitHub repo: multi-head latent attention decoding kernels for Hopper GPUs. entity repo China en - ### [FlashQLA](/model-training/training-infrastructure/flashqla/index.md) by [Qwen](https://qwen.ai/) QwenLM FlashQLA GitHub repo: linear attention kernel library built on TileLang. entity repo China en - ### [Megatron-LM](/model-training/training-infrastructure/megatron-lm/index.md) by [NVIDIA Corporation](https://nvidia.com) NVIDIA Megatron-LM GitHub repo: research code for training transformer models at scale with tensor and pipeline parallelism. entity repo US en - ### [TileKernels](/model-training/training-infrastructure/tilekernels/index.md) by [DeepSeek](https://www.deepseek.com/) DeepSeek TileKernels GitHub repo: library of GPU kernels written in TileLang. entity repo China en - ### [torchtitan](/model-training/training-infrastructure/torchtitan/index.md) by [pytorch](https://pytorch.org) PyTorch torchtitan GitHub repo: PyTorch-native platform for large-scale generative model pretraining. entity repo US en - ### [Triton-distributed](/model-training/training-infrastructure/triton-distributed/index.md) by [ByteDance-Seed](https://seed.bytedance.com/) ByteDance Seed Triton-distributed GitHub repo: distributed compiler and parallel kernels overlapping compute and communication. entity repo China en - ### [TurboDiffusion (thu-ml)](/model-training/training-infrastructure/turbodiffusion-thu-ml/index.md) by [TSAIL group](https://ml.cs.tsinghua.edu.cn) Tsinghua TurboDiffusion GitHub repo: acceleration framework for video diffusion models. entity repo China en - ### [PyTorch FSDP tutorial](/model-training/training-infrastructure/pytorch-fsdp-tutorial/index.md) by [PyTorch Foundation](https://pytorch.org/) PyTorch tutorial on Fully Sharded Data Parallel (FSDP2) for training large models across GPUs. Intermediate–Advanced learning_resource docs US en - ### [Tinker documentation](/model-training/training-infrastructure/tinker-documentation/index.md) by [Thinking Machines Lab](https://thinkingmachines.ai/) Tinker documentation from Thinking Machines Lab: training API for fine-tuning open-weight LLMs, with SDK reference and recipes. Advanced learning_resource docs US en - ### [FlashAttention](/model-training/training-infrastructure/flashattention/index.md) by [Tri Dao et al. (Stanford University)](https://www.stanford.edu/) arXiv paper (2022) introducing FlashAttention, an IO-aware exact attention algorithm for GPUs. Expert learning_resource paper US en [Plain-text Markdown copy](/infrastructure-chips-efficiency/training-systems/index.md)