Training Systems

13 resources · page 1 of 1

• Colossal-AI

  Creator not supplied; see original source.

  HPC-AI Tech Colossal-AI GitHub repo: parallel training system for large AI models.

  Not applicable (entity) · unknown · unknown resource · en|zh

• DeepGEMM-Ascend

  Creator not supplied; see original source.

  DeepSeek DeepGEMM-Ascend GitHub repo: matrix multiplication kernels for Huawei Ascend hardware.

  Not applicable (entity) · unknown · unknown resource · en

• DeepSpeed

  Creator not supplied; see original source.

  DeepSpeed GitHub repo: deep learning optimisation library for large-scale training and inference, origin of ZeRO.

  Not applicable (entity) · unknown · unknown resource · en

• FlashMLA

  Creator not supplied; see original source.

  DeepSeek FlashMLA GitHub repo: multi-head latent attention decoding kernels for Hopper GPUs.

  Not applicable (entity) · unknown · unknown resource · en

• FlashQLA

  Creator not supplied; see original source.

  QwenLM FlashQLA GitHub repo: linear attention kernel library built on TileLang.

  Not applicable (entity) · unknown · unknown resource · en

• Megatron-LM

  Creator not supplied; see original source.

  NVIDIA Megatron-LM GitHub repo: research code for training transformer models at scale with tensor and pipeline parallelism.

  Not applicable (entity) · unknown · unknown resource · en

• TileKernels

  Creator not supplied; see original source.

  DeepSeek TileKernels GitHub repo: library of GPU kernels written in TileLang.

  Not applicable (entity) · unknown · unknown resource · en

• torchtitan

  Creator not supplied; see original source.

  PyTorch torchtitan GitHub repo: PyTorch-native platform for large-scale generative model pretraining.

  Not applicable (entity) · unknown · unknown resource · en

• Triton-distributed

  Creator not supplied; see original source.

  ByteDance Seed Triton-distributed GitHub repo: distributed compiler and parallel kernels overlapping compute and communication.

  Not applicable (entity) · unknown · unknown resource · en

• TurboDiffusion (thu-ml)

  Creator not supplied; see original source.

  Tsinghua TurboDiffusion GitHub repo: acceleration framework for video diffusion models.

  Not applicable (entity) · unknown · unknown resource · en

• PyTorch FSDP tutorial

  Creator not supplied; see original source.

  PyTorch tutorial on Fully Sharded Data Parallel (FSDP2) for training large models across GPUs.

  Intermediate–Advanced · unknown · unknown resource · en

• Tinker documentation

  Creator not supplied; see original source.

  Tinker documentation from Thinking Machines Lab: training API for fine-tuning open-weight LLMs, with SDK reference and recipes.

  Advanced · unknown · unknown resource · en

• FlashAttention

  Creator not supplied; see original source.

  arXiv paper (2022) introducing FlashAttention, an IO-aware exact attention algorithm for GPUs.

  Expert · unknown · unknown resource · en

Plain-text Markdown copy