---
id: "/tags/training-systems/"
title: "Training Systems"
creator: null
creator_url: null
creator_x: null
url: "https://skillissue.dev/tags/training-systems/"
level: null
level_max: null
category: null
subcategory: null
region: null
license: null
license_scope: null
access: null
models: null
source_date: null
date_added: null
resource_kind: null
verification_basis: null
access_evidence: null
domain_top: null
domain_sub: null
last_verified: null
---

# Training Systems

[Beginner](/levels/beginner/index.md) [Intermediate](/levels/intermediate/index.md) [Advanced](/levels/advanced/index.md) [Expert](/levels/expert/index.md)

13 resources · page 1 of 1

- 

### [Colossal-AI](/model-training/training-infrastructure/colossal-ai/index.md)

Creator not supplied; see original source.

HPC-AI Tech Colossal-AI GitHub repo: parallel training system for large AI models.

Not applicable (entity) · unknown · unknown resource · en|zh 

- 

### [DeepGEMM-Ascend](/model-training/training-infrastructure/deepgemm-ascend/index.md)

Creator not supplied; see original source.

DeepSeek DeepGEMM-Ascend GitHub repo: matrix multiplication kernels for Huawei Ascend hardware.

Not applicable (entity) · unknown · unknown resource · en 

- 

### [DeepSpeed](/model-training/training-infrastructure/deepspeed/index.md)

Creator not supplied; see original source.

DeepSpeed GitHub repo: deep learning optimisation library for large-scale training and inference, origin of ZeRO.

Not applicable (entity) · unknown · unknown resource · en 

- 

### [FlashMLA](/model-training/training-infrastructure/flashmla/index.md)

Creator not supplied; see original source.

DeepSeek FlashMLA GitHub repo: multi-head latent attention decoding kernels for Hopper GPUs.

Not applicable (entity) · unknown · unknown resource · en 

- 

### [FlashQLA](/model-training/training-infrastructure/flashqla/index.md)

Creator not supplied; see original source.

QwenLM FlashQLA GitHub repo: linear attention kernel library built on TileLang.

Not applicable (entity) · unknown · unknown resource · en 

- 

### [Megatron-LM](/model-training/training-infrastructure/megatron-lm/index.md)

Creator not supplied; see original source.

NVIDIA Megatron-LM GitHub repo: research code for training transformer models at scale with tensor and pipeline parallelism.

Not applicable (entity) · unknown · unknown resource · en 

- 

### [TileKernels](/model-training/training-infrastructure/tilekernels/index.md)

Creator not supplied; see original source.

DeepSeek TileKernels GitHub repo: library of GPU kernels written in TileLang.

Not applicable (entity) · unknown · unknown resource · en 

- 

### [torchtitan](/model-training/training-infrastructure/torchtitan/index.md)

Creator not supplied; see original source.

PyTorch torchtitan GitHub repo: PyTorch-native platform for large-scale generative model pretraining.

Not applicable (entity) · unknown · unknown resource · en 

- 

### [Triton-distributed](/model-training/training-infrastructure/triton-distributed/index.md)

Creator not supplied; see original source.

ByteDance Seed Triton-distributed GitHub repo: distributed compiler and parallel kernels overlapping compute and communication.

Not applicable (entity) · unknown · unknown resource · en 

- 

### [TurboDiffusion (thu-ml)](/model-training/training-infrastructure/turbodiffusion-thu-ml/index.md)

Creator not supplied; see original source.

Tsinghua TurboDiffusion GitHub repo: acceleration framework for video diffusion models.

Not applicable (entity) · unknown · unknown resource · en 

- 

### [PyTorch FSDP tutorial](/model-training/training-infrastructure/pytorch-fsdp-tutorial/index.md)

Creator not supplied; see original source.

PyTorch tutorial on Fully Sharded Data Parallel (FSDP2) for training large models across GPUs.

Intermediate–Advanced · unknown · unknown resource · en 

- 

### [Tinker documentation](/model-training/training-infrastructure/tinker-documentation/index.md)

Creator not supplied; see original source.

Tinker documentation from Thinking Machines Lab: training API for fine-tuning open-weight LLMs, with SDK reference and recipes.

Advanced · unknown · unknown resource · en 

- 

### [FlashAttention](/model-training/training-infrastructure/flashattention/index.md)

Creator not supplied; see original source.

arXiv paper (2022) introducing FlashAttention, an IO-aware exact attention algorithm for GPUs.

Expert · unknown · unknown resource · en 

[Plain-text Markdown copy](/tags/training-systems/index.md)