DeepSpeed

What is DeepSpeed?

DeepSpeed is a system optimization library from Microsoft for large-scale deep learning training. It is part of the Microsoft AI at Scale initiative.

Key capabilities of DeepSpeed

• ZeRO: Memory optimization that cuts redundancy across distributed training.
• ZeRO-Offload and ZeRO-Infinity: Push memory limits using CPU memory.
• Offloading: Train billion-scale models on limited GPU resources.
• 3D parallelism: Splits work across three dimensions of compute.
• Mixture of Experts: MoE training support.
• Inference optimization: Efficient deployment of transformer models.

How DeepSpeed works

Developers wrap an existing PyTorch training script with DeepSpeed, which partitions model states across GPUs and optionally offloads parameters and computation to CPU memory. 3D parallelism and MoE spread the model across devices. Inference tools serve transformer models at scale.

Who uses DeepSpeed?

DeepSpeed is used by machine learning engineers and research labs training very large models. The vendor lists Megatron-Turing NLG 530B, BLOOM 176B, GLM 130B and YaLM 100B as models trained with it.

DeepSpeed pricing

DeepSpeed is open source and free to use. Compute for training and inference is the main cost.

DeepSpeed alternatives

Alternatives include PyTorch FSDP, which shards training in PyTorch, Megatron-LM, which targets large transformer training, and Hugging Face Accelerate, which simplifies distributed training.