DeepSpeed For Deep Learning Training Course

Overview

DeepSpeed is a deep learning optimization library that makes it easier to scale deep learning models on distributed hardware. Developed by Microsoft, DeepSpeed integrates with PyTorch to provide better scaling, faster training, and improved resource utilization. 

This instructor-led, live training (online or onsite) is aimed at beginner to intermediate-level data scientists and machine learning engineers who wish to improve the performance of their deep learning models.

By the end of this training, participants will be able to:

• Understand the principles of distributed deep learning.

• Install and configure DeepSpeed.

• Scale deep learning models on distributed hardware using DeepSpeed.

• Implement and experiment with DeepSpeed features for optimization and memory efficiency.

Format of the Course

• Interactive lecture and discussion.

• Lots of exercises and practice.

• Hands-on implementation in a live-lab environment.

Course Customization Options

• To request a customized training for this course, please contact us to arrange.

Course Outline

Introduction

• Overview of deep learning scaling challenges

• Overview of DeepSpeed and its features

• DeepSpeed vs. other distributed deep learning libraries

Getting Started

• Setting up the development environment

• Installing PyTorch and DeepSpeed

• Configuring DeepSpeed for distributed training

DeepSpeed Optimization Features

• DeepSpeed training pipeline

• ZeRO (memory optimization)

• Activation checkpointing

• Gradient checkpointing

• Pipeline parallelism

Scaling Models with DeepSpeed

• Basic scaling using DeepSpeed

• Advanced scaling techniques

• Performance considerations and best practices

• Debugging and troubleshooting techniques

Advanced DeepSpeed Topics

• Advanced optimization techniques

• Using DeepSpeed with mixed precision training

• DeepSpeed on different hardware (e.g. GPUs, TPUs)

• DeepSpeed with multiple training nodes

Integrating DeepSpeed with PyTorch

• Integrating DeepSpeed with PyTorch workflows

• Using DeepSpeed with PyTorch Lightning

Troubleshooting

• Debugging common DeepSpeed issues

• Monitoring and logging

Summary and Next Steps

Requirements

• Intermediate knowledge of deep learning principles

• Experience with PyTorch or similar deep learning frameworks

• Familiarity with Python programming

Audience

• Data scientists

• Machine learning engineers

• Developers

• Recap of key concepts and features

• Best practices for using DeepSpeed in production

• Further resources for learning more about DeepSpeed