Evaluating Distributed Model Training using PyTorch APIs

This repo contains the necessary code and bash scripts used for evaluating PyTorch's built-in distributed training module.

Modules covered:

• Fully Sharded Data Parallel (FSDP)
• Pipeline Parallel (PP)

Setup

Four nodes, each consisting of a single GPU, Nvidia 1650, with 4GB VRAM.