Source code: https://github.com/richardkxu/distributed-pytorch
We will cover the following training methods for PyTorch:
torch.nn.DataParallel
torch.nn.DistributedDataParallel
Apex
TensorBoard
logging under distributed training contextWe will cover the following use cases: