You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

MPI Benchmarks

The following uses nodecomm to measure the bandwidth between 2 nodes, with 1-32 processes on each node communicating with its partner on the other node.


TensorFlow Benchmarks


The figure above shows effect of increasing batch size on training rate. Trying to run Resnet 50 v1.5 with batch size 256 using fp32 for calculations results in OOM, which is why that result is omitted.

The figure above shows effect of increasing number of gpus on training rate with fixed batch size for both real (blue) and synthetic (orange) data.


A broader set of results can be seen in the following table:

model

num_gpus

batch_size

use_fp16

use_synth

img_per_sec

resnet50_v1.5

1

32

False

False

305.1581998762749

resnet50_v1.5

1

32

False

True

305.0756935566205

resnet50_v1.5

1

32

True

False

557.608344009375

resnet50_v1.5

1

32

True

True

551.7856108469189

resnet50_v1.5

1

64

False

False

345.86744731677936

resnet50_v1.5

1

64

False

True

343.33576731867544

resnet50_v1.5

1

64

True

False

687.2315721842947

resnet50_v1.5

1

64

True

True

676.3865560680504

resnet50_v1.5

1

128

False

False

365.42804577993536

resnet50_v1.5

1

128

False

True

361.9197194570446

resnet50_v1.5

1

128

True

False

780.5401907002691

resnet50_v1.5

1

128

True

True

787.9725710218387

resnet50_v1.5

1

256

True

False

827.7779526071831

resnet50_v1.5

1

256

True

True

838.8718495151428

resnet50_v1.5

2

32

False

False

606.6988150458972

resnet50_v1.5

2

32

False

True

611.4476119199953

resnet50_v1.5

2

32

True

False

1093.3154950403268

resnet50_v1.5

2

32

True

True

1099.2094967673627

resnet50_v1.5

2

64

False

False

684.8824289499631

resnet50_v1.5

2

64

False

True

682.3657230029216

resnet50_v1.5

2

64

True

False

1337.7101641206896

resnet50_v1.5

2

64

True

True

1356.9406586722887

resnet50_v1.5

2

128

False

False

727.4366020878481

resnet50_v1.5

2

128

False

True

714.9118280128031

resnet50_v1.5

2

128

True

False

1520.3306305011697

resnet50_v1.5

2

128

True

True

1567.667541866167

resnet50_v1.5

2

256

True

False

1663.5347160145286

resnet50_v1.5

2

256

True

True

1685.7265358502561

resnet50_v1.5

4

32

False

False

1099.8515970079466

resnet50_v1.5

4

32

False

True

1158.8182004104326

resnet50_v1.5

4

32

True

False

1806.7010872696358

resnet50_v1.5

4

32

True

True

2140.346448732058

resnet50_v1.5

4

64

False

False

1263.7396753889122

resnet50_v1.5

4

64

False

True

1323.4508475425275

resnet50_v1.5

4

64

True

False

2132.559146072129

resnet50_v1.5

4

64

True

True

2630.0377706917334

resnet50_v1.5

4

128

False

False

1382.0710902623055

resnet50_v1.5

4

128

False

True

1414.0828353841684

resnet50_v1.5

4

128

True

False

2557.1778582125

resnet50_v1.5

4

128

True

True

3065.898543897684

resnet50_v1.5

4

256

True

False

2864.1530570235277

resnet50_v1.5

4

256

True

True

3308.184721599039

inception3

1

32

False

False

231.6304999196131

inception3

1

32

False

True

231.62821340833554

inception3

1

32

True

False

397.8483574516062

inception3

1

32

True

True

397.61870298268843

inception3

1

64

False

False

255.79774049318286

inception3

1

64

False

True

254.96217612362278

inception3

1

64

True

False

464.3570055911474

inception3

1

64

True

True

466.5224032998034

inception3

1

128

False

False

265.4893643297776

inception3

1

128

False

True

265.9010072543052

inception3

1

128

True

False

523.6175139095815

inception3

1

128

True

True

529.3233120661808

inception3

1

256

True

False

548.3831188132465

inception3

1

256

True

True

558.6182587782852

inception3

2

32

False

False

461.21245811536164

inception3

2

32

False

True

461.7955449394074

inception3

2

32

True

False

776.9118181766349

inception3

2

32

True

True

803.1606840744608

inception3

2

64

False

False

505.62156764514185

inception3

2

64

False

True

510.4880571128736

inception3

2

64

True

False

910.3834390431858

inception3

2

64

True

True

941.0765499887631

inception3

2

128

False

False

530.073420084333

inception3

2

128

False

True

532.0966824977393

inception3

2

128

True

False

1031.042745566893

inception3

2

128

True

True

1059.298960623237

inception3

2

256

True

False

1097.3179582016833

inception3

2

256

True

True

1112.944630855092

inception3

4

32

False

False

846.1137665298596

inception3

4

32

False

True

895.5092904990373

inception3

4

32

True

False

1271.1921617960843

inception3

4

32

True

True

1562.0603997601588

inception3

4

64

False

False

967.953751912867

inception3

4

64

False

True

994.9078067162436

inception3

4

64

True

False

1513.182913280423

inception3

4

64

True

True

1842.119923117435

inception3

4

128

False

False

992.0283404721883

inception3

4

128

False

True

1027.2404515646604

inception3

4

128

True

False

1751.5253071397437

inception3

4

128

True

True

2089.504434176232

inception3

4

256

True

False

1992.6604934734398

inception3

4

256

True

True

2174.329529373497

  • No labels