MPI Benchmarks

The following uses nodecomm to measure the bandwidth between 2 nodes, with 1-32 processes on each node communicating with its partner on the other node.


TensorFlow Benchmarks


 

The figure above shows effect of increasing batch size on training rate. Trying to run Resnet 50 v1.5 with batch size 256 using fp32 for calculations results in OOM, which is why that result is omitted.

The figure above shows effect of increasing number of gpus on training rate with fixed batch size for both real (blue) and synthetic (orange) data.


A broader set of results can be seen in the following table, with results accumulated from 5 separate trials:

model

num_gpus

batch_size

use_fp16

use_synth

avg img_per_sec

std dev img_per_sec

resnet50_v1.5

1

32

False

False

300.7115095987747

0.8334252017088293

resnet50_v1.5

1

32

False

True

303.2748894498352

0.518768277442615

resnet50_v1.5

1

32

True

False

552.8543259023911

3.7419280205833876

resnet50_v1.5

1

32

True

True

551.3152982879091

1.9256151492887408

resnet50_v1.5

1

64

False

False

340.3455581202521

0.4352822822221454

resnet50_v1.5

1

64

False

True

341.4178425579963

1.399820974317093

resnet50_v1.5

1

64

True

False

679.092793370331

1.811493519714515

resnet50_v1.5

1

64

True

True

679.4220076333565

0.4846137767752205

resnet50_v1.5

1

128

False

False

365.97794971142156

0.6660901988191761

resnet50_v1.5

1

128

False

True

365.1002182169415

0.290659074503093

resnet50_v1.5

1

128

True

False

780.7795427908641

2.6289190299425735

resnet50_v1.5

1

128

True

True

782.0787833811031

1.4035397167521415

resnet50_v1.5

1

256

True

False

841.0209098895596

0.9327257479352289

resnet50_v1.5

1

256

True

True

841.6281134422322

0.6754027617726148

resnet50_v1.5

2

32

False

False

597.7123015353376

2.5358048324228486

resnet50_v1.5

2

32

False

True

602.1851277710804

1.4932197457537746

resnet50_v1.5

2

32

True

False

1092.0792847503365

5.317212492608127

resnet50_v1.5

2

32

True

True

1101.2976316254073

3.1791943633783832

resnet50_v1.5

2

64

False

False

680.4331810776715

1.1986421991311502

resnet50_v1.5

2

64

False

True

680.168662439279

1.628589727697411

resnet50_v1.5

2

64

True

False

1350.6915781119133

4.771936223937959

resnet50_v1.5

2

64

True

True

1337.9212057737357

4.3318244554829635

resnet50_v1.5

2

128

False

False

722.1292515998915

1.7079082542095976

resnet50_v1.5

2

128

False

True

724.5907650506392

1.3016121917621204

resnet50_v1.5

2

128

True

False

1560.6063143166916

5.574390740438405

resnet50_v1.5

2

128

True

True

1558.8925541664316

3.7645037153069234

resnet50_v1.5

2

256

True

False

1681.9367020855304

2.135490739835972

resnet50_v1.5

2

256

True

True

1679.1043258972677

1.9428881159794065

resnet50_v1.5

4

32

False

False

1137.8151032757787

3.6070602577235196

resnet50_v1.5

4

32

False

True

1147.7653259816507

10.086122200344256

resnet50_v1.5

4

32

True

False

2090.181533002181

27.753203322924346

resnet50_v1.5

4

32

True

True

2118.951749390374

9.671318775529995

resnet50_v1.5

4

64

False

False

1323.4940363814712

2.4288696630804347

resnet50_v1.5

4

64

False

True

1336.5457321023537

2.8028756633168586

resnet50_v1.5

4

64

True

False

2627.208652367316

30.590239282719534

resnet50_v1.5

4

64

True

True

2605.2372571940177

16.77388699379653

resnet50_v1.5

4

128

False

False

1428.9004479146274

5.19872681737769

resnet50_v1.5

4

128

False

True

1431.0649693247092

2.1140696182827456

resnet50_v1.5

4

128

True

False

3038.123973486709

32.14983993002834

resnet50_v1.5

4

128

True

True

3050.6240630834754

13.087954923831944

resnet50_v1.5

4

256

True

False

3269.060930519893

18.13964231626883

resnet50_v1.5

4

256

True

True

3288.0027951477073

10.833033249948164

inception3

1

32

False

False

229.6400062758412

1.011150139067665

inception3

1

32

False

True

231.35776988245647

0.7390853739290063

inception3

1

32

True

False

390.8430427702688

1.5232497382000254

inception3

1

32

True

True

392.77650557982815

0.7988964068960409

inception3

1

64

False

False

254.08237225221546

0.7822494442939251

inception3

1

64

False

True

254.54280397180582

0.9677860725157185

inception3

1

64

True

False

459.27229023783576

1.0561819758930817

inception3

1

64

True

True

465.8370224010511

1.9406403927259364

inception3

1

128

False

False

266.6960069340822

0.6349003776029812

inception3

1

128

False

True

267.3287620209714

0.3378865479524414

inception3

1

128

True

False

522.7059605744032

1.2095275730192268

inception3

1

128

True

True

528.411119339248

1.886616179173554

inception3

1

256

True

False

555.9733275297333

0.793152246051491

inception3

1

256

True

True

555.8605933224619

1.0050188430980673

inception3

2

32

False

False

453.8981275778566

1.4942589199911878

inception3

2

32

False

True

460.07629649023164

2.572961501771087

inception3

2

32

True

False

785.3002107622603

2.808072532884698

inception3

2

32

True

True

794.9652159658681

3.612256492187289

inception3

2

64

False

False

504.9115747109531

0.9842426876013127

inception3

2

64

False

True

505.8260905072816

1.4738932419714106

inception3

2

64

True

False

920.4145181495171

3.1651197427361715

inception3

2

64

True

True

929.0658731171085

5.145909922759968

inception3

2

128

False

False

531.5809808901386

4.152618166592716

inception3

2

128

False

True

531.3011081204222

3.821408673513115

inception3

2

128

True

False

1043.3751588125951

6.360344252646946

inception3

2

128

True

True

1052.1427785292055

0.8095010705701331

inception3

2

256

True

False

1108.9748739263916

9.386603241427991

inception3

2

256

True

True

1113.16626158571

6.4319767570544535

inception3

4

32

False

False

872.3219766840209

4.856647166737354

inception3

4

32

False

True

888.3055054654682

2.5289892419845943

inception3

4

32

True

False

1495.0846768394943

10.965843253024328

inception3

4

32

True

True

1539.5491706395067

9.667722873324415

inception3

4

64

False

False

992.8041857703738

2.4128076229880566

inception3

4

64

False

True

999.2568604412305

2.9532997352535655

inception3

4

64

True

False

1790.7335622916307

12.568524155858082

inception3

4

64

True

True

1813.338802302623

1.7710531654085244

inception3

4

128

False

False

1044.764217747176

6.219768494743513

inception3

4

128

False

True

1041.9846289297782

16.16776883872252

inception3

4

128

True

False

2045.2687789799281

15.112275686406946

inception3

4

128

True

True

2063.858653134279

8.19186760727177

inception3

4

256

True

False

2186.246302564507

32.645185806377874

inception3

4

256

True

True

2194.476607046693

20.772815631509243

  • No labels