Versions Compared

Old Version 3

changes.mady.by.user Benjamin Rabe

Saved on

compared with

New Version Current

changes.mady.by.user Dawei Mu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The following uses nodecomm to measure the bandwidth between 2 nodes, with 1-32 processes on each node communicating with its partner on the other node.


TensorFlow Benchmarks


 

The figure above shows effect of increasing batch size on training rate. Trying to run Resnet 50 v1.5 with batch size 256 using fp32 for calculations results in OOM, which is why that result is omitted.

Image RemovedImage Added

The figure above shows effect of increasing number of gpus on training rate with fixed batch size for both real (blue) and synthetic (orange) data.

...

A broader set of results can be seen in the following table, with results accumulated from 5 separate trials:

model

num_gpus

batch_size

use_fp16

use_synth

avg img_per_sec

std dev img_per_sec

resnet50_v1.5

1

32

False

False

300.7115095987747

0.8334252017088293305.1581998762749

resnet50_v1.5

1

32

False

True

303.2748894498352

0.518768277442615305.0756935566205

resnet50_v1.5

1

32

True

False

552.8543259023911

5573.6083440093757419280205833876

resnet50_v1.5

1

32

True

True

551.3152982879091

1.78561084691899256151492887408

resnet50_v1.5

1

64

False

False

340.3455581202521

3450.867447316779364352822822221454

resnet50_v1.5

1

64

False

True

341.4178425579963

1.399820974317093343.33576731867544

resnet50_v1.5

1

64

True

False

679.092793370331

6871.2315721842947811493519714515

resnet50_v1.5

1

64

True

True

679.4220076333565

0.4846137767752205676.3865560680504

resnet50_v1.5

1

128

False

False

365.4280457799353697794971142156

0.6660901988191761

resnet50_v1.5

1

128

False

True

365.1002182169415

3610.9197194570446290659074503093

resnet50_v1.5

1

128

True

False

780.7795427908641

2.54019070026916289190299425735

resnet50_v1.5

1

128

True

True

782.0787833811031

1.4035397167521415787.9725710218387

resnet50_v1.5

1

256

True

False

841.0209098895596

8270.77795260718319327257479352289

resnet50_v1.5

1

256

True

True

841.6281134422322

0.6754027617726148838.8718495151428

resnet50_v1.5

2

32

False

False

597.7123015353376

2.5358048324228486606.6988150458972

resnet50_v1.5

2

32

False

True

602.1851277710804

1.4932197457537746611.4476119199953

resnet50_v1.5

2

32

True

False

1092.0792847503365

10935.3154950403268317212492608127

resnet50_v1.5

2

32

True

True

1101.2976316254073

10993.20949676736271791943633783832

resnet50_v1.5

2

64

False

False

680.4331810776715

1.1986421991311502684.8824289499631

resnet50_v1.5

2

64

False

True

680.168662439279

1.628589727697411682.3657230029216

resnet50_v1.5

2

64

True

False

1350.6915781119133

4.7719362239379591337.7101641206896

resnet50_v1.5

2

64

True

True

1337.9212057737357

13564.94065867228873318244554829635

resnet50_v1.5

2

128

False

False

722.1292515998915

1.7079082542095976727.4366020878481

resnet50_v1.5

2

128

False

True

724.5907650506392

1.3016121917621204714.9118280128031

resnet50_v1.5

2

128

True

False

1560.6063143166916

15205.3306305011697574390740438405

resnet50_v1.5

2

128

True

True

1558.8925541664316

3.76450371530692341567.667541866167

resnet50_v1.5

2

256

True

False

1681.9367020855304

2.1354907398359721663.5347160145286

resnet50_v1.5

2

256

True

True

1679.1043258972677

1.94288811597940651685.7265358502561

resnet50_v1.5

4

32

False

False

1137.8151032757787

3.60706025772351961099.8515970079466

resnet50_v1.5

4

32

False

True

1147.7653259816507

10.0861222003442561158.8182004104326

resnet50_v1.5

4

32

True

False

2090.181533002181

180627.7010872696358753203322924346

resnet50_v1.5

4

32

True

True

2118.951749390374

9.6713187755299952140.346448732058

resnet50_v1.5

4

64

False

False

1323.4940363814712

2.42886966308043471263.7396753889122

resnet50_v1.5

4

64

False

True

1336.5457321023537

2.80287566331685861323.4508475425275

resnet50_v1.5

4

64

True

False

2627.208652367316

30.5902392827195342132.559146072129

resnet50_v1.5

4

64

True

True

2605.2372571940177

16.773886993796532630.0377706917334

resnet50_v1.5

4

128

False

False

1428.9004479146274

5.198726817377691382.0710902623055

resnet50_v1.5

4

128

False

True

1431.0649693247092

2.11406961828274561414.0828353841684

resnet50_v1.5

4

128

True

False

3038.123973486709

255732.177858212514983993002834

resnet50_v1.5

4

128

True

True

3050.6240630834754

13.0879549238319443065.898543897684

resnet50_v1.5

4

256

True

False

3269.060930519893

286418.153057023527713964231626883

resnet50_v1.5

4

256

True

True

3288.0027951477073

330810.184721599039833033249948164

inception3

1

32

False

False

229.6400062758412

2311.6304999196131011150139067665

inception3

1

32

False

True

231.6282134083355435776988245647

0.7390853739290063

inception3

1

32

True

False

390.8430427702688

3971.84835745160625232497382000254

inception3

1

32

True

True

392.77650557982815

0.7988964068960409397.61870298268843

inception3

1

64

False

False

254.08237225221546

2550.797740493182867822494442939251

inception3

1

64

False

True

254.9621761236227854280397180582

0.9677860725157185

inception3

1

64

True

False

459.27229023783576

4641.35700559114740561819758930817

inception3

1

64

True

True

465.8370224010511

1.9406403927259364466.5224032998034

inception3

1

128

False

False

266.6960069340822

0.6349003776029812265.4893643297776

inception3

1

128

False

True

267.3287620209714

2650.90100725430523378865479524414

inception3

1

128

True

False

522.7059605744032

1.2095275730192268523.6175139095815

inception3

1

128

True

True

528.411119339248

5291.3233120661808886616179173554

inception3

1

256

True

False

555.9733275297333

0.793152246051491548.3831188132465

inception3

1

256

True

True

555.8605933224619

5581.61825877828520050188430980673

inception3

2

32

False

False

453.8981275778566

1.4942589199911878461.21245811536164

inception3

2

32

False

True

460.07629649023164

2.572961501771087461.7955449394074

inception3

2

32

True

False

785.3002107622603

2.808072532884698776.9118181766349

inception3

2

32

True

True

794.9652159658681

8033.1606840744608612256492187289

inception3

2

64

False

False

504.9115747109531

5050.621567645141859842426876013127

inception3

2

64

False

True

505.8260905072816

1.4738932419714106510.4880571128736

inception3

2

64

True

False

920.4145181495171

9103.38343904318581651197427361715

inception3

2

64

True

True

929.0658731171085

9415.0765499887631145909922759968

inception3

2

128

False

False

531.5809808901386

5304.073420084333152618166592716

inception3

2

128

False

True

531.3011081204222

3.821408673513115532.0966824977393

inception3

2

128

True

False

1043.3751588125951

6.3603442526469461031.042745566893

inception3

2

128

True

True

1052.1427785292055

10590.2989606232378095010705701331

inception3

2

256

True

False

1108.9748739263916

9.3866032414279911097.3179582016833

inception3

2

256

True

True

1113.16626158571

6.43197675705445351112.944630855092

inception3

4

32

False

False

872.3219766840209

8464.1137665298596856647166737354

inception3

4

32

False

True

888.3055054654682

8952.50929049903735289892419845943

inception3

4

32

True

False

1495.0846768394943

10.9658432530243281271.1921617960843

inception3

4

32

True

True

1539.5491706395067

15629.0603997601588667722873324415

inception3

4

64

False

False

992.8041857703738

2.4128076229880566967.953751912867

inception3

4

64

False

True

999.2568604412305

9942.90780671624369532997352535655

inception3

4

64

True

False

1790.7335622916307

12.5685241558580821513.182913280423

inception3

4

64

True

True

1813.338802302623

1.77105316540852441842.119923117435

inception3

4

128

False

False

1044.764217747176

9926.0283404721883219768494743513

inception3

4

128

False

True

1041.9846289297782

102716.240451564660416776883872252

inception3

4

128

True

False

2045.2687789799281

15.1122756864069461751.5253071397437

inception3

4

128

True

True

2063.858653134279

20898.50443417623219186760727177

inception3

4

256

True

False

2186.246302564507

32.6451858063778741992.6604934734398

inception3

4

256

True

True

2194.476607046693

217420.329529373497772815631509243