Page History

...

Code Block

title	gcc and openmpi/4.1.5+cuda

[arnoldg@dt-login04 cuda-aware-olcf]$ cat compile.sh
nvcc -c ping_pong_cuda_aware.cu 
mpicc -o ping_pong_cuda_aware ping_pong_cuda_aware.o -lcudart -lcuda

### 2 nodes , 1 core per node
[arnoldg@dt-login04 cuda-aware-olcf]$ srun ping_pong
Transfer size (B):          8, Transfer Time (s):     0.000028800, Bandwidth (GB/s):     0.000258699
Transfer size (B):         16, Transfer Time (s):     0.000024511, Bandwidth (GB/s):     0.000607946
Transfer size (B):         32, Transfer Time (s):     0.000024504, Bandwidth (GB/s):     0.001216236
Transfer size (B):         64, Transfer Time (s):     0.000024508, Bandwidth (GB/s):     0.002432084
Transfer size (B):        128, Transfer Time (s):     0.000024502, Bandwidth (GB/s):     0.004865362
Transfer size (B):        256, Transfer Time (s):     0.000024503, Bandwidth (GB/s):     0.009730004
Transfer size (B):        512, Transfer Time (s):     0.000051003, Bandwidth (GB/s):     0.009349196
Transfer size (B):       1024, Transfer Time (s):     0.000046495, Bandwidth (GB/s):     0.020511295
Transfer size (B):       2048, Transfer Time (s):     0.000074002, Bandwidth (GB/s):     0.025774236
Transfer size (B):       4096, Transfer Time (s):     0.000156007, Bandwidth (GB/s):     0.024452162
Transfer size (B):       8192, Transfer Time (s):     0.000212012, Bandwidth (GB/s):     0.035985738
Transfer size (B):      16384, Transfer Time (s):     0.000212993, Bandwidth (GB/s):     0.071639946
Transfer size (B):      32768, Transfer Time (s):     0.000407249, Bandwidth (GB/s):     0.074936002
Transfer size (B):      65536, Transfer Time (s):     0.000064525, Bandwidth (GB/s):     0.945919110
Transfer size (B):     131072, Transfer Time (s):     0.000123387, Bandwidth (GB/s):     0.989332648
Transfer size (B):     262144, Transfer Time (s):     0.000223807, Bandwidth (GB/s):     1.090851978
Transfer size (B):     524288, Transfer Time (s):     0.000369294, Bandwidth (GB/s):     1.322203569
Transfer size (B):    1048576, Transfer Time (s):     0.000613924, Bandwidth (GB/s):     1.590688320
Transfer size (B):    2097152, Transfer Time (s):     0.001097033, Bandwidth (GB/s):     1.780371056
Transfer size (B):    4194304, Transfer Time (s):     0.002173799, Bandwidth (GB/s):     1.796968938
Transfer size (B):    8388608, Transfer Time (s):     0.004155268, Bandwidth (GB/s):     1.880143607
Transfer size (B):   16777216, Transfer Time (s):     0.007972952, Bandwidth (GB/s):     1.959750802
Transfer size (B):   33554432, Transfer Time (s):     0.015780976, Bandwidth (GB/s):     1.980232452
Transfer size (B):   67108864, Transfer Time (s):     0.031320847, Bandwidth (GB/s):     1.995476069
Transfer size (B):  134217728, Transfer Time (s):     0.062092963, Bandwidth (GB/s):     2.013110578
Transfer size (B):  268435456, Transfer Time (s):     0.118608354, Bandwidth (GB/s):     2.107777326
Transfer size (B):  536870912, Transfer Time (s):     0.234608678, Bandwidth (GB/s):     2.131208467
Transfer size (B): 1073741824, Transfer Time (s):     0.443599408, Bandwidth (GB/s):     2.254286145

### 2 nodes, 1 core per node , 1 gpu per node
[arnoldg@dt-login04 cuda-aware-olcf]$ srun ping_pong_cuda_aware
Transfer size (B):          8, Transfer Time (s):     0.000048964, Bandwidth (GB/s):     0.000152165
Transfer size (B):         16, Transfer Time (s):     0.000048999, Bandwidth (GB/s):     0.000304111
Transfer size (B):         32, Transfer Time (s):     0.000048998, Bandwidth (GB/s):     0.000608230
Transfer size (B):         64, Transfer Time (s):     0.000049009, Bandwidth (GB/s):     0.001216208
Transfer size (B):        128, Transfer Time (s):     0.000048994, Bandwidth (GB/s):     0.002433118
Transfer size (B):        256, Transfer Time (s):     0.000049003, Bandwidth (GB/s):     0.004865372
Transfer size (B):        512, Transfer Time (s):     0.000050997, Bandwidth (GB/s):     0.009350298
Transfer size (B):       1024, Transfer Time (s):     0.000092998, Bandwidth (GB/s):     0.010254775
Transfer size (B):       2048, Transfer Time (s):     0.000141558, Bandwidth (GB/s):     0.013473972
Transfer size (B):       4096, Transfer Time (s):     0.000209211, Bandwidth (GB/s):     0.018233769
Transfer size (B):       8192, Transfer Time (s):     0.000210205, Bandwidth (GB/s):     0.036294980
Transfer size (B):      16384, Transfer Time (s):     0.000237235, Bandwidth (GB/s):     0.064319354
Transfer size (B):      32768, Transfer Time (s):     0.000449374, Bandwidth (GB/s):     0.067911337
Transfer size (B):      65536, Transfer Time (s):     0.000134989, Bandwidth (GB/s):     0.452148675
Transfer size (B):     131072, Transfer Time (s):     0.000175172, Bandwidth (GB/s):     0.696858934
Transfer size (B):     262144, Transfer Time (s):     0.000263563, Bandwidth (GB/s):     0.926307537
Transfer size (B):     524288, Transfer Time (s):     0.000454662, Bandwidth (GB/s):     1.073943296
Transfer size (B):    1048576, Transfer Time (s):     0.000636923, Bandwidth (GB/s):     1.533251233
Transfer size (B):    2097152, Transfer Time (s):     0.001033314, Bandwidth (GB/s):     1.890156478
Transfer size (B):    4194304, Transfer Time (s):     0.001887483, Bandwidth (GB/s):     2.069555042
Transfer size (B):    8388608, Transfer Time (s):     0.003595216, Bandwidth (GB/s):     2.173026678
Transfer size (B):   16777216, Transfer Time (s):     0.006722890, Bandwidth (GB/s):     2.324149242
Transfer size (B):   33554432, Transfer Time (s):     0.013281675, Bandwidth (GB/s):     2.352865892
Transfer size (B):   67108864, Transfer Time (s):     0.026546471, Bandwidth (GB/s):     2.354361872
Transfer size (B):  134217728, Transfer Time (s):     0.053021420, Bandwidth (GB/s):     2.357537783
Transfer size (B):  268435456, Transfer Time (s):     0.108629334, Bandwidth (GB/s):     2.301404150
Transfer size (B):  536870912, Transfer Time (s):     0.216473238, Bandwidth (GB/s):     2.309754331
Transfer size (B): 1073741824, Transfer Time (s):     0.446161794, Bandwidth (GB/s):     2.241339384
[arnoldg@dt-login04 cuda-aware-olcf]$

Using non-cuda-aware and just staging the data using traditional cuda calls, but with our best mpi ( openmpi/4.1.6 using FI_PROVIDER cxi ), performance is about 2x better:

Code Block

title	openmpi/4.1.6 with cuda staged version

Transfer size (B):          8, Transfer Time (s):     0.000013760, Bandwidth (GB/s):     0.000541470
Transfer size (B):         16, Transfer Time (s):     0.000015137, Bandwidth (GB/s):     0.000984401
Transfer size (B):         32, Transfer Time (s):     0.000014056, Bandwidth (GB/s):     0.002120306
Transfer size (B):         64, Transfer Time (s):     0.000014179, Bandwidth (GB/s):     0.004203727
Transfer size (B):        128, Transfer Time (s):     0.000014915, Bandwidth (GB/s):     0.007992320
Transfer size (B):        256, Transfer Time (s):     0.000014713, Bandwidth (GB/s):     0.016204301
Transfer size (B):        512, Transfer Time (s):     0.000015178, Bandwidth (GB/s):     0.031416626
Transfer size (B):       1024, Transfer Time (s):     0.000015057, Bandwidth (GB/s):     0.063335754
Transfer size (B):       2048, Transfer Time (s):     0.000015482, Bandwidth (GB/s):     0.123197580
Transfer size (B):       4096, Transfer Time (s):     0.000015829, Bandwidth (GB/s):     0.240994509
Transfer size (B):       8192, Transfer Time (s):     0.000019132, Bandwidth (GB/s):     0.398773506
Transfer size (B):      16384, Transfer Time (s):     0.000025477, Bandwidth (GB/s):     0.598935610
Transfer size (B):      32768, Transfer Time (s):     0.000033831, Bandwidth (GB/s):     0.902050795
Transfer size (B):      65536, Transfer Time (s):     0.000039537, Bandwidth (GB/s):     1.543744663
Transfer size (B):     131072, Transfer Time (s):     0.000058515, Bandwidth (GB/s):     2.086149580
Transfer size (B):     262144, Transfer Time (s):     0.000094330, Bandwidth (GB/s):     2.588149678
Transfer size (B):     524288, Transfer Time (s):     0.000160523, Bandwidth (GB/s):     3.041809385
Transfer size (B):    1048576, Transfer Time (s):     0.000260600, Bandwidth (GB/s):     3.747357543
Transfer size (B):    2097152, Transfer Time (s):     0.000501920, Bandwidth (GB/s):     3.891305984
Transfer size (B):    4194304, Transfer Time (s):     0.000966248, Bandwidth (GB/s):     4.042697467
Transfer size (B):    8388608, Transfer Time (s):     0.001854184, Bandwidth (GB/s):     4.213443167
Transfer size (B):   16777216, Transfer Time (s):     0.003628512, Bandwidth (GB/s):     4.306172912
Transfer size (B):   33554432, Transfer Time (s):     0.007129726, Bandwidth (GB/s):     4.383057527
Transfer size (B):   67108864, Transfer Time (s):     0.013566608, Bandwidth (GB/s):     4.606899369
Transfer size (B):  134217728, Transfer Time (s):     0.026968276, Bandwidth (GB/s):     4.635075703
Transfer size (B):  268435456, Transfer Time (s):     0.055595515, Bandwidth (GB/s):     4.496765646
Transfer size (B):  536870912, Transfer Time (s):     0.108996281, Bandwidth (GB/s):     4.587312487
Transfer size (B): 1073741824, Transfer Time (s):     0.218830109, Bandwidth (GB/s):     4.569755079

Page tree

Versions Compared

Old Version 6

New Version Current

Key