The NERSC_TIME is the time in seconds for the application to complete (reaches MPI_Finalize). The PARATEC LARGE input was used with 256 MPI tasks.
The table shows various runs with different settings, including some data when the node did not have the XPMEM module loaded in the kernel.

For job 29591

Memmap mmap size: 705237504 pages (2888652816384 bytes) mapped ranks 256
GRU mmap size: 705237504 pages (2888652816384 bytes) mapped ranks 256
Memmap_init complete. shm base=0x2ad2cd8e2000, sym_static=0

For job 22251

Memmap mmap size: 559104 pages (2290089984 bytes) mapped ranks 256
GRU mmap size: 559104 pages (2290089984 bytes) mapped ranks 256
Memmap_init complete. shm base=0x2b4380609000, sym_static=0

For job 21696

MPI: Module xpmem is not loaded.  MPI memory mapping features (single copy
     transfers, one-sided communication and SHMEM functions) are disabled.
     XPMEM must be enabled on your system to use these features.

Settings used in all runs

setenv MPI_VERBOSE
setenv MPI_DISPLAY_SETTINGS
setenv MPI_SHARED_VERBOSE
setenv MPI_DSM_VERBOSE
setenv MPI_MEMMAP_VERBOSE
setenv MPI_COLL_OPT_VERBOSE

setenv MPI_SHARED_NEIGHBORHOOD host

setenv MPI_BUFFER_MAX 2000
setenv MPI_BUFS_PER_PROC 256
setenv MPI_GROUP_MAX 128

MPI_DSM_DISTRIBUTE was set as default.

  • No labels