...
| Warning | ||
|---|---|---|
| ||
IME performance for directory/metadata operations is slower than /scratch (it is not the place to extract or copy millions of files). Do those operations (rsync, tar, etc) in /scratch. |
To get additional benefit performance from the IME software features without changing IO routines, use the posix2ime library by ld preload(LD_PRELOAD'd), to intercept standard Posix IO calls with IME API calls. We have included a module "posix2ime" that that does this for you (read more about posix2ime below).
...
The posix2ime module is available and loading it will LD_PRELOAD the library for your shell or batch script and all subsequent commands. The library is described at: DDNStorage/posix_2_ime: POSIX to IME Native API (github.com) .
| Info | ||
|---|---|---|
| ||
| At this time, use of the posix2ime library requires dedicated ( #SBATCH --exclusive ) nodes for your job script or srun command. |
| Code Block | ||||
|---|---|---|---|---|
| ||||
#!/bin/bash #SBATCH --mem=64g #SBATCH --nodes=4 #SBATCH --ntasks-per-node=4 #SBATCH --exclusive #SBATCH --cpus-per-task=16 #SBATCH --partition=cpu #SBATCH --account=bbka-delta-cpu #SBATCH --time=00:15:00 #SBATCH --job-name=posix2ime-ior-dedicated BFS_DIR=/scratch/bbka/arnoldg/ime_example IME_DIR=/ime/bbka/arnoldg/ime_example SAMPLE_INPUT_FILE=myinputfile # do many-files operations in /scratch before # using ime: cd $BFS_DIR; tar xvf inputbundle.tar ... # bring the scratch directory into IME ime-ctl --recursive --block --prestage $IME_DIR # run the job/workflow in IME # do serialized commands (avoiding many-files types of operations) cd $IME_DIR stat $SAMPLE_INPUT_FILE # Use posix2ime for large block and/or parallel i/o module load posix2ime time srun /u/arnoldg/ior/src/ior -F -b64m # turn off posix2ime unset LD_PRELOAD # turns off posix2ime module # synchronize IME back out to the Scratch directory ( $BFS_DIR ) ime-ctl --recursive --block --sync $IME_DIR exit |
...