What is DDN IME?
How to use IME
The preferred way to use the Delta IME is as a read-cache for frequently read data and as a write/read cache for small file io.
Caution: the /scratch and /ime file systems share the same name space. The rm command will delete files on both file system.
You can purge the contents of files from the cache , but not the presence of the file. Please see below.
IME commands
Please see the man page for ime-ctl or the attached developer guide document for details.
Stage in and out single files
The ime-ctl command is used to stage and purge files from the caching /ime file system.
ime-ctl --prestage /ime/abcd/${USER}/file01
To sync the contents of a file created or changed that resides on /ime
ime-ctl --sync /ime/abcd/${USER}/file01
To purge the cached contents of a file on /ime
ime-ctl --purge /ime/abcd/${USER}/file01
Note that purging a file only clears the contents of the file from /ime. The /scratch and /ime file systems share the same name space which allows files and directories to be seen from either the caching front-end /ime or back-end /scratch.
Staging multiple files and directories
To recursively stage the contents of a directory and the files and directories below, in this case a directory called /scratch/abcd/${USER}/data_di, use the recursive
ime-ctl --prestage --recursive --block /ime/abcd/${USER}/data_dir
The --block option makes sure the stage or sync is complete before returning.
Checking file stage/cache status
To check if a file has been staged to the IME cache in /ime or has its contents synced back to the back-end file system use the ime-ctl --frag-stat command.
In this example a file that was created as /scratch/abcd/${USER}/file01 that has not been staged to /ime. The file will be visible as /ime/abcd/${USER}/file01.
Not staged to /ime: all entries are showing "0" for the Dirty, Clean and Syncing entries.
$ ime-ctl --frag-stat /ime/abcd/${USER}/file01
File: `/ime/abcd/${USER}/file01'
Number of bytes:
Dirty: 0
Clean: 0
Syncing: 0
Data on Slices:
After staging the file to /ime, the number of bytes in the "Clean" category show that the data on the cache is current.
$ ime-ctl --prestage /ime/abcd/${USER}/file01
$ ime-ctl --frag-stat /ime/abcd/${USER}/file01
File: `/ime/abcd/${USER}/file01'
Number of bytes:
Dirty: 0
Clean: 16777216
Syncing: 0
Data on Slices: 0
If the file /ime/abcd/${USER}/file01 was modified (appended, replaced, etc) one would see entries in the Dirty category:
$ ime-ctl --frag-stat /ime/abcd/${USER}/file01
File: `/ime/abcd/${USER}/file01'
Number of bytes:
Dirty: 8388608
Clean: 16777216
Syncing: 0
Data on Slices: 0
After using ime-ctl --sync to flush the changes to the back-end file system, the dirty entries will be back to 0.
$ ime-ctl --sync /ime/abcd/${USER}/file01
$ ime-ctl --frag-stat /ime/abcd/${USER}/file01
File: `/ime/abcd/${USER}/file01'
Number of bytes:
Dirty: 0
Clean: 25165824
Syncing: 0
Data on Slices: 0
IME posix2ime library
The posix2ime module is available and loading it will LD_PRELOAD the library for your shell or batch script and all subsequent commands. The library is described at: DDNStorage/posix_2_ime: POSIX to IME Native API (github.com)
# in progress, not yet tested/complete, use 4x4 node layout for starters BFS_DIR=/scratch/bbka/arnoldg/ime_example IME_DIR=/ime/bbka/arnoldg/ime_example SAMPLE_INPUT_TAR_BUNDLE=$BFS_DIR/include.tar # Run the following in scratch, without any of the IME ctl , just using scratch cd $BFS_DIR time tar tvf $SAMPLE_INPUT_TAR_BUNDLE # would use xvf to extract time srun ior ... # compare to IME workflow: # bring the scratch directory into IME time ime-ctl --recursive --prestage $BFS_DIR # run the job/workflow in IME cd $IME_DIR time tar tvf $SAMPLE_INPUT_TAR_BUNDLE time srun ior ... # synchronize the IME and Scratch directory ( $BFS_DIR ) time ime-ctl --recursive --sync $IME_DIR