What is DDN IME?
How to use IME
The preferred way to use the Delta IME is as a read-cache for frequently read data and as a write/read cache for small file io.
Caution: the /scratch and /ime file systems share the same name space. The rm command will delete files on both file system.
You can purge the contents of files from the cache , but not the metadata for the file, by first ime-ctl --sync and then ime-ctl --purge
IME commands
Please see the man page for ime-ctl or the attached developer guide document for details.
Stage in and out single files
The ime-ctl command is used to stage and purge files from the caching /ime file system.
ime-ctl --prestage /ime/abcd/${USER}/file01
To sync the contents of a file created or changed that resides on /ime
ime-ctl --sync /ime/abcd/${USER}/file01
To purge the cached contents of a file on /ime
ime-ctl --purge /ime/abcd/${USER}/file01
Note that purging a file only clears the contents of the file from /ime. The /scratch and /ime file systems share the same name space which allows files and directories to be seen from either the caching front-end /ime or back-end /scratch.
Staging multiple files and directories
To recursively stage the contents of a directory and the files and directories below, in this case a directory called /scratch/abcd/${USER}/data_di, use the recursive
ime-ctl --prestage --recursive --block /ime/abcd/${USER}/data_dir
The --block option
makes sure the stage or sync is complete before returning.
Checking file stage/cache status
To check if a file has been staged to the IME cache in /ime
or has its contents synced back to the back-end file system use the ime-ctl --frag-stat
command.
In this example a file that was created as /scratch/abcd/${USER}/file01
that has not been staged to /ime. The file will be visible as /ime/abcd/${USER}/file01.
Not staged to /ime: all entries are showing "0" for the Dirty, Clean and Syncing entries.
$ ime-ctl --frag-stat /ime/abcd/${USER}/file01 File: `/ime/abcd/${USER}/file01' Number of bytes: Dirty: 0 Clean: 0 Syncing: 0 Data on Slices:
After staging the file to /ime, the number of bytes in the "Clean" category show that the data on the cache is current.
$ ime-ctl --prestage /ime/abcd/${USER}/file01 $ ime-ctl --frag-stat /ime/abcd/${USER}/file01 File: `/ime/abcd/${USER}/file01' Number of bytes: Dirty: 0 Clean: 16777216 Syncing: 0 Data on Slices: 0
If the file /ime/abcd/${USER}/file01
was modified (appended, replaced, etc) one would see entries in the Dirty category:
$ ime-ctl --frag-stat /ime/abcd/${USER}/file01 File: `/ime/abcd/${USER}/file01' Number of bytes: Dirty: 8388608 Clean: 16777216 Syncing: 0 Data on Slices: 0
After using ime-ctl --sync
to flush the changes to the back-end file system, the dirty entries will be back to 0.
$ ime-ctl --sync /ime/abcd/${USER}/file01 $ ime-ctl --frag-stat /ime/abcd/${USER}/file01 File: `/ime/abcd/${USER}/file01' Number of bytes: Dirty: 0 Clean: 25165824 Syncing: 0 Data on Slices: 0
IME posix2ime library
The posix2ime module is available and loading it will LD_PRELOAD the library for your shell or batch script and all subsequent commands. The library is described at: DDNStorage/posix_2_ime: POSIX to IME Native API (github.com)
# in progress, not yet tested/complete, use 4x4 node layout for starters BFS_DIR=/scratch/bbka/arnoldg/ime_example IME_DIR=/ime/bbka/arnoldg/ime_example SAMPLE_INPUT_TAR_BUNDLE=$BFS_DIR/include.tar # Run the following in scratch, without any of the IME ctl , just using scratch cd $BFS_DIR time tar tvf $SAMPLE_INPUT_TAR_BUNDLE # would use xvf to extract time srun ior ... # compare to IME workflow: # bring the scratch directory into IME time ime-ctl --recursive --prestage $BFS_DIR # run the job/workflow in IME cd $IME_DIR time tar tvf $SAMPLE_INPUT_TAR_BUNDLE time srun ior ... # synchronize the IME and Scratch directory ( $BFS_DIR ) time ime-ctl --recursive --sync $IME_DIR