Page History

<div style="background-color: yellow; border: 2px solid red; margin: 4px; padding: 2px; font-weight: bold; text-align: center;">
The Delta documentation has moved to <a href="https://docs.ncsa.illinois.edu/systems/delta/">https://docs.ncsa.illinois.edu/systems/delta/</a>. Please update any bookmarks you may have.
<br>
Click in the link above if you are not automatically redirected in 5 seconds.
</br>
</div>
<meta http-equiv="refresh" content="5; URL='https://docs.ncsa.illinois.edu/systems/delta/'" />

What is DDN IME?

Infinite Memory Engine (IME®) is a DDN solution for fast data tiering between the compute nodes and a file system in a high performance computing environment.

Conceptually the storage subsystem looks like the following

Image Added

© DDN 2020

How to use IME

The preferred way to use the Delta IME is as a read-cache for frequently read data and as a write/read cache for small file io.

It is possible to use exiting utilities and applications with files residing or created on /ime. Performance will be equal to or better than using /scratch directly for i/o to files. 

...

To get additional performance from the IME software features without changing IO routines, use the posix2ime library (LD_PRELOAD'd), to intercept standard Posix IO calls with IME API calls.  We have included a module posix2ime that does this for you (read more about posix2ime below).

...

...

You can purge the contents of files from the cache , but not the presence of the file. Please see below. 

There are some important caveats when using the /ime file system for something other than a read-cache. See section 2.2 Data Consistency Model in the developer guide document .

Users must maintain close-to-open consistency when multiple clients access the same files. This requirement guarantees that any other client will see the latest changes made by one client as soon as the client opens the file. A client must synchronize all file data and metadata changes when it closes a file and unconditionally retrieve a file’s attributes when it opens a file, ignoring any information it may have cached about the file. IME implements an enhanced close-to-open consistency model, allowing IME to be lock free.

IME commands

Please see the man page for ime-ctl or the attached developer guide document for details.

...

The posix2ime module is available and loading it will LD_PRELOAD the library for your shell or batch script and all subsequent commands.  The library is described at: DDNStorage/posix_2_ime: POSIX to IME Native API (github.com) . 

#!/bin/bash 
#SBATCH --mem=64g
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=4
#SBATCH --exclusive
#SBATCH --cpus-per-task=16
#SBATCH --partition=cpu
#SBATCH --account=bbka-delta-cpu
#SBATCH --time=00:15:00
#SBATCH --job-name=posix2ime-ior-dedicated
 

# in progress, not yet tested/complete, use 4x4 node layout for starters

BFS_DIR=/scratch/bbka/arnoldg/ime_example
IME_DIR=/ime/bbka/arnoldg/ime_example
SAMPLE_INPUT_TAR_BUNDLE=$BFS_DIR/include.tar
FILE=myinputfile
 
# Rundo themany-files followingoperations in /scratch, withoutbefore any
# of the IME ctl , just using scratch
using ime: cd $BFS_DIR
time; tar tvf $SAMPLE_INPUT_TAR_BUNDLE  # would use xvf to extract
time srun iorxvf inputbundle.tar ...

# compare to IME workflow:

# bring the scratch directory into IME
time ime-ctl --recursive --block --prestage $BFS$IME_DIR

# run the job/workflow in IME
# do serialized commands (avoiding many-files types of operations)
cd $IME_DIR
time tar tvfstat $SAMPLE_INPUT_TAR_BUNDLEFILE

# Use posix2ime for large block and/or parallel i/o 
module load posix2ime
time srun ior .../u/arnoldg/ior/src/ior -F -b64m
# turn off posix2ime
unset LD_PRELOAD  # turns off posix2ime module

# synchronize IME back theout IMEto andthe Scratch directory ( $BFS_DIR )
time ime-ctl --recursive --block --sync $IME_DIR

exit

Attachments