Host name: hal.ncsa.illinois.edu

Hardware

• 16 IBM AC922 nodes
  • IBM 8335-GTG AC922 server
    • 2x 20-core IBM POWER9 CPU @ 2.00GHz
    • 256 GB DDR4
  • 4x NVIDIA V100 GPUs
    • 5120 cores
    • 16 GB HBM 2
  • 2-Port EDR 100 GB IB ConnectX-5 Adapter

• RHEL 7.6
• CUDA 10.1.105
  • cuDNN 7.5.0
  • NCCL 2.4.2
• IBM XLC 16.1.1
• IBM XLFORTRAN 16.1.1
• Advance toolchain for Linux on Power 12.0
• PowerAI 1.6.0

• to connect to the cluster, run "ssh username@hal.ncsa.illinois.edu"
• to submit sbatch job, run "sbatch run_script.sb"
  #### run_scripts.sb
  #!/bin/bash
  #SBATCH --job-name="hostname"
  #SBATCH --output="hostname.%j.%N.out"
  #SBATCH --error="hostname.%j.%N.err"
  #SBATCH --partition=debug
  #SBATCH --nodes=4
  #SBATCH --ntasks-per-node=1
  #SBATCH --export=ALL
  #SBATCH -t 00:10:00
  srun /bin/hostname
  ####
• to submit interactive job, run
  "srun --partition=debug --pty --nodes=1 --ntasks-per-node=32 \
  --gres=gpu:v100:1 -t 01:30:00 --wait=0 --export=ALL /bin/bash"
• to load IBM powerai module, run "module load ibm/powerai"
• to check loaded module, run "module list"
• to check available modules, run "module avail"