Page History

<div style="background-color: yellow; border: 2px solid red; margin: 4px; padding: 2px; font-weight: bold; text-align: center;">
The HAL documentation has moved to <a href="https://docs.ncsa.illinois.edu/systems/hal/">https://docs.ncsa.illinois.edu/systems/hal/</a>. Please update any bookmarks you may have.
<br>
Click the link above if you are not automatically redirected in 7 seconds.
</br>
</div>
<meta http-equiv="refresh" content="7; URL='https://docs.ncsa.illinois.edu/systems/hal/en/latest/user-guide/running-jobs.html'" />

For complete SLURM documentation, see https://slurm.schedmd.com/. Here we only show simple examples with system-specific instructions.

HAL Slurm Wrapper Suite (Recommended)

Introduction

The HAL Slurm Wrapper Suite was designed to help users use the HAL system easily and efficiently. The current version is "swsuite-v0.4", which includes

srun (slurm command) → swrun : request resources to run interactive jobs.

sbatch (slurm command) → swbatch : request resource to submit a batch script to Slurm.

squeue (slurm command) → swqueue : check current running jobs and computational resource status.

Rule of Thumb

• Minimize the required input options.
• Consistent with the original "slurm" run-script format.
• Submits job to suitable partition based on the number of GPUs needed (number of nodes for CPU partition).

Usage

• swrun -p <partition_name> -c <cpu_per_gpu> -t <walltime> -r <reservation_name>
  • <partition_name> (required) : cpun1, cpun2, cpun4, cpun8, gpux1, gpux2, gpux3, gpux4, gpux8, gpux12, gpux16.
  • <cpu_per_gpu> (optional) : 16 cpus (default), range from 16 cpus to 40 cpus.
  • <walltime> (optional) : 4 hours (default), range from 1 hour to 24 hours in integer format.
  • <reservation_name> (optional) : reservation name granted to user.
  • example: swrun -p gpux4 -c 40 -t 24 (request a full node: 1x node, 4x gpus, 160x cpus, 24x hours)
  • Using interactive jobs to run long-running scripts is not recommended. If you are going to walk away from your computer while your script is running, consider submitting a batch job. Unattended interactive sessions can remain idle until they run out of walltime and thus block out resources from other users. We will issue warnings when we find resource-heavy idle interactive sessions and repeated offenses may result in revocation of access rights.
• swbatch <run_script>
  • <run_script> (required) : same as original slurm batch.
  • <job_name> (optional) : job name.
  • <output_file> (optional) : output file name.
  • <error_file> (optional) : error file name.
  • <partition_name> (required) : cpun1, cpun2, cpun4, cpun8, gpux1, gpux2, gpux3, gpux4, gpux8, gpux12, gpux16.
  • <cpu_per_gpu> (optional) : 16 cpus (default), range from 16 cpus to 40 cpus.
  • <walltime> (optional) : 24 hours (default), range from 1 hour to 24 hours in integer format.
  • <reservation_name> (optional) : reservation name granted to user.

  • example: swbatch demo.swb

    Code Block
    language bash title demo.swb
    #!/bin/bash #SBATCH --job-name="demo" #SBATCH --output="demo.%j.%N.out" #SBATCH --error="demo.%j.%N.err" #SBATCH --partition=gpux1 #SBATCH --time=4 srun hostname

    
    

• swqueue
  • example: 
  • Image Added

New Job Queues (SWSuite only)

HAL Wrapper Suite Example Job Scripts

New users should check the example job scripts at "/opt/samples/runscripts" and request adequate resources.

Native SLURM style

Available Queues

Submit Interactive Job with "srun"

srun --partition=debug --pty --nodes=1 \
     --ntasks-per-node=16 --cores-per-socket=4 \
     --threads-per-core=4 --sockets-per-node=1 \
     --mem-per-cpu=1200

Submit Interactive Job with "srun"

srun --partition=debug --pty --nodes=1 \
--ntasks-per-node=8 --gres=gpu:v100:1 \
-t 01:30:00 --wait=0 \
--export=ALL /bin/bash

Submit Interactive Job with "salloc"

salloc --account=[account_name] --partition=debug --nodes=1 --ntasks-per-node=8 --gres=gpu:v100:1 \
     --time=12 01:0030:00

ssh [username]@[allocated_machine_name] --wait=0 \
     --export=ALL /bin/bash

Submit Batch Job

sbatch [job_script]

Check Job Status

squeue                # check all jobs from all users 
squeue -u [username]user_name] # check all jobs belong to user_name

Cancel Running Job

scancel [job_id] # cancel job with [job_id]

PBS style

Some PBS commands are supported by SLURM.

Check Node Status

pbsnodes

Check Job Status

qstat -f [job_number]

Check Queue Status

qstat

Delete Job

qdel-u [job_id]number]

Submit Batch Job

$ cat test.pbs
#!/usr/bin/sh
#PBS -N test
#PBS -l nodes=1
#PBS -l walltime=10:00

hostname
$ qsub test.pbs
107
$ cat test.pbs.o107
hal01.hal.ncsa.illinois.edu