HAL system provides GPU profile functionality via NVIDIA Nsight System CLI and NVIDIA Nsight Compute CLI. Users can generate the profile result files on HAL system and then download them to their local machine to visualize.

NVIDIA Nsight Systems CLI

NVIDIA Nsight Systems is a low overhead performance analysis tool designed to provide insights developers need to optimize their software. Unbiased activity data is visualized within the tool to help users investigate bottlenecks, avoid inferring false-positives, and pursue optimizations with higher probability of performance gains. Users will be able to identify issues, such as GPU starvation, unnecessary GPU synchronization, insufficient CPU parallelizing, and even unexpectedly expensive algorithms across the CPUs and GPUs of their target platform.

nsys [global-options] profile [options] <application> [application-arguments]

NVIDIA Nsight Compute CLI

NVIDIA Nsight Compute CLI (nv-nsight-cu-cli) provides a non-interactive way to profile applications from the command line. It can print the results directly on the command line or store them in a report file. It can also be used to simply launch the target application and later attach with NVIDIA Nsight Compute or another nv-nsight-cu-cli instance.

nv-nsight-cu-cli -o profile <application> [application-arguments]

Example

We have prepared a simple VecAdd program under /opt/samples/profile/VecAdd, for NVIDIA Nsight System CLI

swrun -p gpux1
module load cuda
nsys profile /opt/samples/profile/VecAdd

and for NVIDIA Nsight Compute CLI

swrun -p gpux1 
module load cuda
module load nsight_compute/2019.5.1
nv-nsight-cu-cli -o profile /opt/samples/profile/VecAdd