Performance Analysis of Your GPU CUDA Kernels with Nsight Compute CLI

Article ID: 706
Last updated: 14 Sep, 2026

Nsight Compute is a tool that collects metrics via hardware counters and software instrumentations for deep-dive profiling and guided performance analysis of the CUDA kernels. GPU architectures supported by Nsight Compute started with the NVIDIA Volta processors.

To understand the profile results from Nsight Compute, you need some knowledge of the GPU hardware.

TIP: Before using Nsight Compute, consider using Nsight Systems to analyze the overall performance on both the CPU and GPU of your application.

Nsight Compute has a command-line interface (CLI), called ncu, for profiling in non-interactive mode; and a graphical user interface (GUI), called ncu-ui, for profiling in interactive mode. This article only provides basic information on using ncu. For detailed information about ncu and ncu-ui, refer to NVIDIA documentation listed in the References section.

Depending on the nature of the application, some CUDA kernels may be launched multiple times during a run (for example, a kernel called within a loop). For each kernel launch, the name of the kernel function (e.g., vectorAdd_A in the example below) and the progress of data collection is shown in the standard output (STDOUT). To collect all requested profile information for the many metrics that are included for profiling, you might need to replay the kernels multiple times. When the collection is completed, it will also show the number of replay passes of that kernel. For example:

==PROF== Profiling "vectorAdd_A" - 0: 0%....50%....100% - 46 passes
WARNING: Collecting performance data using ncu can incur significant runtime overhead on the application, as discussed here.

Nsight Compute is included in the CUDA Toolkit, which in turn is included in the NVIDIA HPC software development kit (SDK) distributions (for example, NVHPC). The latest available version of ncu is recommended, as it likely provides more features for profiling.

Command-Line Options

Some of the ncu options to consider when profiling are listed below. For all options and more details, read output from the ncu -h command after loading a nvhpc module.

  • -o [--export] report_name

    Saves profile results in a binary report file called report_name.ncu-rep. For example, if you use -o my_ncu_output, the profile results are saved in the my_ncu_output.ncu-rep file. After profiling, the binary report file can be imported for analysis via ncu or ncu-ui. Without this option, a profile result file is created temporarily with some results written in text format to STDOUT. The profile result file is then removed.

  • -k [--kernel_name] kernel_name

    Chooses a specific kernel, with exact match to kernel_name, to profile. For example, -k vectorAdd_A, which is an exact match to the kernel listed after ==PROF== shown above.

  • -c [--launch-count] a_number

    Limits the number of collected profile results to a_number. For example, -c 2.

  • -s [--launch-skip] a_number

    Skips a given number of kernel launches before data collection starts. The default is 0.

  • --set set_identifier

    Chooses among the pre-defined profiling sets: basic, detailed, full, nvlink, pmsampling, roofline. The default is basic, which is less comprehensive and incurs less profiling overhead in runtime and memory usage.

    The basic set includes four sections: LaunchStats, Occupancy, SpeedOfLight, WorkloadDistribution. The detailed set and the full set include the same four sections as in the basic set, plus additional sections. Run the ncu --list-sets command to see what sections are included in each pre-defined set.

    Note: The WorkloadDistribution section is not available in nvhpc/24.3 and earlier.

  • --section section_identifier

    Chooses a specific section to profile. For example, --section ComputeWorkloadAnalysis.

    Run the ncu --list-sections command to see the list of available sections. Each section includes a group of metrics to be profiled, as pre-defined in Nsight_Compute_installation_location/sections/section-identifier.section.

  • --metrics arg

    Specify all metrics to be profiled, separated by comma. For example, --metrics sm__cycles_active.avg,sm__cycles_elapsed.sum.

    Run the ncu --list-metrics and ncu --query-metrics commands on a GPU node where you intend to run your application to see the available metrics.

Profile Your Application

The examples shown in this article use the Cabeus front ends (CFEs) for the GPUs with x86_64 based CPUs (mil_a100).

For GPUs with aarch64-based CPUs, such as gh200, please be aware of the following:

  • You should use one of the following front ends: cghfe01 or cghfe02.
  • The module path and available nvhpc modulefiles may be different.

In order to avoid version incompatibility, before profiling, it is recommended to build your application using the compilers, math, and CUDA libraries included in the same nvhpc version you plan to use ncu for profiling.

Within a PBS job, run the module use and module avail commands as shown below to find all available nvhpc versions, then load the latest version. The following example assumes the latest version is 24.7.

CFE% qsub -I -q gpu_devel -lselect=1:ngpus=4:model=mil_a100
a_GPU_node> source /usr/local/lib/global.profile   #if using bash
a_GPU_node> module avail nvhpc
a_GPU_node> module load nvhpc/24.7    # with OpenMPI version 4
or
a_GPU_node> module load nvhpc/24.7_openmpi3  # with OpenMPI version 3

For Single-Process Applications

Without any additional options, ncu launches and instruments your application, for example, a.out, and collects the basic set of profile results of all CUDA kernels in the application launched during execution.

ncu -o report_name [other ncu options] a.out [a.out options]

An output file report_name.ncu-rep containing the profile results is saved.

For Single-Node Multi-Process Applications

The following example will save profile results from all ranks into a single report file, report_name.ncu_rep.

ncu --target-processes all -o report_name [other ncu options] \
mpirun [mpirun options] a.out [a.out options]

Note: The mpirun command line above is too long to be formatted as one line, so it is broken with a backslash (\).

For Multi-Node Multi-Process Applications

The following will save profile results with one file per rank, e.g., report_name_0.ncu_rep, report_name_1.ncu_rep, and so on.

For bash,
mpirun [mpirun options] ncu -o report_name_%q{OMPI_COMM_WORLD_RANK} \
[other ncu options] a.out [a.out options]

For csh,
mpirun [mpirun options] ncu -o report_name_%q\{OMPI_COMM_WORLD_RANK\} \ 
[other ncu options] a.out [a.out options]

Note: The mpirun command lines above are too long to be formatted as one line, so they are broken with a backslash (\).

Profiling Just One Rank

NVIDIA suggests using the following script, wrap.sh, for profiling just one rank (e.g., rank 0):

(For bash):

#!/bin/bash
if [[ $OMPI_COMM_WORLD_RANK == 0 ]]; then
   ncu -o report-name_${OMPI_COMM_WORLD_RANK}  --target-processes all "$@"
else
   "$@"
fi

(For csh):

#!/bin/csh
if ( $OMPI_COMM_WORLD_RANK == 0 ) then
ncu -o report-name_${OMPI_COMM_WORLD_RANK} --target-processes all $*
else
$*
endif

To profile and then run the script, do:

mpirun [mpi options] ./wrap.sh a.out [a.out options]

View Profile Report

This step can be performed on either a GPU compute node or a front-end node where Nsight Compute is available.

To get a quick text summary from the binary report, do:

ncu --import report_name.ncu-rep
or
ncu -i report_name.ncu-rep

If the profiling was done with the basic set, this will generate four sections, including GPU Speed Of Light Throughput, Launch Statistics, Occupancy, and GPU and Memory Workload Distribution, for each profiled kernel. The output may also provide additional explanation on the profile results and optimization advice based on rules created by NVIDIA engineers.

You may also add the --print-details all option to show more details.

ncu --print-details all --import report-name.ncu-rep

Use of Nsight Compute ncu-ui may provide more comprehensive analysis, including the source line information if the application was compiled with -gpu=lineinfo for nvfortran, nvc, and nvc++ and --generate-line-info or -lineinfo for nvcc. However, the front-ends cfe[01,02] and cghfe[01,02] do not yet have the required Qt software installed to enable the use of ncu-ui. If you have Nsight Compute and Qt installed on your local workstation, you may copy your report-name.ncu-rep there for analysis.

References

Contact Us

General User Assistance

Security

  • Report security issues 24x7x365
  • Toll-free: 1-877-NASA-SEC (1-877-627-2732)
  • E-mail: soc@nasa.gov

User Documentation

High-End Computing Capability (HECC) Portfolio Office

NASA High-End Computing Program

Tell Us About It

Our goal is furnish all the information you need to efficiently and effectively use the HECC resources needed for your NASA computational projects.

We welcome your input on features and topics that you would like to see included on this website.

Please send us email with your wish list and other feedback.