|
Article ID: 706
Last updated: 14 Sep, 2026
Nsight Compute is a tool that collects metrics via hardware counters and software instrumentations for deep-dive profiling and guided performance analysis of the CUDA kernels. GPU architectures supported by Nsight Compute started with the NVIDIA Volta processors. To understand the profile results from Nsight Compute, you need some knowledge of the GPU hardware. TIP: Before using Nsight Compute, consider using Nsight Systems to analyze the overall performance on both the CPU and GPU of your application.
Nsight Compute has a command-line interface (CLI), called ncu, for profiling in non-interactive mode; and a graphical user interface (GUI), called ncu-ui, for profiling in interactive mode. This article only provides basic information on using ncu. For detailed information about ncu and ncu-ui, refer to NVIDIA documentation listed in the References section. Depending on the nature of the application, some CUDA kernels may be launched multiple times during a run (for example, a kernel called within a loop). For each kernel launch, the name of the kernel function (e.g., vectorAdd_A in the example below) and the progress of data collection is shown in the standard output (STDOUT). To collect all requested profile information for the many metrics that are included for profiling, you might need to replay the kernels multiple times. When the collection is completed, it will also show the number of replay passes of that kernel. For example:
WARNING: Collecting performance data using ncu can incur significant runtime overhead on the application, as discussed here.
Nsight Compute is included in the CUDA Toolkit, which in turn is included in the NVIDIA HPC software development kit (SDK) distributions (for example, NVHPC). The latest available version of ncu is recommended, as it likely provides more features for profiling. Command-Line OptionsSome of the ncu options to consider when profiling are listed below. For all options and more details, read output from the ncu -h command after loading a nvhpc module.
Profile Your ApplicationThe examples shown in this article use the Cabeus front ends (CFEs) for the GPUs with x86_64 based CPUs (mil_a100). For GPUs with aarch64-based CPUs, such as gh200, please be aware of the following:
In order to avoid version incompatibility, before profiling, it is recommended to build your application using the compilers, math, and CUDA libraries included in the same nvhpc version you plan to use ncu for profiling. Within a PBS job, run the module use and module avail commands as shown below to find all available nvhpc versions, then load the latest version. The following example assumes the latest version is 24.7.
For Single-Process ApplicationsWithout any additional options, ncu launches and instruments your application, for example, a.out, and collects the basic set of profile results of all CUDA kernels in the application launched during execution.
An output file report_name.ncu-rep containing the profile results is saved. For Single-Node Multi-Process ApplicationsThe following example will save profile results from all ranks into a single report file, report_name.ncu_rep.
Note: The mpirun command line above is too long to be formatted as one line, so it is broken with a backslash (\). For Multi-Node Multi-Process ApplicationsThe following will save profile results with one file per rank, e.g., report_name_0.ncu_rep, report_name_1.ncu_rep, and so on.
Note: The mpirun command lines above are too long to be formatted as one line, so they are broken with a backslash (\). Profiling Just One RankNVIDIA suggests using the following script, wrap.sh, for profiling just one rank (e.g., rank 0): (For bash):
(For csh):
To profile and then run the script, do:
View Profile ReportThis step can be performed on either a GPU compute node or a front-end node where Nsight Compute is available. To get a quick text summary from the binary report, do:
If the profiling was done with the basic set, this will generate four sections, including GPU Speed Of Light Throughput, Launch Statistics, Occupancy, and GPU and Memory Workload Distribution, for each profiled kernel. The output may also provide additional explanation on the profile results and optimization advice based on rules created by NVIDIA engineers. You may also add the --print-details all option to show more details.
Use of Nsight Compute ncu-ui may provide more comprehensive analysis, including the source line information if the application was compiled with -gpu=lineinfo for nvfortran, nvc, and nvc++ and --generate-line-info or -lineinfo for nvcc. However, the front-ends cfe[01,02] and cghfe[01,02] do not yet have the required Qt software installed to enable the use of ncu-ui. If you have Nsight Compute and Qt installed on your local workstation, you may copy your report-name.ncu-rep there for analysis. References
|