The NAS Cabeus cluster currently includes NVIDIA GPUs with the Ampere (mil_a100) and Hopper (gh200) microarchitectures. This article provides some information on the compute capability of the various microarchitectures and how to compile applications written in C/C++ with CUDA extensions for a target microarchitecture.

For applications written in ISO C++, ISO Fortran, OpenACC or CUDA Fortran, read the various sections in Programming for NVIDIA GPUs to learn how to compile for a target microarchitecture.

Compute Capabilities

The high-level features of each architecture can be found using the command nvaccelinfo included in a NVIDIA HPC Software Development Kit (SDK) installed as a module such as nvhpc/23.11. For example, to find information about the A100 GPUs in a NAS mil_a100 node, perform the following:

Cabeus_Front_End% qsub -I -q gpu_devel -lselect=1:ngpus=1:model=mil_a100
PBS cb12n05> module use -a /nasa/modulefiles/testing
PBS cb12n05> module load nvhpc/23.11   
PBS cb12n05> nvaccelinfo
CUDA Driver Version:           12030
Device Number:                 0
Device Name:                   NVIDIA A100-SXM4-80GB
Device Revision Number:        8.0
Global Memory Size:            84986691584
Number of Multiprocessors:     108
Concurrent Copy and Execution: Yes
Total Constant Memory:         65536
Total Shared Memory per Block: 49152
Registers per Block:           65536
Warp Size:                     32
Maximum Threads per Block:     1024
Maximum Block Dimensions:      1024, 1024, 64
Maximum Grid Dimensions:       2147483647 x 65535 x 65535
....
Max Threads Per SMP:           2048
Async Engines:                 5
Unified Addressing:            Yes
Managed Memory:                Yes
Concurrent Managed Memory:     Yes
Preemption Supported:          Yes
Cooperative Launch:            Yes
Default Target:                cc80

Lower level details of each microarchitecture are encapsulated in its compute capability. The compute capability of a GPU is represented as X.Y, where X is a generation number and Y is a version in that generation. For example, the compute capability of a A100 and H100 is 7.0, 8.0, and 9.0, respectively. This is reflected in the last line of the nvaccelinfo output, as shown in the sample output for an A100 GPU above, cc80.

Details of each compute capability are shown in two tables: (1) Feature Support per Compute Capability and (2) the Technical Specifications per Compute Capability.

A compute feature is introduced with a compute architecture with the intention that the feature will be available on all subsequent architectures. For example, the features "Tensor Cores" and "Mixed Precision Warp-Matrix Functions" were introduced starting with compute capability 7.x. The feature "Hardware-accelerated memcpy_async" was introduced in 8.x , and "Distributed Shared Memory" in 9.0.

An application must be compiled with the proper compute capability setting(s) in order to run on a target GPU architecture.

Compilation with NVCC

NVIDIA CUDA Compiler Driver (NVCC) is used for compiling CUDA C/C++ applications which usually use .cu suffix in their filenames. NVCC is available after loading an nvhpc modulefile. For example,

PBS cb12n05> module load nvhpc/23.11
PBS cb12n05> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Sep__8_19:17:24_PDT_2023
Cuda compilation tools, release 12.3, V12.3.52
Build cuda_12.3.r12.3/compiler.33281558_0

NVCC Compilation Model

NVCC follows a split compilation model to handle the host code and the device code separately. An approximate flow is as follows:

• Preprocesses and separates the device code from the host code
• Parses the device code and the host code separately
• Compiles the device code using proprietary NVIDIA compilers and assembler
• Compiles the host code using a C/C++ host compiler (for example, gcc, g++ under /usr/bin)
• Embeds the compiled device code as fatbinary images in the host object file
• Links specific CUDA runtime libraries

The actual compilation trajectory is quite elaborate. Refer to The CUDA Compilation Trajectory for more information. You can also get some ideas by adding the nvcc flags ‑‑keep or ‑keep to keep all intermediate files that are generated during internal compilation steps and/or ‑‑verbose or ‑v to list the compilation commands generated by nvcc.

Note: NVCC provides the flexibility of choosing a different host compiler through the flag ‑‑compiler‑bindir or ‑‑ccbin followed by the path and name of the compiler, for example, ‑‑ccbin /nasa/intel/..../bin/intel64/icc. If it produces an error stating the compiler version is not supported (such as gcc > version 12) and you want to proceed anyway, you can use the flag ‑‑allow‑unsupported‑compiler.

compute_XY, sm_XY and Portability Across Different Compute Capabilities

A complexity in compiling the device code is described in GPU Compilation. In short, NVCC is based on the LLVM open source compiler infrastructure and it compiles the device code in two stages:

1. Converts the device code (.cu) to intermediate PTX instructions (with suffix .ptx).

   Note: PTX stands for Parallel Thread Execution. A .ptx code, in ASCII format, is similar to an assembly code.

   This stage is controlled by the nvcc compiler flag ‑‑gpu‑architecture <arch> (with a short-hand notation of ‑arch) where <arch> is a virtual GPU architecture which is specified with features supported by a GPU (as seen in the Feature Support per Compute Capability Table) but not its target-specific instruction sets.

   Some of the virtual GPU architectures are: compute_50, compute_60, compute_70, compute_80, compute_90. Run the command nvcc ‑‑list‑gpu‑arch to see a complete list supported by the nvcc compiler.

2. Optimizes and translates the .ptx file to generate a CUDA binary (.cubin) containing target-specific instructions for a real GPU architecture.

   This stage is dictated by the nvcc compiler flag ‑‑gpu‑code <code> (with a short-hand notation of ‑code) where <code> specifies the name of the real GPU architecture of a NVIDIA GPU to assemble and optimize the PTX for.

   The real GPU architectures are represented as sm_XY such as sm_50, .., sm_90, in CUDA. Run the command nvcc ‑‑list‑gpu‑code to see a complete list supported by the nvcc compiler.

   This second stage can be performed during compile time or Just-in-Time (JIT) during run time.

   • Compile Time

     To generate a target-specific (for example, a GPU with compute capability 8.0) .cubin during compile time, use these flags:

     nvcc x.cu --gpu-architecture=compute_80 --gpu-code=sm_80

     or

     nvcc x.cu -arch=compute_80 -code=sm_80

     or

     nvcc x.cu --gencode arch=compute_80,code=sm_80

     Note: In this example, the intermediate .ptx will be discarded and the resulting executable containing a .cubin image only targets a GPU with compute capability 8.0. It will not run on GPUs with compute capability 7.0 or 9.0.

   • JIT Compilation During Run Time

     To delay the generation of a target-specific .cubin, the intermediate .ptx file will be included in the executable. During runtime, the current architecture is detected and the compiler builds the appropriate .cubin JIT (thus causing a start-up delay) for execution. This is allowed as long as the detected architecture has a compute capability identical to or newer than the specified virtual GPU architecture. Thus, this is a mechanism to maintain the portability of an executable for newer generations of GPUs. Using a virtual GPU architecture with a compute capability of 8.0 as an example, this is done using the command:

     nvcc x.cu --gpu-architecture=compute_80 --gpu-code=compute_80

     or

     nvcc x.cu -arch=compute_80 -code=compute_80

     or

     nvcc x.cu --gencode arch=compute_80,code=compute_80

     Note: In this example, the resulting executable will compile JIT and run successfully for GPUs with compute capability 8.0 and 9.0, but not 7.0.

     TIP: To alleviate the start-up penalty of running on a GPU with the same compute capability as the specified virtual architecture while maintaining the portability to a newer GPU compute capability, one can include both sm_xy and compute_xy for the ‑‑gpu‑code flag. For example:

     nvcc x.cu ‑‑gpu‑architecture=compute_80 ‑‑gpu‑code=sm_80,compute_80

     Note: No space should be used between sm_80 and ,compute_80.

     Caveat: An executable built under x86_64 is not compatible with the aarch64 (LLVM-based ARM) operating system. Therefore, you can not JIT build/run an executable on an aarch64 system using an existing executable from x86_64.

Default

When the flags ‑‑gpu‑architecture and ‑‑gpu‑code are not specified as in:

nvcc x.cu

it defaults to

nvcc x.cu --gpu-architecture=compute_52 --gpu-code=sm_52,compute_52

where the compute capability 5.2 is for the NVIDIA Maxwell GPUs.

Exception for --gpu-architecture Flag

Normally, the option to this flag must be a virtual GPU architecture compute_XY. In the event that no ‑‑gpu‑code is used, then one can provide a single real GPU architecture sm_XY to the ‑‑gpu‑architecture flag. For example,

nvcc x.cu --gpu-architecture=sm_80

is equivalent to

nvcc x.cu --gpu-architecture=compute_80 --gpu_code=sm_80,compute_80

Fatbinaries

Another way to increase portability across GPUs with different compute capabilities is to create a fatbinary which includes multiple .cubin images and a .ptx image. For example,

nvcc x.cu --gpu-architecture=compute_70 --gpu-code=sm_70 \
          --gpu-architecture=compute_80 --gpu-code=sm_80 \
          --gpu-architecture=compute_80 --gpu-code=compute_80

will allow the CUDA driver to select the most appropriate .cubin according to the current native architecture (a V100 or a A100). If no appropriate .cubin is present for the current architecture (e.g., a H100 with compute capability 9.0), the .ptx will be loaded and be compiled JIT (for running on the H100).

% module load nvhpc/23.11
% cuobjdump your_executable

References

• NVIDIA CUDA Compiler Driver NVCC
• CUDA C++ Programming Guide
• CUDA C++ Best Practice Guide