CUDA stands for Compute Unified Device Architecture. It is a parallel computing platform and programming model created by NVIDIA in 2006 for accelerating computing on General-Purpose Graphics Processing Units (GPGPUs).

This article provides a high-level view of two of the components in CUDA—the CUDA Toolkit and the CUDA driver—and the compatability between them.

CUDA Toolkit

Everything needed to develop, optimize and deploy applications written in CUDA is included in the CUDA Toolkit. It is released multiple times per year by NVIDIA to provide new features, performance improvement, and critical bug fixes. The Toolkit includes:

A C/C++ Compiler

The NVIDIA CUDA compiler (nvcc) is for compiling applications written in standard C/C++ with CUDA extensions. The CUDA applications, which typically use the suffix .cu, are turned into machine codes by the compiler for running on the GPUs.

Notes:

• You can find many CUDA samples in github.
• The NVIDIA HPC compilers, nvc, nvc++ and nvfortran (rebranded from the PGI compilers) are not part of the CUDA Toolkit. However, both the HPC compilers and nvcc are included in the NVIDIA HPC Software Development Kit (SDK).

GPU-accelerated CUDA Libraries

Besides explicitly incorporating CUDA extensions to the standard C/C++ languages, an application may also utilize some of these pre-built GPU-accelerated Math Libraries included in the CUDA Toolkit:

• cuBLAS: basic linear algebra
• cuFFT: fast fourier transform
• cuRAND: random number generation
• cuSOLVER: dense and sparse direct solver
• cuSPARSE: BLAS for sparse matrix
• cuTENSOR: tensor linear algebra
• CUDA math library: standard mathematical function library

These libraries provide the highest level of abstraction in GPU programming.

The NVIDIA HPC compilers (nvc, nvc++, and nvfortran) provide a flag -cudalib[=cublas|cufft|..] for linking to these CUDA libraries. Check out examples available under the 4_CUDA_Libraries section of the CUDA samples or the directory /nasa/arm/software/nvhpc/24.11/Linux_aarch64/24.11/examples/CUDA-Libraries on a NAS GH200 front end system chghfe[01,02].

Besides the Math Libraries, the CUDA Toolkit also includes:

• Thrust, the C++ Parallel Algorithms Library: Allows C++ developers to accelerate operations such as sort, scan, transform, and reduction with just a few lines of code. Thrust, a header-only library, is available under, for example, /nasa/arm/software/nvhpc/24.11/Linux_aarch64/24.11/cuda/12.6/targets/sbsa-linux/include/thrust.
• Communication Library: Consists of MPI (OpenMPI and HPCX-enhanced OpenMPI), NVIDIA Collective Communication Library (NCCL), and NVSHMEM. They are available under, for example, /nasa/arm/software/nvhpc/24.11/Linux_aarch64/24.11/comm_libs.

NVIDIA also provides GPU-accelerated Data Processing Libraries, Deep Learning Core Libraries, and more. However, they are not included in the CUDA Toolkit. Read NVIDIA CUDA-X Libraries for more information.

A CUDA Runtime Library

The CUDA runtime (libcudart.so, cuda_runtime.h, cuda_runtime_api.h) is one of the software layers that can be implicitly or explicitly used in GPU programming. It sits below the user application and GPU-accelerated libraries and above the CUDA driver.

The CUDA runtime API provides developers with a high-level C++ interface for simplified device management, error handling, memory management, stream management, and more. For example, the functions cudaMalloc, cudaMemset, cudaMemcpy, cudaFree are included in the CUDA runtime memory management.

Debugging and Optimization Tools

• CUDA-GDB: A command-line tool for unified CPU and CUDA debugging.
• NVIDIA Nsight tools (A100 and newer GPUs)
  • Nsight Systems (nsys, nsys-ui): a high-level application-wide profiler for GPU and CPU sampling and tracing
  • Nsight Compute (ncu, ncu-ui): an interactive low-level kernel profiler for digging deeper into the CUDA kernels

CUDA Toolkit on NAS Systems

For systems running with the Linux operating system, the CUDA Toolkit is available for either the x86_64 architecture (for example, the NAS mil_a100 nodes) or the ARM architecture (Apple-developed arm64 and LLVM-developed aarch64, such as the NAS gh200).

The CUDA Toolkit is included in the NVIDIA HPC SDK. At NAS, multiple versions of the HPC SDK are available as modules (for example, nvhpc/24.11). In an HPC SDK release, there may be one or more CUDA Toolkit versions included. For example, on the GH200 front-end or compute nodes, the available CUDA Toolkit versions and default are:

Notes:

• The CUDA Toolkit follows an .X.Y.Z version scheme where .X stands for major version, .Y stands for minor version, and .Z stands for release/patch version. Different major versions are not binary compatible. Same major version but different minor versions are binary compatible but may not be source compatible (i.e., you may have to modify your source code to comply with changes in the APIs introduced in a different monor version).
• .Z is not shown in the CUDA Toolkit versions in the above table.

CUDA Driver

Complementary to the CUDA Toolkit whose components are mainly used for building applications, the CUDA driver components are mainly used for running applications.

In order to run a CUDA application, the system should have a CUDA enabled GPU and a NVIDIA driver that is compatible with the CUDA Toolkit that was used to build the application itself.

There are two components in the CUDA driver:

• User-mode driver (libcuda.so, cuda.h): The CUDA driver API provides a low-level interface for applications to target NVIDIA hardware.
• Kernel-mode driver (nvidia.ko): It provides low-level access to the NVIDIA GPU hardware.

CUDA Driver on NAS Systems

Unlike the CUDA Toolkit, which can have multiple versions installed, only one version of the CUDA driver can be installed on a GPU node by a system administrator. You can find the kernel module version installed using either the nvidia-smi or the modinfo command. For example:

a_GPU_node% nvidia-smi | grep -i version
| NVIDIA-SMI 610.57.04        KMD Version: 610.57.04     CUDA UMD Version: 13.3  

a_GPU_node% modinfo /usr/lib/modules/`uname -r`/extra/nvidia/nvidia.ko | grep ^version
version:        610.57.04

Note: No CUDA driver is installed on the CPU-only nodes such as CFE and pbspl4.

CUDA Compatibility

Running a CUDA application requires the system with at least one CUDA-capable GPU and a driver that is compatible with the CUDA Toolkit.

• Each CUDA release has a Toolkit version and a corresponding driver version. For example, Toolkit 12.3 update 1 corresponds to driver 545.23.08, while Toolkit 13.3 update 1 corresponds to driver 610.43.02. See Table 3 in CUDA Toolkit and Corresponding Driver Versions in NVIDIA CUDA Toolkit Release Notes.
• The CUDA driver is backward compatible: an application compiled with an older CUDA Toolkit version can run with a newer driver release.
• Applications built with a new CUDA Toolkit require a minimum older CUDA driver release.
  • Application built with CUDA Toolkit 11.x require CUDA driver >= 450.80.2
  • Application built with CUDA Toolkit 12.x require CUDA driver >= 525.60.13
  • Application built with CUDA Toolkit 13.x require CUDA driver >= 580.65.06

References

• What is CUDA?
• CUDA Toolkit Release Notes
• CUDA Runtime API
• CUDA Driver API
• CUDA Compatibility