|
Article ID: 704
Last updated: 14 Sep, 2026
This article provides basic hardware architecture information to help you get familiar with the NVIDIA A100 and H100 graphics proessing units (GPUs) used in the NAS GPU nodes (mil_a100 and gh200). Refer to the official guides from NVIDIA listed in the Reference section for detailed information. Note: The H100 described in this article is integrated in a Grace Hopper Superchip (GH200) which includes an ARM-based Grace CPU and a H100 GPU. Be aware that NVIDIA adds new features with each new generation of GPU and some of the descriptions here may not apply to GPUs newer than the H100. Compute HierarchyThe following describes a GPU, followed by a Streaming Multiprocessor (SM) inside a GPU, and lastly, the CUDA cores and Tensor cores inside an SM. In addition, we introduce the concepts of threads and warps needed for GPU programming. GPUsA GPU is also referred to as a coprocessor or a device. A GPU node (for example, the NAS mil_a100, gh200) may contain only 1 or multiple GPUs. A GPU:
Streaming MultiprocessorsA Streaming Multiprocessor (SM) is an individual processing unit in a GPU. The SMs are responsible for executing threads in parallel. Each SM includes:
As an example, this diagram from a NVIDIA white paper shows the configuration of an SM.
The function and capability of an SM is more in line with a single core in a multi-core CPU.
With CUDA, launching a kernel (i.e., a function to be executed in parallel on a GPU) with
results in executing the kernel once for every thread in every thread block. With many SMs per GPU, and many cores per SM, one can run thousands of threads on thousands of cores simultaneously to hide memory latency. Threads in CUDAA thread is a stream of instructions and data assigned to one single CUDA core. It is extremely lightweight with very little overhead. A thread block, also known as a cooperative thread array, is a group of threads that can cooperate with one another by synchronizing their execution and efficiently sharing data through a low latency shared memory. In CUDA, threads are organized into thread blocks. Many thread blocks are organized into a grid. Each thread block is executed by one SM and it cannot be migrated to other SMs. Prior to the introduction of thread block clusters in the H100, thread blocks were required to execute independently, and thus can execute in any order. The maximum number of threads per thread block is 1,024, as seen in output after running the nvaccelinfo command (available after loading an nvhpc module) on a GPU. This limit is imposed by the resources available in an SM to be shared by all threads in a thread block. The number of thread blocks in a grid is dictated by the size of the data to be processed. Within a CUDA kernel, a thread is identified through a combination of:
For example:
Note: The H100 introduces a new feature called thread block clusters, which extends the CUDA programming hierarchy to include four levels: threads, thread blocks, (optional) thread block clusters, and grids. The thread block clusters enable multiple thread blocks running concurrently across multiple SMs to synchronize and collaboratively fetch and exchange data by accessing other SM's shared memory. This new feature is known as 'distributed shared memory (DSMEM)' because the shared memory's virtual address space is logically distributed across all the blocks in the cluster. Warps in CUDAA warp is a group of 32 threads within a thread block that executes the same instruction simultaneously. These threads are scheduled in batches by a warp scheduler to execute on an SM. The size of a thread block is set by an application. However, it should be a multiple of 32 due to the warp scheduling. A common choice of a thread block size is 256. Note: In concept, executing a warp (32 threads) on multiple cores of an SM is similar to executing vector instructions (using xmm/ymm/zmm registers on 4/8/16 data elements in a vector) on a CPU core. CUDA CoresCUDA cores perform general-purpose computing like the CPU cores. Some of the main differences between CPU cores and CUDA cores are as follows:
Tensor CoresTensor cores are specialized GPU cores designed to accelerate matrix-multiply-and-add (MMA) operations (D = A x B + C), which are at the core of deep learning/AI computations for AI model training and inference and many HPC applications. The first generation of Tensor cores was introduced in the NVIDIA Volta architecture, followed by the second generation in Turing, third generation in Ampere, and fourth generation in Hopper. Note: Ampere and Hopper resources—but not Turing—are available at NAS. In the A100, each 3rd-generation Tensor core performs 8 x 4 x 8 (that is, 256) FP16/FP32 mixed precision FMA operations per clock. A single A100 SM, with 4 Tensor cores, can deliver 4 x 256 = 1024 FP16 FMA operations or 2048 FP16 floating point operations per clock. Besides FP16 precision input, A100 Tensor cores also add support for, among others, FP64, new BrainFloat-16 (BF16), and new TensorFloat-32 (TF32) precision input. In the H100 4th-generation Tensor cores, FP8 precision input is supported, which provides 2x the throughput of FP16. Memory HierarchySome noticeable differences between CPU memory hierarchy and GPU memory hierarchy are:
These differences speak to the fact that GPUs are designed to maximize throughput instead of minimizing latency. The high throughput is realized via a large number of registers and the use of high bandwidth memory. Instead of minimizing latency, GPUs rely on hiding latency through asychronous SIMT operations. This section describes different levels of memory in GPUs starting from the fastest to the slowest. Note: The latencies listed below for different levels of GPU memory are all approximations. They are meant to provide a rough comparison of latencies in GPUs and CPUs. RegistersA register is a small, on-chip high-speed memory location used for storing data to be immediately used by the processing functional units. A register in an SM is private to a thread. Registers read/write from/to either L1 cache or the shared memory of an SM. Each register holds a single 4-byte data. With 65,536 registers per SM, the total register size is 256 KB per SM. This size is larger than the L1/shared memory in the SM. The large number of registers, together with the large number of cores, allow many threads to run in parallel. When some threads are waiting for data from L1/L2/global memory, other threads can be executed. This effectively hides the latency. L1/Shared MemoryThe combined L1 data cache and shared memory is on-chip and is private to each SM. The size of this combined L1/shared memory varies for different GPUs. For A100 (192 KB), this size is smaller than the size of the registers. For the H100, the size is 256 KB, same as the size of the registers. The partition between L1 data cache and shared memory is set by an application. However, there is a limit on the configurable shared memory size per SM: 164 KB for the A100 and 228 KB for the H100. The latency of L1/shared memory is approximately 20 - 30 cycles as reported by independent benchmarking works listed in the Reference section. In comparison, L1 latency in CPUs is roughly five cycles. Shared MemoryMemory access latency is lower with shared memory (located inside the processor chip) than global memory (located off the processor chip). This makes shared memory ideal for storing frequently accessed data to significantly speed up data retrieval and computation. In a CUDA application, use __shared__ to declare a variable or array in shared memory for use by a thread block. All of the data in a shared memory region of a thread block can be accessed by any thread in a thread block. This allows threads in a thread block to communicate and share data with one another. On the other hand, data of a thread block in the shared memory is not visible to threads in other thread blocks. According to the Tuning Guides of various GPU architectures:
To maintain architectural compatability, static shared memory allocation is limited to 48 KB. Usage above 48 KB needs to be dynamic shared memory. All blocks on an SM can share the physical memory resources provided by the SM. L1 Data CacheL1 cache is a buffering space when reading/writing data from/to L2 cache. In an SM, the space unused as shared memory is left as L1 data cache. The size of an L1 cache line is 128 bytes, which is the size of data for a warp of 32 threads. L2 CacheL2 cache is another on-chip cache shared across all SMs. All threads in a block can access L2 cache. L2 cache keeps data read/written from/to device memory to service requests from SMs in the GPU. The latency of L2 cache in GPU (shared by SMs) is approximately 200 cycles as reported by independent benchmarking works for A100, listed in the Reference section. In comparison, latency of L3 cache in CPUs (shared by CPU cores) is approximately 50 - 70 cycles. Note: Many multi-core CPU architectures include private L1 and L2 cache for each core and shared L3 cache among the cores. Device MemoryThe device memory is off-chip and is made of different generations of High Bandwidth Memory (HBM) for different GPUs. As listed below, HBM provides much higher memory bandwidth compared to Double Data Rate 4th generation (DDR4, ~200 GB/s from 1 socket) or Low-Power DDR5X (LPDDR5X, ~500 GB/s from 1 socket) memory used in CPUs.
Latency of device memory is between about 200 to 1000 cycles, as reported by independent benchmarking work for the A100, listed in the Reference section. Assuming a CUDA core runs at 1.5 GHz, the 200 to 1000 cycles would translate to 133 to 666 nanoseconds (ns). In comparison, memory latency in CPUs varies between roughly 50 and 200 ns. The device memory is used as different forms of memory:
References
|
