Basics on NVIDIA GPU Hardware Architecture

Article ID: 704
Last updated: 14 Sep, 2026

This article provides basic hardware architecture information to help you get familiar with the NVIDIA A100 and H100 graphics proessing units (GPUs) used in the NAS GPU nodes (mil_a100 and gh200). Refer to the official guides from NVIDIA listed in the Reference section for detailed information.

Note: The H100 described in this article is integrated in a Grace Hopper Superchip (GH200) which includes an ARM-based Grace CPU and a H100 GPU.

Be aware that NVIDIA adds new features with each new generation of GPU and some of the descriptions here may not apply to GPUs newer than the H100.

Compute Hierarchy

The following describes a GPU, followed by a Streaming Multiprocessor (SM) inside a GPU, and lastly, the CUDA cores and Tensor cores inside an SM. In addition, we introduce the concepts of threads and warps needed for GPU programming.

GPUs

A GPU is also referred to as a coprocessor or a device. A GPU node (for example, the NAS mil_a100, gh200) may contain only 1 or multiple GPUs. A GPU:

  • Connects to a CPU (also referred to as a host) via:
    • A100: PCIe 4.0 x16 with 64 GB/s bi-directional bandwidth.
    • H100 in GH200: NVLink 4.0 used as NVLink Chip-2-Chip (C2C) with 900 GB/s bi-directional bandwidth (7X that of PCIe 5.0 x16 with 128 GB/s).
  • Has multiple NVLinks for connection with other GPUs or other hardware (NVLink is a high-bandwidth, low-latency shared memory interconnect):
    • A100: 12 NVLink 3.0 links, 600 GB/s bi-directional bandwidth
    • H100: 18 NVLink 4.0 links, 900 GB/s bi-directional bandwidth
  • Includes many Streaming Multiprocessors (SMs):
    • A100: 108 SMs
    • H100: 132 SMs
  • Includes L2 cache shared among the SMs:
    • A100: 40 MB (~379 KB/SM)
    • H100: 60 MB (~465 KB/SM)
  • Includes device memory shared among the SMs:
    • A100: HBM2e 80 GB (~758 MB/SM)
    • H100: HBM3 96 GB (~745 MB/SM)

Streaming Multiprocessors

A Streaming Multiprocessor (SM) is an individual processing unit in a GPU. The SMs are responsible for executing threads in parallel. Each SM includes:

  • Many single precision (FP32) CUDA cores, double precision (FP64) CUDA cores, and Tensor cores capable of mixed precision operations.
    • A100: 64 (single), 32 (double), 4 (Tensor)
    • H100: 128 (single), 64 (double), 4 (Tensor)
  • Many 32-bit (4 bytes) registers:
    • A100: 65536, total size 256 KB
    • H100: 65536, total size 256 KB

    The large number of registers enable a GPU to run a very large number of threads.

  • A combined L1 data cache/Shared Memory shared among threads running on the cores:
    • A100: 192 KB
    • H100: 256 KB

As an example, this diagram from a NVIDIA white paper shows the configuration of an SM.

The function and capability of an SM is more in line with a single core in a multi-core CPU.

  • A CPU core containing vector registers (such as the xmm, ymm and zmm registers of x86 microprocessors) executes single instruction multiple data (SIMD) instructions on multiple data in vectors simultaneously. For example, an xmm, ymm, or zmm register can hold four, eight or sixteen single-precision floating-point numbers, respectively, for use by floating point functional units.
  • An SM, with many scalar cores and registers, executes single instruction multiple threads (SIMT) instructions in a thread block in parallel. Multiple SMs may work together on executing many thread blocks.

With CUDA, launching a kernel (i.e., a function to be executed in parallel on a GPU) with

kernel_name<<<number_of_blocks, number_of_threads_per_block>>>(...);

results in executing the kernel once for every thread in every thread block.

With many SMs per GPU, and many cores per SM, one can run thousands of threads on thousands of cores simultaneously to hide memory latency.

Threads in CUDA

A thread is a stream of instructions and data assigned to one single CUDA core. It is extremely lightweight with very little overhead.

A thread block, also known as a cooperative thread array, is a group of threads that can cooperate with one another by synchronizing their execution and efficiently sharing data through a low latency shared memory.

In CUDA, threads are organized into thread blocks. Many thread blocks are organized into a grid. Each thread block is executed by one SM and it cannot be migrated to other SMs. Prior to the introduction of thread block clusters in the H100, thread blocks were required to execute independently, and thus can execute in any order.

The maximum number of threads per thread block is 1,024, as seen in output after running the nvaccelinfo command (available after loading an nvhpc module) on a GPU. This limit is imposed by the resources available in an SM to be shared by all threads in a thread block. The number of thread blocks in a grid is dictated by the size of the data to be processed.

Within a CUDA kernel, a thread is identified through a combination of:

blockIDx.x, blockIDx.y, blockIDx.z : the indices of a block in a 3D grid
blockDim.x * blockDim.y * blockDim.z : the number of threads in a 3D block
threadIdx.x, threadIDx.y, threadIDx.z : the indices of a thread in a 3D block

For example:

i = blockIDx.x * blockDim.x + threadIdx.x ;
j = blockIDx.y * blockDim.y + threadIdx.y ; 
k = blockIDx.z * blockDim.z + threadIdx.z ; 

Note: The H100 introduces a new feature called thread block clusters, which extends the CUDA programming hierarchy to include four levels: threads, thread blocks, (optional) thread block clusters, and grids. The thread block clusters enable multiple thread blocks running concurrently across multiple SMs to synchronize and collaboratively fetch and exchange data by accessing other SM's shared memory. This new feature is known as 'distributed shared memory (DSMEM)' because the shared memory's virtual address space is logically distributed across all the blocks in the cluster.

Warps in CUDA

A warp is a group of 32 threads within a thread block that executes the same instruction simultaneously. These threads are scheduled in batches by a warp scheduler to execute on an SM.

The size of a thread block is set by an application. However, it should be a multiple of 32 due to the warp scheduling. A common choice of a thread block size is 256.

Note: In concept, executing a warp (32 threads) on multiple cores of an SM is similar to executing vector instructions (using xmm/ymm/zmm registers on 4/8/16 data elements in a vector) on a CPU core.

CUDA Cores

CUDA cores perform general-purpose computing like the CPU cores. Some of the main differences between CPU cores and CUDA cores are as follows:

  • A CPU core:
    • Is a complex and powerful core.
    • Runs at a higher clock speed,; for example, 2.45 GHz for AMD EPYC 7763, ~3.4 GHz for Arm Neoverse-V2 chip used in NVIDIA Grace CPU.
    • Can switch between multiple tasks such as out-of-order or speculative operations.
    • May have multiple vector registers and is capable of performing vector operations.
    • Has its own L1 cache and maybe also L2 cache.

    Limited parallelism is achieved from a combination of (1) vector operations for each core, and (2) multiple cores per CPU.

  • A CUDA core:
    • Is a simple and less powerful core.
    • Runs at a lower clock speed (1410/1980 MHz for A100/H100, respectively).
    • Is designed to do simple and repetitive operations focusing on number crunching, much less on out-of-order or speculative operations.
    • By itself, can only do a scalar operation through the use of scalar registers and integer or floating point functional units.
    • Does not have its own cache (the L1 cache is shared among all cores in an SM).

    Very high parallelism is achieved through thousands of CUDA cores in a GPU working concurrently.

    Note: GPU devices of compute capabilitiy 2.0 and later are capable of single and double precision arithmetic following the IEEE 754 standard and have hardware units for performing fused multiply-add in both single and double precision.

    The compute capability of A100 and H100 is 8.0 and 9.0, respectively.

Tensor Cores

Tensor cores are specialized GPU cores designed to accelerate matrix-multiply-and-add (MMA) operations (D = A x B + C), which are at the core of deep learning/AI computations for AI model training and inference and many HPC applications. The first generation of Tensor cores was introduced in the NVIDIA Volta architecture, followed by the second generation in Turing, third generation in Ampere, and fourth generation in Hopper.

Note: Ampere and Hopper resources—but not Turing—are available at NAS.

In the A100, each 3rd-generation Tensor core performs 8 x 4 x 8 (that is, 256) FP16/FP32 mixed precision FMA operations per clock. A single A100 SM, with 4 Tensor cores, can deliver 4 x 256 = 1024 FP16 FMA operations or 2048 FP16 floating point operations per clock. Besides FP16 precision input, A100 Tensor cores also add support for, among others, FP64, new BrainFloat-16 (BF16), and new TensorFloat-32 (TF32) precision input.

In the H100 4th-generation Tensor cores, FP8 precision input is supported, which provides 2x the throughput of FP16.

Memory Hierarchy

Some noticeable differences between CPU memory hierarchy and GPU memory hierarchy are:

  1. The CPU memory hierarchy looks like a pyramid where the faster forms of memory with smaller sizes sit on top and slower ones with larger sizes are below.

    The GPU memory hierarchy does not look like a pyramid. Specifically, as shown below, the L1 cache size can be smaller than the register size.

  2. There tend to be more layers of caches in CPUs than GPUs. For example, the AMD Milan and NVIDIA Grace CPU processors all have three levels of caches: L1, L2, and L3.

    On the other hand, the NVIDIA A100 and H100 only have L1 and L2 caches.

These differences speak to the fact that GPUs are designed to maximize throughput instead of minimizing latency. The high throughput is realized via a large number of registers and the use of high bandwidth memory. Instead of minimizing latency, GPUs rely on hiding latency through asychronous SIMT operations.

This section describes different levels of memory in GPUs starting from the fastest to the slowest.

Note: The latencies listed below for different levels of GPU memory are all approximations. They are meant to provide a rough comparison of latencies in GPUs and CPUs.

Registers

A register is a small, on-chip high-speed memory location used for storing data to be immediately used by the processing functional units. A register in an SM is private to a thread. Registers read/write from/to either L1 cache or the shared memory of an SM.

Each register holds a single 4-byte data. With 65,536 registers per SM, the total register size is 256 KB per SM. This size is larger than the L1/shared memory in the SM.

The large number of registers, together with the large number of cores, allow many threads to run in parallel. When some threads are waiting for data from L1/L2/global memory, other threads can be executed. This effectively hides the latency.

L1/Shared Memory

The combined L1 data cache and shared memory is on-chip and is private to each SM.

The size of this combined L1/shared memory varies for different GPUs. For A100 (192 KB), this size is smaller than the size of the registers. For the H100, the size is 256 KB, same as the size of the registers.

The partition between L1 data cache and shared memory is set by an application. However, there is a limit on the configurable shared memory size per SM: 164 KB for the A100 and 228 KB for the H100.

The latency of L1/shared memory is approximately 20 - 30 cycles as reported by independent benchmarking works listed in the Reference section. In comparison, L1 latency in CPUs is roughly five cycles.

Shared Memory

Memory access latency is lower with shared memory (located inside the processor chip) than global memory (located off the processor chip). This makes shared memory ideal for storing frequently accessed data to significantly speed up data retrieval and computation.

In a CUDA application, use __shared__ to declare a variable or array in shared memory for use by a thread block.

All of the data in a shared memory region of a thread block can be accessed by any thread in a thread block. This allows threads in a thread block to communicate and share data with one another. On the other hand, data of a thread block in the shared memory is not visible to threads in other thread blocks.

According to the Tuning Guides of various GPU architectures:

  • The A100 supports shared memory capacities of 0, 8, 16, 32, 64, 100, 132, or 164 KB per SM. A single thread block can address up to 163 KB of shared memory.
  • The H100 supports shared memory capacities of 0, 8, 16, 32, 64, 100, 132, 164, 196, and 228 KB per SM. A single thread block can address up to 227 KB of shared memory.

To maintain architectural compatability, static shared memory allocation is limited to 48 KB. Usage above 48 KB needs to be dynamic shared memory.

All blocks on an SM can share the physical memory resources provided by the SM.

L1 Data Cache

L1 cache is a buffering space when reading/writing data from/to L2 cache.

In an SM, the space unused as shared memory is left as L1 data cache. The size of an L1 cache line is 128 bytes, which is the size of data for a warp of 32 threads.

L2 Cache

L2 cache is another on-chip cache shared across all SMs. All threads in a block can access L2 cache.

L2 cache keeps data read/written from/to device memory to service requests from SMs in the GPU.

The latency of L2 cache in GPU (shared by SMs) is approximately 200 cycles as reported by independent benchmarking works for A100, listed in the Reference section.

In comparison, latency of L3 cache in CPUs (shared by CPU cores) is approximately 50 - 70 cycles.

Note: Many multi-core CPU architectures include private L1 and L2 cache for each core and shared L3 cache among the cores.

Device Memory

The device memory is off-chip and is made of different generations of High Bandwidth Memory (HBM) for different GPUs. As listed below, HBM provides much higher memory bandwidth compared to Double Data Rate 4th generation (DDR4, ~200 GB/s from 1 socket) or Low-Power DDR5X (LPDDR5X, ~500 GB/s from 1 socket) memory used in CPUs.

  • HBM2e for NAS mil_a00: 2039 GB/s
  • HBM3 for NAS gh200 nodes: up to 4000 GB/s

Latency of device memory is between about 200 to 1000 cycles, as reported by independent benchmarking work for the A100, listed in the Reference section. Assuming a CUDA core runs at 1.5 GHz, the 200 to 1000 cycles would translate to 133 to 666 nanoseconds (ns).

In comparison, memory latency in CPUs varies between roughly 50 and 200 ns.

The device memory is used as different forms of memory:

  • Global memory

    This is the primary storage for data that is shared across multiple blocks of threads.

    Most of the space in the device memory is used as global memory.

  • Local memory

    This is for storing data private to each thread that cannot fit into registers.

    The maximum amount of local memory per thread is 512 KB as seen in this NVIDIA article.

  • Constant memory

    This is a read-only memory space used for storing global data that do not change while a kernel is running. Its contents can be broadcast to multiple threads in a block. It is shared by all threads and has the lifetime of the application.

    The constant memory is very small in size - 64 KB as seen here.

  • Texture memory

    This is for storing texture data used in video rendering.

    The size used as texture memory per SM varies - 28 to 192 KB for A100 and 28 to 256 KB for H100, as seen here.

References

Contact Us

General User Assistance

Security

  • Report security issues 24x7x365
  • Toll-free: 1-877-NASA-SEC (1-877-627-2732)
  • E-mail: soc@nasa.gov

User Documentation

High-End Computing Capability (HECC) Portfolio Office

NASA High-End Computing Program

Tell Us About It

Our goal is furnish all the information you need to efficiently and effectively use the HECC resources needed for your NASA computational projects.

We welcome your input on features and topics that you would like to see included on this website.

Please send us email with your wish list and other feedback.