This article provides basic information on getting started with running a Podman container on one or more GPUs of a NAS Grace Hopper (gh200) or Milan A100 (mil_a100) node.

Note: Running Podman containers across nodes is not allowed per NAS security policy.

For demonstration, the image nvcr.io/nvidia/cuda:12.9.1-devel-ubuntu22.04 is used in this article.

Before You Begin: Before pulling a Docker/Podman image from a remote repository, for example, the NVIDIA GPU Cloud (NGC) containers, read its online documentation (if available) to make sure that its CPU host architecture, GPU types supported (for example, H100, A100, and so on), and CUDA version are compatible with the GPU compute node you intend to run the container on.

Pulling an Image with the Correct Architecture

The container image you are interested in using might be built for one or both of these CPU host architectures:

• CPU host architecture on the gh200 nodes: arm64
• CPU host architecture on the mil_a100 nodes: x86_64 or amd64

Before pulling a container image, check the manifest list of the container. In this example, the output of the following command line will show that both arm64 and amd64 host architectures are supported:

NAS_front_end% podman manifest inspect nvcr.io/nvidia/cuda:12.9.1-devel-ubuntu22.04

Then, pull the image you want using the option --arch=xxx. For example, to pull an image with the arm64 architecture, run:

NAS_front_end% podman pull --arch=arm64 nvcr.io/nvidia/cuda:12.9.1-devel-ubuntu22.04

You can also check the architecture after the image has been stored locally. For example:

NAS_front_end% podman images
REPOSITORY          TAG                       IMAGE ID      CREATED        SIZE
nvcr.io/nvidia/cuda 12.9.1-devel-ubuntu22.04  ec9b21c93944  2 weeks ago    8.77 GB

NAS_front_end% podman image inspect ec9b21c93944 | grep -i arch

Tip: You can modify the tag of the image with a different name to make its architecture obvious to you. For example, to tag an image with arm64, run:

NAS_front_end% podman tag ec9b21c93944 nvcr.io/nvidia/cuda:12.9.1-devel-ubuntu22.04-arm64

NAS_front_end% podman images
REPOSITORY           TAG                             IMAGE ID      CREATED        SIZE
nvcr.io/nvidia/cuda  12.9.1-devel-ubuntu22.04        ec9b21c93944  2 weeks ago    8.77 GB
nvcr.io/nvidia/cuda  12.9.1-devel-ubuntu22.04-arm64  ec9b21c93944  2 weeks ago    8.77 GB

NAS_front_end% podman rmi nvcr.io/nvidia/cuda:12.9.1-devel-ubuntu22.04

NAS_front_end% podman images
REPOSITORY           TAG                             IMAGE ID      CREATED        SIZE
nvcr.io/nvidia/cuda  12.9.1-devel-ubuntu22.04-arm64  ec9b21c93944  2 weeks ago    8.77 GB

Testing the Image with the NVIDIA System Management Interface (nvidia-smi)

1. Submit an interactive PBS job to access a GPU compute node. This example uses a gh200 node:

   NAS_front_end% qsub -I -q xxx -lselect=1:ncpus=72:ngpus=1:mem=460GB:model=gh200

2. Add all GPU devices on the node to the container.
   • For the gh200 nodes running with Red Hat 9.x and Podman 5.x, add to the podman run command:

     --gpus=all

   • For the mil_a100 nodes running with Red Hat 8.x and Podman 4.9x, add:

     --device nvidia.com/gpu=all 
     or 
     --device=nvidia.com/gpu=all

     Note: A mil_a100 node has a total of four GPUs. If you don't need all four GPUs, you can specify which one to use via a relative GPU ID or the GPU's universally unique identifier (UUID). For example, to add GPU 1 and GPU 3 (as seen on the host) to the container, use:

     --device=nvidia.com/gpu=1 --device=nvidia.com/gpu=3

     To use the UUIDs:

     --device nvidia.com/gpu=GPU-d4f2037f-1d66-de91-3f3c-1341443640b0 
     --device nvidia.com/gpu=GPU-2f2f99cf-a25c-bcdc-0389-fcc95a620c72

3. Verify that the container can see and use the GPU(s):

   Run the command nvidia-smi -L inside the container. For example, on a gh200 node:

   GPU_node% podman run -it --rm --gpus=all ec9b21c93944 nvidia-smi -L
   
   ========== 
   == CUDA == 
   ========== 
   CUDA Version 12.9.1 
   ... 
   GPU 0: NVIDIA GH200 480GB (UUID: GPU-bf077a13-dcd7-7c4d-6ba5-15ba1dddf6f3)