The Milan nodes are currently installed on the computer floor of the main NAS building.

This is a simplified configuration of an EPYC Milan node with two sockets. Each socket contains 64 cores in eight Core Complex Dies (CCDs, each enclosed in a green box) and one I/O die (IOD, enclosed in a yellow box). The ♾ sign represents the Infinity Fabric.

The Milan processor to be incorporated into the Aitken cluster is EPYC 7763 with a base clock of 2.45 GHz and a maximum boost clock of 3.5 GHz.

The Milan (Zen 3) microarchitecture bears many similarities with the Rome (Zen 2) microarchitecture. For a description on the processor, cache, memory subsystems, intra-socket and inter-socket interconnect for Milan, refer to the information provided for the Rome processor.

Main Differences Between Milan and Rome

Key differences between the two microarchitectures are described in this section.

Core Complex Dies (CCDs)

A Rome CCD contains two separate Core CompleXes (CCXs), each with 4 cores and 16 MB of L3 cache, while a Milan CCD contains a unified CCX with 8 cores and 32 MB of L3 cache which results in:

• Direct communication and therefore lower core-to-core latency among the 8 cores in a CCD.
• Doubling the size of L3 cache: all 32 MB can be accessed by a single core in the CCD.
• Average L3 cache latency: 39 cycles for Rome, 46 cycles for Milan.
• 19% higher instructions per cycle (IPC) on average.

Clock Coupling

With Rome, you can choose whether to couple the Infinity Fabric clock (FCLK) and the DRAM memory clock (MEMCLK) via a BIOS setting. With Milan, these two clocks are synchronized, resulting in a reduced memory latency.

Note: The maximum Infinity Fabric clock speed is 1,467 MHz for Rome and 1,600 MHz for Milan.

NUMA Domain

In Milan, when the BIOS setting BIOS.ProcSettings.CcxAsNumaDomain is enabled, each CCD is exposed as a NUMA node and a dual-socket Milan node will have a total of 16 NUMA nodes instead of 8. Use the following command to find out the number of NUMA nodes configured in a Milan node: numactl -H

Note: Each NAS Rome node is configured with 4 NUMA Nodes (NPS4) per socket, for a total of 8 NUMA Nodes per Rome node.

CPU Core

Milan is a 64-bit x86 server microprocessor. Like the Rome processor, Milan supports AVX2 (256 bit) but not the AVX-512 instructions. For a complete list of instructions supported by Milan, run the following command on a Milan node: cat /proc/cpuinfo

Each Milan core has two 256-bit Fused Multiply-Add (FMA) units and can deliver up to 16 double-precision floating point operations (flops) per cycle. Therefore, the peak double-precision flops per node is: 128 cores x 2.45 GHz x 16 = 5.02 teraflops (TF). With 512 Milan nodes in Aitken, the peak is 2.57 petaflops (PF).

The simultaneous multi-threading (SMT) capability allows 2 execution threads to execute simultaneously per core, is enabled/disabled in BIOS.

Note: When the Milan nodes are in their early production period, it is likely that some nodes will have SMT enabled while others will have SMT disabled. To check the SMT setting on a Milan node, use the following command:

Milan_node% lscpu | grep "Thread(s) per core"

If the output is 2, SMT is enabled. If it is 1, SMT is disabled.

Note: Linux support for the Zen microarchitecture started with Linux kernel 4.10. The current Red Hat-based operating system—Tri-Lab Operating System Stack Version 4 (TOSS 4)—uses Linux kernel 4.18.0, which supports the Rome and Milan nodes.

You can use the AMD Optimizing C/C++ (AOCC), GNU, and Intel compilers to compile codes to run on the Milan nodes. To generate optimized x64 code for the Zen3 microarchitecture, consider these compiler flags:

• AOCC compiler: -march=znver3

  Note: AOCC consists of a C/C++ compiler (clang) and a Fortran compiler (flang). AOCL (AMD Open CPU Libraries = AMS CPU math library suite) consists of optimized math libraries. AOCC 3.0 and AOCL 3.0 became available when the 3rd Gen EPYC processors launched. For information about additional optimization for the 4th generation EPYC processors, see AOCC and AOCL.

• GNU compiler: -march=znver3

  Note: -march=znver3 is available in gcc 11.x and above but not earlier versions. With gcc 8.x, use -march=znver1 instead.

• Intel compiler: -march=core-avx2 (preferred) or -axCORE-AVX2

For more information, see AMD's Compiler Options Quick Reference Guide for the AMD EPYC 7xx3-series Processors.

Inter-Node Network

There are 512 Milan nodes in the Aitken cluster, which uses the HPE Apollo 9000 system architecture. They are partitioned as follows:

• 4 racks in the cluster (512 nodes total)
• 4 enclosures per rack (128 nodes/rack)
• 8 trays per enclosure (32 nodes/enclosure)
• 4 nodes per compute tray

The hostname of each node is r[x1]c[x2]t[x3]n[x4] where x1 = 705 - 708; x2 = 1 to 4; x3 = 1 to 8; and x4 = 1 to 4.

Like the Rome nodes, the inter-node connection of the Milan nodes in Aitken relies on the use of PCIe 4.0 lanes and one dual single-port IB HDR 200 Gbit/s Mezzanine card on each node and two IB HDR 200 Gbit/s premium switches per enclosure. One switch per enclosure facilitates the ib0 fabric while the other facilitates the ib1 fabric. For more information, see the Inter-Node Network section of the Rome Processors.

Connection between Milan and Rome

When the Milan nodes are relocated to the new compute module, they will be connected to the Rome nodes with additional HDR cables on the 8th dimension of the Aitken cluster.

References

• AMD EPYC 7763
• High Performance Computing (HPC) Tuning Guide for the AMD EPYC 7003 Series Processors
• Workload Tuning Guide for AMD EPYC 7003 Processors
• Compiler Options Quick Reference Guide for the AMD EPYC 7xx3-series Processors