Endeavour3 and Endeavour4 are two HPE Superdome Flex servers. They replace Endeavour1 and Endeavour2 as the NAS resources for supporting applications that need access to large cache-coherent, global shared-memory capabilities in a single system image (SSI).

Each of these two systems has 896 cores, six terabytes (TB) of memory, and a theoretical double-precision floating point peak performance of 77.414 teraflops. Each system is configured with a total of 32 sockets in this hierarchy:

• 28 cores and 192 gigabytes (GB) of memory in one socket.
• Four sockets in one chassis.
• Eight chassis in one system.

Processors and Memory Configuration in a Socket

At the processor level, Endeavour3 and 4 are based on the Intel Xeon Platinum 8280 processor, which is similar to Aitken's Intel Xeon Gold 6248 Cascade Lake processors.

The main improvements provided by the Platinum 8280 processor are:

• The Endeavour systems have more cores per socket: 28 cores vs. 20 cores.
• The CPU clock speed is higher: 2.70 GHz vs. 2.50 GHz.
• The L3 cache size per socket is greater: 38.5 megabytes (MB) vs. 27.5 MB.
• The memory size per socket is increased: 192 GB vs. 96 GB.
• There is more DDR4 memory DIMM per memory channel: 32 GB vs. 16 GB.

For instruction sets, cache hierarchy, and memory subsystem information, see Preparing to Run on Endeavour.

Chassis Configuration

As shown in the following base chassis diagram, published by HPE, there are four sockets connected in a ring fashion via 10.4 GT/s Intel Ultra Path Interconnect (UPI) links. Each socket is also connected to one of two custom-designed HPE Superdome Flex ASICs, via an Intel UPI link, to send remote targeted cache-coherent data traffic to the external chassis.

Memory latency within each socket is approximately 100 nanoseconds (ns). However, due to sub-NUMA design within a socket (see the Cascade Lake configuration diagram), it will be faster if the CPU that a process runs on is in the same sub-domain as the memory where the data is located. Memory latency between two sockets through a UPI link is approximately 130 ns.

All memory channels in each chassis are fully independent, and they can run simultaneously at DRAM data transfer rates up to 2,933 MegaTransfers per second (MT/s). This provides a theoretical total memory bandwidth of 564 GB/s; (memory bandwidth per socket is 141 GB/s). HPE's published STREAM TRIAD memory bandwidth is more than 360 GB/s per chassis.

System Interconnect

The HPE Superdome Flex server has a modular architecture based on a four-socket chassis. As shown in the following diagram published by HPE, the eight chassis of a 32-socket server are connected through a cabled crossbar interconnection fabric called HPE Superdome Flex Grid.

Each of the 16 application-specific integrated circuits (ASICs) in the grid has a very large directory cache to track the cache line state and maintain coherency across all attached processor sockets. A Superdome Flex ASIC (also known as NUMAlink 8) provides 16 flex grid ports, each capable of 13.3 GB/s data rates for maximum flex grid bandwidth. The total bi-sectional crossbar grid bandwidth for a 32-socket Superdome Flex server is more than 850 GB/s. With this crossbar interconnection, memory latency is minimized (< 400 ns), since there is only one hop between any two ASICs.

The HPE Superdome Flex Grid provides adaptive routing features designed specifically to enhance performance by routing traffic through the optimal latency path available, and provides superior uptime by automatically routing traffic around failed components.

Processor, Memory, and Network Subsystems Configuration

Detailed configuration statistics for Endeavour 3 and Endeavour 4 are listed below:

References

HPE Superdome Flex Server Architecture and RAS (PDF)