Microarchitecture

The Intel Broadwell processor incorporated into the Electra cluster is the 14-core E5-2680v4 model with a clock speed of 2.4 GHz. As the 14 nanometer (nm) die shrink of the Haswell microarchitecture, the Broadwell processor uses less power and is more efficient than the Haswell processor.

Instruction Sets

Like the Haswell processor, Broadwell supports single instruction, multiple data (SIMD) instruction sets, including several generations of Streaming SIMD Extensions (SSE, SSE2, SSE3, Supplemental SSE3, and SSE4), Advanced Encryption Standard (AES), and Advanced Vector Extensions (AVX and AVX2). Broadwell also supports some new instruction sets, including the Multi-Precision Add-Carry Instruction Extensions (ADX) for arbitrary-precision integer operations.

Main Features of AVX2

Features include:

• Floating-point fused multiply-add (FMA) support, which can double the number of peak floating-point operations compared with those run without FMA. With 256-bit floating-point vector registers and two floating-point functional units, each capable of FMA, a Broadwell core can deliver 16 floating-point operations.
• Integer-vector instructions support has been extended from 128-bit to 256-bit capability.
• Gather vectorization support, enhanced to enable vector elements to be loaded from non-contiguous memory locations.

Compiler Support for AVX2

AVX2 is formally supported by Intel compilers starting with version 12.1. To take advantage of AVX2, use the following compiler and options:

Compiler

We recommend using the latest Intel compiler comp-intel.

Options

Use either -xCORE-AVX2 or -axCORE-AVX2.

Both the -xCORE-AVX2 and -axCORE-AVX2 options turn on -fma, which computes a x b + c in one rounding. This provides higher accuracy than -no-fma, which computes a x b + c in two steps (first a x b, then + c) and two roundings. Turn off FMA if you want your results to match legacy ones.

For Fortran codes, consider using the option -align array32byte to align your vector arrays on 32-byte boundaries for best performance and reliability. This option is available in Intel compiler version 13 and later.

AVX2 is also supported in the GNU Compiler Collection (GCC) starting with version 4.7. Use compiler options such as -march=core-avx-2 or -mavx2, -mfma if you want to use AVX2.

Hyperthreading

Hyperthreading is turned ON.

Turbo Boost

Turbo Boost is turned ON.

Memory Subsystems

The memory hierarchy of Broadwell is as follows:

• L1 instruction cache: 32 KB, private to each core
• L1 data cache: 32 KB, private to each core
• L2 cache: 256 KB, private to each core
• L3 cache: 35 MB, shared by 14 cores in each socket
• Memory: 64 GB per socket, total of 128 GB per node

The Broadwell nodes are equipped with 2,400 MHz DDR4 memory to provide higher memory bandwidth. There are four memory channels per socket. Each channel can be connected with a maximum of two memory DIMMs. Of the eight memory DIMM slots for each socket, four are populated with 16-GB error correcting code (ECC)-registered DDR4 memory, for a total of 64 GB per socket. With two sockets in a node, the total memory per node is 128 GB.

Connecting the two sockets are two Intel QPI links running at a speed of 9.6 gigatransfers per second (GT/s). Each link contains separate lanes for the two directions. The total bandwidth (2 links x 2 directions) is 38.4 GB/sec.

Network Subsystem

The network subsystem of the Broadwell nodes is the same as that of the Haswell nodes, as shown in the following diagram. Each Broadwell node is equipped with two PCI Express (PCIe) interfaces (one from each socket). One PCIe interface is connected to the ib0 InfiniBand (IB) fabric via a single-port, four-lane, Fourteen Data Rate (4X FDR) host channel adapter (HCA), in a dual single-port FDR IB mezzanine card. The other PCIe interface is connected to the ib1 fabric via another single-port 4x FDR HCA in the same mezzanine card.