To help you prepare for running jobs on Electra's Broadwell compute nodes, this short review includes information on the general configuration, compiling your code, running PBS jobs, and checking allocation usage.

Overview of Electra Broadwell Nodes

Electra is housed in a module outside of the primary NAS facility. The system has 16 Broadwell racks, each containing 72 nodes. Each of the 72 nodes contains two 14-core E5-2680v4 (2.4 GHz) processor chips and 128 GB of memory.

Electra's InfiniBand fabric (ib1) is used mainly for I/O and is connected to the NAS Lustre filesystems. In addition, Electra and all other NAS compute systems share the same home filesystems. It also share the front-end systems (PFE) and PBS server (pbspl1) with some NAS compute nodes such as sky_ele, sky_el, and rom_ait. You can access Electra only through PBS jobs submitted from the PFEs.

Usage of Electra Broadwell nodes is charged to your project's HECC Standard Billing Unit (SBU) allocation.

Compiling Your Code For Broadwell Nodes

Like the Haswell processors, the Broadwell processors support the Advanced Vector Extensions 2 (AVX2) instructions, in addition to AVX (introduced with Sandy Bridge processors), SSE4.2 (introduced with Nehalem processors), and earlier generations of SSE.

AVX2 supports floating point fused multiply-add, integer vector instructions extended to 256-bit, and vectorization gather support, among other features. Your application may be able to take advantage of AVX2, however, not all applications can make effective use of this set of instructions. We recommend that you use the latest Intel compiler, by using the command module load comp-intel and experiment with the following sets of compiler options before making production runs on the Broadwell nodes:

• -O2 -xCORE-AVX2
• -O3 -xCORE-AVX2

These compiler options generate an executable that is optimized for running on Broadwell and Haswell, but cannot run on any of the older processor types. If you prefer to generate a single executable that will run on many Intel processor types (Sandy Bridge, Ivy Bridge, Haswell, Broadwell, Skylake, and Cascade Lake), try using one of the following sets of compiler options:

• -O2 (or -O3)
• -O2 (or -O3) -axCORE-AVX512,CORE-AVX2 -xAVX

The use of -axCORE-AVX512,CORE-AVX2 allows the compiler to generate multiple code paths with suitable optimization to be determined at run time.

If your application does not show performance improvements with -xCORE-AVX2 or -axCORE-AVX512,CORE-AVX2 -xAVX (as compared with just -O2 or -O3) when running on Broadwell nodes, it is more advantageous to use the executable built with just -o2 or -o3 for production runs.

Notes:

• If you have an MPI code, we strongly recommend that you load the mpi-hpe/mpt module, which always points to the NAS recommended MPT version.
• Ensure your jobs run correctly on Broadwell nodes before you start production work.

Running PBS Jobs on Broadwell Nodes

To request Broadwell nodes, use model=bro_ele in your PBS script.

#PBS -l select=10:ncpus=28:mpiprocs=28:model=bro_ele
#PBS -l site=static_broadwell

Sample PBS Script For Broadwell Nodes

#PBS -l select=10:ncpus=28:mpiprocs=28:model=bro_ele
#PBS -l walltime=8:00:00
#PBS -q normal
module load comp-intel/2020.4.304 mpi-hpe/mpt
cd $PBS_O_WORKDIR
mpiexec -np 280 ./a.out

For more information about Broadwell nodes, see:

• Electra Configuration Details
• Broadwell Processors