When you request PBS resources for your job, you must specify one or more Aitken or Electra node model (processor) types, so it is helpful to keep this information in mind:

• Usage charges on each HECC node type are based on a common Standard Billing Unit (SBU).
• The total physical memory of each node type varies from 128 GB to 512 GB.

The following table provides the SBU rate and memory for each Aitken and Electra node type:

Note: The amount of memory available to a PBS job is less than the node's total physical memory, because the system kernel can use up to 4 GB of memory in each node.

Checking Available Nodes Before Requesting Resources

If your application can run on any of the Aitken or Electra processor types, you might be able to reduce wait time by requesting a type that has more free nodes (nodes currently unoccupied by other running jobs).

Before you submit your job, you can find out the current numbers of used and free nodes for each processor type by running the node_stats.sh script, which is accessible by loading the scicon/cli_tools modulefile. For example:

% module load scicon/cli_tools
% node_stats.sh

You can also identify which processor models are in use for each job currently running on HECC systems. Run the following command, and check the Model field in the output:

% qstat -a -W o=+model

Resource Request Examples

As shown in these examples, use the model=[bro_ele,sky_ele,cas_ait,rom_ait, mil_ait] attribute to request the node model type(s) for your job.

The default operating system for all model types is TOSS (Tri-Lab Operating System Stack). For all model types, the default is aoe=toss4:aoe=toss4 to your PBS resource request.)

Example 1: Basic Resource Request

Each of the following sample command lines requests a single node model type for a 128-process job:

#PBS -l select=5:ncpus=28:model=bro_ele
#to run all 28 cores on each of the 5 Broadwell nodes
#(12 cores in the 5th node will go unused)

#PBS -l select=4:ncpus=40:model=sky_ele
#to run all 40 cores on each of the 4 Skylake nodes
#(32 cores in the 4th node will go unused)

#PBS -l select=4:ncpus=40:model=cas_ait
#to run all 40 cores on each of the 4 Cascade Lake nodes
#(32 cores in the 4th node will go unused)

#PBS -l select=1:ncpus=128:model=rom_ait
#to run all 128 cores on 1 Rome node

#PBS -l select=1:ncpus=128:model=mil_ait
#to run all 128 cores on 1 Milan node

You can specify both the queue type -q normal, debug, long, low, or devel and the processor type. For example:

#PBS -q normal
#PBS -l select=128:ncpus=128:model=mil_ait

Example 2: Requesting Different ncpus in a Multi-Node Job

For a multi-node PBS job, the number of CPUs ncpus used in each node can be different. This feature is useful for jobs that need more memory for some processes, but less for other processes. You can request resources in "chunks" for a job with varying ncpus per node.

This example shows a request of two resource chunks. In the first chunk, one CPU in one Broadwell node is requested, and in the second chunk, 28 CPUs in each of three other Broadwell nodes are requested:

#PBS -l select=1:ncpus=1:model=bro_ele+3:ncpus=8:model=bro_ele

Example 3: Requesting Multiple Node Model Types

A PBS job can run across more than one node model type if the nodes are all from Electra or all from Aitken, but not across the systems. This can be useful in two scenarios:

• When you cannot find enough free nodes within one model for your job
• When some of your processes need more memory while others need much less

You can accomplish this by specifying the resources in chunks, with one chunk requesting one node model type and another chunk requesting a different type.

This example shows a request for one Skylake node (which provides ~185 GB of memory per node) and three Broadwell nodes (which provide ~120 GB of memory per node):

#PBS -lplace=scatter:excl:group=model

#PBS -lselect=1:ncpus=24:mpiprocs=24:model=sky_ele+3:ncpus=28:mpiprocs=28:model=bro_ele