The sky_wide queue is intended to help you run very wide ("at scale") jobs on Electra's Skylake nodes by ensuring that at least once a week, the jobs in the queue will have enough available nodes to run. Job requests submitted to the sky_wide queue must specify between 576 and 1,152 Skylake nodes (between 25% and 50% of the total number of Skylake nodes, which is currently 2,304). The maximum wall time is currently 5 days.

Submitting Your Job to the sky_wide Queue

You can submit jobs to the queue each week. Jobs submitted to the queue by Thursday at 5:00 p.m. (Pacific Time) are scheduled to run starting the following Monday at 10:00 a.m. PBS runs the jobs one at a time, prioritized by job size; wider jobs are run before those requesting fewer nodes.

When it is your job's turn to run, a reservation is created for the number of nodes requested, plus a few extra nodes. The job is then moved from the sky_wide queue into the reservation's queue, where it starts running.

If the job fails early, you can submit a replacement job to the reservation queue without losing your place in line. Eventually, the queue will go idle or the reservation will expire, and the next job will be selected.

Important: Jobs submitted to the sky_wide queue after Thursday at 5:00 p.m. (Pacific Time) will not be scheduled to run the following Monday; rather, they will carry over to the next week.

Sample Job Submission Script

The following sample script includes several instructions. Each line is described below the script.

#PBS -q sky_wide
#PBS -l select=NNN:model=sky_ele:ncpus=nn:aoe=sles12
#PBS -l max_walltime="24:00:00",min_walltime="5:30:00"
#PBS -m abe
#PBS -l site="extra_nodes=4:extra_time=20:idle_timeout=20"

Job Submission

The first two lines request the sky_wide queue and submit your job:

#PBS -q sky_wide
#PBS -l select=NNN:model=sky_ele:ncpus=nn:aoe=sles12

Wall Time Request

This line specifies how much wall time the job needs. Keep in mind that if the job fails, you can submit a replacement job, but its wall time must fit within the time remaining from the original job. By specifying a maximum and minimum useful wall time in the original job script, you can use the same script, unedited, for the replacement job. The sample script requests a total of 24 hours of time, and any replacement job will not run unless at least 5.5 hours of wall time remain:

#PBS -l max_walltime="24:00:00",min_walltime="5:30:00"

Note: This technique, a PBS feature known as shrink-to-fit, is described in the article Running Jobs Before Dedicated Time.

Notification

If there are several jobs in the sky_wide queue, your job might not be the first to run. You can tell PBS to send you an email notification when your job starts, ends, or aborts, so you'll know whether you need to submit a replacement job:

#PBS -m abe

Additional Options

PBS automatically adds a few extra nodes to the job reservation, beyond the number of nodes in the request, in case nodes fail during the run. That way, a replacement job will still have enough reserved nodes. The default is 12 nodes. To specify a different value, use site=extra_nodes=nn:

#PBS -l site=extra_nodes=4

Recommendation: There are four Skylake nodes per blade. Because it is possible for a failing node to take down the whole blade, it makes sense to set aside extra nodes in multiples of four.

PBS also adds extra time to each job reservation. The default is 15 minutes. To specify different value, use site=extra_time=nn:

#PBS -l site=extra_time=20

Once your per-job reservation starts, PBS monitors the reservation. If the reservation queue is idle for a set number of minutes, the reservation is terminated early. The default is 30 minutes. To specify a different value, use site=idle_timeout=nn:

#PBS -l site=idle_timeout=20

To specify multiple site options in one line, separate them by colons (":"), as shown in the sample script:

#PBS -l site="extra_nodes=4:extra_time=20:idle_timeout=20"

Note: When you consider these requests, keep in mind that your account is billed for the full number of nodes and for the actual duration of the job reservation.

Checking Status of Wide Jobs

Run the pbs_scale_jobs command, which uses qstat to display the status of jobs submitted to the sky_wide queue. For example:

$ pbs_scale_jobs
                                                  Req'd    Elap
JobID        User     Queue    Jobname    TSK Nds wallt S wallt Eff
------------ -------- -------- -------- ----- --- ----- - ----- ---
3827.testpbs zsmith   R3853    testing3 24000 600 10:15 Q 02:26  --
3792.testpbs zsmith   sky_wide testing1 23040 576 00:15 Q 03:11  --
3808.testpbs zsmith   sky_wide testing2 23040 576 00:15 Q 03:11  --

== Jobs to consider next cycle ==
                                                  Req'd    Elap
JobID        User     Queue    Jobname    TSK Nds wallt S wallt Eff
------------ -------- -------- -------- ----- --- ----- - ----- ---
4001.testpbs zsmith   sky_wide testing5 23040 576 00:15 Q 01:10  --
4002.testpbs zsmith   sky_wide testing6 23040 576 00:15 Q 01:10  --

In this example, the jobs named testing1–3 were submitted before the cutoff time. Jobs testing5 and testing6 were submitted too late for this week's run and will be considered next week.

The testing3 job has been moved to its run queue (R3853, in this example). Note that although testing3 was submitted after testing1 and 2, the PBS scheduler runs it first because it uses more nodes than the other jobs. 

Submitting Replacement Jobs

Replacement jobs are submitted to the reservation queue associated with the original job, rather than to the sky_wide queue. This queue will have a name like R12345, which changes for each reservation.

However, there is an easier way to submit a replacement job to the reservation queue. You can use the queue's alias, which takes the form NAS_job_username (where username is your NAS user name).

Also, instead of editing your qsub script to change the queue name when you submit the replacement, you can use the original job script, but override the queue on the command line. For example:

qsub -q NAS_job_zsmith job_script