If your job does not run after it has been successfully submitted, it might be due to one of the following reasons:
The queue has reached its maximum run limit
Your job is waiting for resources
Your mission share has run out
The system is going into dedicated time
Scheduling is turned off
Your job has been placed on hold
Your home filesystem or default /nobackup filesystem is down
Your hard quota limit for disk space has been exceeded
Each scenario is described in the sections below, along with troubleshooting steps. The following commands provide useful information about the job that can help you resolve the issue:
tracejob job_id
Displays log messages for a PBS job on the PBS server. The table below displays the relationship among the various NAS compute systems, the compute model types, the front-end systems to use, and the PBS server that serves each model type. For jobs served by the PBS server pbspl1 or pbspl4, you can log into the server and run the command yourself. For jobs served by other PBS server, such as pbs05a, the command can only be run by a NAS staff member.
qstat -as job_id
Displays all information about a job on a front-end for the compute resource model type you submitted your job for. When checking Endeavour jobs on a PFE, the job_id must include both the job sequence and the server name, pbspl4. For example:
qstat -as 2468.pbspl4
System Name
Compute Model
Front-End
PBS Server
LDAN
ldan
pfe[20-22,25,26],lfe[5-8]
pbspl1
Electra
bro_ele
pfe[20-22,25,26]
pbspl1
Electra
sky_ele
pfe[20-22,25,26]
pbspl1
Aitken
cas_ait
pfe[20-22,25,26]
pbspl1
Aitken
rom_ait
pfe[20-22,25,26]
pbspl1
Aitken
mil_ait
afe[01,02]
pbspl1
Endeavour
cas_end
pfe[20-22,25,26]
pbspl4
Cabeus
rom_gpu
cfe[01,02]
pbspl4
Cabeus
mil_a100
cfe[01,02]
pbspl4
Cabeus
gh200
cghfe[01,02]
pbs05a
Athena
tur_ath
athfe[01-04]
pbs06a
The Queue Has Reached Its Maximum Run Limit
Some queues have a maximum run (max_run) limit—a specified maximum number of jobs that each user can run. If the number of jobs you are running has reached that limit, any jobs waiting in the queue will not begin until one of the running jobs is completed or terminated.
To check if a queue has a max_run limit set, login to an appropriate front-end and run:
qstat -fQ queue_name | grep max_run
For example, the devel queue served by pbspl1 has a max_run limit of 1 per user, the debug and ldan queues served by pbspl1 each have a max_run limit of 2 per user.
Your Job is Waiting for Resources
Your job might be waiting for resources for one of the following reasons:
All resources are busy running jobs, and no other jobs can be started until resources become available again
PBS is draining the system and not running any new jobs in order to accommodate a high-priority job
Users have submitted too many jobs at once (for example, more than 100), so the PBS scheduler is busy sorting jobs and cannot start new jobs effectively
You requested a specific node or group of nodes to run your job, which might cause the job to wait in the queue longer than if nodes were not specified
To view job status and events for jobd served by pbspl1 or pbspl4, run the tracejob utility on the appropriate PBS server. For example:
pfe25% ssh pbspl1
pbspl1% tracejob 234567
Job: 234567.pbspl1.nas.nasa.gov
02/15/2019 00:23:55 S Job Modified at request of Scheduler@pbspl1.nas.nasa.gov
02/15/2019 00:23:55 L No available resources on nodes
02/15/2019 00:38:47 S Job Modified at request of Scheduler@pbspl1.nas.nasa.gov
02/15/2019 00:50:30 S Job Modified at request of Scheduler@pbspl1.nas.nasa.gov
02/15/2019 01:51:21 S Job Modified at request of Scheduler@pbspl1.nas.nasa.gov
02/15/2019 01:55:38 S Job Modified at request of Scheduler@pbspl1.nas.nasa.gov
02/15/2019 02:16:44 S Job Modified at request of Scheduler@pbspl1.nas.nasa.gov
02/15/2019 07:39:48 L Considering job to run
02/15/2019 07:39:48 L Job is requesting an exclusive node and node is in use
TIP: If the scheduler has not yet reviewed the job, no information will be available and the tracejob utility will not provide any output.
If your job requests an exclusive node and that node is in use, you can wait for the requested node, request a different node, or submit the job again without requesting a specific node.
To view the current node usage for each processor type, run the node_stats.sh script. For example:
If all of the cores within your mission directorate's share have been assigned or if running the new job would exceed your mission share, your job will not run. If resources appear to be available, they belong to other missions.
To view all information about your job, run qstat -as job_id. In the following sample output, a comment (line 5) indicates that the job would exceed the mission share:
pfe25% qstat -as 778574
JobID User Queue Jobname CPUs wallt Ss wallt Eff wallt
------------- ------- ------ ------- ---- ----- -- ----- --- -----
778574.pbspl1 zsmith normal my_GC 12 04:00 Q 07:06 -- 04:00
Job would exceed mission CPU share
To view the distribution of shares among all mission directorates, run qstat -W shares=-. For example:
If running your job would exceed your mission share, you might be able to borrow nodes from other mission directorates. To borrow nodes, your job must not request a wall-clock time that is too long (more than 4 hours). See Mission Shares Policy for more information.
The System is Going into Dedicated Time
When dedicated time is scheduled for hardware and/or software work, the PBS scheduler will not start a job if its projected end-time runs past the beginning of the dedicated time.
If you can reduce the requested wall-clock time so that your job will finish running prior to dedicated time, then your job can then be considered for running. To change the wall-clock time request for your job, follow the example below:
pfe25% qalter -l walltime=hh:mm:ss job_id
TIP: For Endeavour jobs, the job_id must include both the job sequence number and the Endeavour server name (for example, 2468.pbspl4).
To find out whether the system is in dedicated time, run the schedule all command. For example:
pfe25% schedule all
No scheduled downtime for the specified period.
Scheduling is Turned Off
Sometimes job scheduling is turned off by a system administrator. This is usually done when system or PBS issues need to be resolved before jobs can be scheduled to run. When this happens, you should see the following message near the beginning of the qstat -au your_userid output.
+++Scheduling turned off.
Your Job Has Been Placed On Hold ("H" Mode)
A job can be placed on hold either by the job owner or by someone who has root privilege, such as a system administrator. If your job has been placed on hold by a system administrator, you should get an email explaining the reason for the hold.
Your Home Filesystem or Default /nobackup Filesystem is Down
When a PBS job starts, the PBS prologue checks to determine whether your home filesystem and default /nobackup filesystem are available before executing the commands in your script. If your default /nobackup filesystem is down, PBS cannot run your job and will put the job back in the queue. If your PBS job does not need any file in that filesystem, you can tell PBS that your job will not use the default /nobackup to allow your job to run.
For example, if your default is /nobackupp17, you can add the following in your PBS script:
#PBS -l nobackupp17=0
Your Hard Quota Limit for Disk Space Has Been Exceeded
NAS quotas have hard limits and soft limits. Hard limits should never be exceeded. Soft limits can be exceeded temporarily, for a grace period of 14 days. If your data remains over the soft limit for more than 14 days, the soft limit is enforced as a hard limit.
