Many jobs running across multiple nodes perform setup and pre-processing tasks on the first node, then launch computations across many nodes, and finish with post-processing and cleanup steps running only on the first node. The post-processing and cleanup steps may take minutes or even hours, during which the remaining nodes in the job are idle. You can use the pbs_release_nodes command to release these idle nodes so that you are not charged for them, and they can be made available to other jobs.
Some users have already taken steps to minimize the number of idle nodes by separating their workflow into multiple jobs of appropriate sizes. The downside to this approach is additional time waiting in the queue for these multiple jobs. Using pbs_release_nodes may allow these jobs to be combined, reducing the amount of wait time in the queue.
Node Tracking and Job Accounting
When you use the pbs_release_nodes command, the released nodes are removed from the contents of $PBS_NODEFILE. The job accounting system tracks when the nodes are released, and adjusts the amount of SBUs charged accordingly.
Running the pbs_release_nodes Command
You can either include the pbs_release_nodes command in your job script or you can run it interactively, for example, on a PFE. If you run it in your job script, you can use the $PBS_JOBID environment variable to supply the job_id. You can use pbs_release_nodes multiple times in a single job.
There are two ways to use the pbs_release_nodes command. In both cases, the first node is retained.
Keep the first node and release all other nodes
If you want to retain the first node only, run the command as follows to release the remaining nodes:
pfe21% pbs_release_nodes -j job_id -a
Release one or more specific nodes
If you want to retain one or more nodes in addition to the first node, run the command as follows to release specific idle nodes:
To find out which nodes are being used by a job before you run the pbs_release_nodes command, use one of the following methods:
For interactive jobs: Run qstat -f job_id. In the output, look for the nodes listed under the exec_vnode job attribute.
For job scripts: Run cat $PBS_NODEFILE. The output provides a list of the nodes in use.
Note: $PBS_NODEFILE may have duplicate entries for nodes, depending on how resources were requested. See man pbs_resources for details.
Sample Job Script with pbs_release_nodes Command
The following sample job script keeps the first node and releases all other nodes before running post-processing steps:
#PBS -S /bin/csh
#PBS -N cfd
#PBS -l select=4:ncpus=8:mpiprocs=8:model=sky_ele
#PBS -l walltime=4:00:00
#PBS -j oe
#PBS -W group_list=a0801
#PBS -m e
module load comp-intel/
module load mpi-hpe/mpt
cd $PBS_O_WORKDIR
mpiexec dplace -s1 -c 16-23 ./grinder < run_input > intermediate_output
# Before executing some post-processing steps that will run only on the
# first node of the job you can release all other nodes
pbs_release_nodes -j $PBS_JOBID -a
# Now run post-processing steps
./post_process < intermediate_output > final_output
# -end of script-
