You can use a script called pdsh_gdb to debug a PBS job while the job is running. The script is available iafter loading the module /scicon/cli_tools on NAS compute systems.

Launching this script from a front-end node allows you to connect to each compute node of a PBS job and create a stack trace of each process. The aggregated stack trace from each process will be written to a user-specified directory (by default, it is written to ~/tmp).

Here is an example of how to use the pdsh_gdb script.

% module load scicon/cli_tools
% mkdir tmp
% pdsh_gdb -j job_id -d tmp -s -u username -n group_id

Note: The -n group_id is needed if you have specified a non-default project group ID (GID) via #PBS -W group_list=group_id for the PBS job that you want to debug with pdsh_gdb.

For Endeavour jobs, be sure to include both the job sequence number and the Endeavour server name, pbspl4, in the job_id (for example, 2468.pbspl4).

To view more detailed usage information, launch pdsh_gdb without any option:

% pdsh_gdb 

The pdsh_gdb script was developed by NAS staff member Steve Heistand.