You can use the qtop.pl tool to check the memory usage of jobs running on Aitken or Electra. This tool is a Perl script that uses Secure Shell (SSH) to access the nodes of a PBS job in order to perform the top command. The output provides memory usage information for the whole node and for each process.

For Aitken and Electra jobs, you can run qtop.pl on the PFEs.

The qtop.pl tool is available after loading the module scicon/cli_tools or by adding to your path in the /u/scicon/tools/bin directory.

For example:
set path = ( $path /u/scicon/tools/bin )

Syntax:

pfe21% qtop.pl [-b] [-p n] [-P s] [-h n] [-H s] [-t s] [-N s] job_id

-b

(for running in background or batch) don't run 'resize' command

-p n

show at most n processes per host

-P s

show only procs in s, a comma-separated list of ranges
e.g. -P 1,8-9

-h

don't show the column header line

-H s

show only header lines in s, comma-separated ranges
e.g. -H 1-2,7
e.g. -H 0 (don't show any lines)

-t s

pass string s (must be one argument) to top command

-n s

show output only from nodes in s, comma-separated ranges
e.g. -n 0,2-3 (relative node numbers)

-N s

show output only from nodes in s, a comma-separated list
e.g. -N r1i1n14,r1i1n15 (absolute node numbers)

Example: To skip the header and list 8 procs per host:

pfe21% qtop.pl -H 0 -p 8 996093
all nodes in job 996093:  r184i2n12
r184i2n12   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
          20027 zsmith    25   0 23.8g 148m 5320 R  101  0.6   5172:37 a.out
          20028 zsmith    25   0 23.8g 140m 5140 R  101  0.6   5173:35 a.out
          20029 zsmith    25   0 23.9g 286m 6640 R  101  1.2   5172:23 a.out
          20030 zsmith    25   0 23.9g 245m 5040 R  101  1.0   5171:18 a.out
          20031 zsmith    25   0 23.9g 265m 6040 R  101  1.1   5171:46 a.out
          20032 zsmith    25   0 23.9g 246m 5300 R  101  1.0   5171:00 a.out
          20033 zsmith    25   0 23.8g 158m 5476 R  101  0.7   5172:41 a.out
          20034 zsmith    25   0 23.8g 148m 5280 R  101  0.6   5173:02 a.out

The qtop.pl script was provided by NAS staff member Bob Hood.