You can use the qsh.pl tool to check the memory usage of jobs running on Aitken or Electra. This tool is a Perl script that uses Secure Shell (SSH) protocol to access all of the nodes used by a PBS job in order to run a command that you supply.

For Aitken and Electra jobs, you can run qsh.pl on the PFEs.

The qsh.pl tool is available in the /u/scicon/tools/bin directory.

Syntax:

pfe21% qsh.pl job_id command 

You can use this script to check the amount of free memory in the nodes of your PBS job.

Example:

pfe21% qsh.pl 30329 "cat /proc/meminfo"

running "cat /proc/meminfo" on:  r56i2n14 r56i2n15
r56i2n14 :
  MemTotal:      8079728 kB
  MemFree:        857936 kB
  Buffers:             0 kB
  Cached:        3775472 kB
...
r56i2n15 :
  MemTotal:      8079728 kB
  MemFree:       5840920 kB
  Buffers:             0 kB
  Cached:         784280 kB
...

The qsh.pl script was provided by NAS staff member Bob Hood.