For single-node I/O that uses memory-based temporary filesystem, we recommend using a job-based temporary directory, $TMPDIR, instead of the node-wide /tmp directory. This recommendation is particularly important for jobs running on the NAS GPU nodes with x86_64-based CPUs (mil_a100) where node sharing is enabled.
Background
All of the NAS compute nodes—including the CPU nodes in Electra, Aitken, and the GPU nodes in Cabeus—are configured to allow up to 50% of the node's main memory for use as a temporary filesystem (/tmp) to achieve faster single-node I/O. However, using the /tmp directory is not feasible for multi-node I/O, as /tmp is a filesystem local to a single node.
Many users use the /tmp directory on the CPU nodes to speed up their jobs' I/O. When these jobs end, data under /tmp are not always removed by the application or user. In order to provide a clean /tmp directory for a new job that uses the same node, the PBS prologue will first unmount /tmp to clean up leftover data from the previous job, then will remount it for the new job.
On the GPU nodes where node sharing is enabled, the resources on each node (including /tmp) can be shared by multiple jobs. Therefore, the process of cleaning up /tmp depends on whether a new job is the only job on a node or whether there are other jobs running on the node:
If a new job starting is the only job on a node, the PBS prologue will use the same process as on CPU nodes by unmounting and remounting /tmp.
If there are other jobs currently running on the same node, the prologue will not unmount /tmp, because doing so would remove the data needed by the other running jobs. Instead, the prologue will look for /tmp/pbs.jobID directories that do not have a running job, and delete those directories if they exist. The $TMPDIR path is /var/tmp/pbs.jobid.
Therefore, to ease the removal of user data from old jobs, you should use the $TMPDIR directory instead of writing directly to /tmp.
Using the $TMPDIR Directory
Note: For some NAS systems, /var/tmp is a symlink of /tmp. For the Milan A100 (mil_a100) nodes, /var/tmp and /tmp are two different directories, but they both use the same 50% main memory pool of the nodes.
How much $TMPDIR space can a job use?
On the GPU nodes, $TMPDIR cannot use more than the amount of CPU host memory requested by a job. For example, in the following PBS job request, the 16 GB of CPU host memory requested by the job will be shared by the memory usage of the application; buffer cache usage of the application (if buffered I/O is enabled); and the usage for storing data under /tmp or $TMPDIR:
Therefore, the maximum space available for $TMPDIR is 16 GB in this example.
In addition, the combined $TMPDIR usage by all jobs running on the same node cannot exceed 50% of the node's main memory. If another job running on the same node as your job already has data stored in the job's $TMPDIR with an amount equal to the size of 50% of the node's main memory, you will encounter a "No space left on device" error when you attempt to write data into your job's $TMPDIR.
Sample Script for Using $TMPDIR
A job's resources (CPU, GPU, and memory) are accessible only during the lifetime of the job. To use $TMPDIR for I/O, you must copy data into $TMPDIR before the data are needed by your application. Any generated data needed for post-processing or restarting the next job must be copied to the $HOME or Lustre /nobackup directory before the current job is terminated. You can accomplish this by using the following sample PBS script:
