The Lou data analysis nodes (LDANs) provide dedicated PBS resources to perform post-processing tasks on your Lou mass storage data. You can also use them for performing pre-processing or post-processing tasks on active data residing in the home or /nobackup filesystems of NAS compute resources. The LDANs can be accessed only through PBS jobs in which resources are dedicated to the job owner.

The LDANs contain Intel Xeon Gold 6154 "Skylake" processors, equipped with more memory than the Electra Skylake nodes, as follows:

• Each ldan[11-12] has 36 cores and 768 GB of memory
• Each ldan[13-14] has 36 cores and 1.5 TB of memory

The Lou, Lustre, and home filesystems of NAS compute resources are mounted on the LDANs. You can access them in a PBS job running on an LDAN by using the following paths:

Lou:

/u/username or /lou/lax where X is one of 1-6

Lustre:

/nobackup/username

Home:

/homeX/u/username where X is one of 1,3,4,5,6,7

Software Modules

All of the software packages that you use on the front ends or compute nodes of any NAS x86_64 compute resouces are also available on the LDANs. However, both the Lou front-end (LFE) nodes and LDANs use startup files from your Lou home directory rather than your home directory of the NAS compute systems.

Therefore, if your PBS jobs rely on loading specific software modules or environment variables in the system startup files (such as .cshrc), in order to run them on the LDANs you must modify your startup files on Lou to load the modules and set appropriate environment variables.

Running PBS Jobs on LDANs

The PBS server for the LDANs is pbspl1. To use the LDANs, submit your jobs to the ldan queue. Each job can use only one LDAN for up to three days, and each user can have a maximum of two jobs running simultaneously.

Before You Begin: If you want to process archive data that has been migrated offline to tape, use the Data Migration Facility command dmget to migrate the data back to disk before submitting jobs to the LDANs.

Submitting Your PBS Job

You can submit interactive PBS jobs to the LDANs from either the LFEs or the PFEs. You can submit PBS job scripts from either your Lustre filesystem (/nobackup/username) or your Lou home filesystem (/u/username).  

Use the :mem=xxxGB attribute to specify how much memory you need for your job. This will instruct PBS to allocate an appropriate LDAN.

Note: Keep in mind that the amount of memory on a node that is available for your job is slightly less than the total physical memory, because the system kernel can use up to 4 GB of memory in each node. 

Sample PBS Script

#PBS -lselect=1:ncpus=36:mem=750GB
#PBS -lwalltime=2:00:00
#PBS -q ldan

### Remember to load necessary modules
### or set environment variables that you need

module load xxx

### Note that the default directory a PBS job
### is in when it starts is the Lou home filesystem.
### Change directory to the appropriate directory
### for your pre- or post- processing work.

cd $PBS_O_WORKDIR

### If your executable for pre- or post- processing
### is located under your Pleiades home filesystem,
### use the pathname /homeX/username/bin/....
### do not use the pathname /u/username/bin/...

/homeX/username/bin/your_processing_code < input > output

Running Graphics Applications on LDANs

If you want to run graphics applications that consume enough memory to require one of the larger-memory LDANs, submit an interactive PBS job from one of the LFEs that specifies mem=1040GB. For example:

lfe% qsub -I -q ldan -lselect=1:mem=1040GB,walltime=1:00:00

This will provide you with an LDAN for the duration of the specified wall time. Then, from any other terminal window, you can use SSH to connect directly to the LDAN assigned to your job, and run your graphics application interactively. If you have SSH passthrough set up, you can even SSH directly in to the LDAN from your workstation by adding the LDANs as allowed SSH passthrough targets in your .ssh/config file.

To run graphics applications directly from a PBS session submitted from an LFE, you need to export your DISPLAY environment by adding the -X option to the qsub command, as follows:

lfe% qsub -I -X -q ldan -lselect=1:mem=750GB,walltime=1:00:00

If you are a remote user, you can also explore using Virtual Network Computing (VNC) to connect to NAS systems.