All NAS systems share several filesystems intended to provide working space for compute jobs. These filesystems, called "nobackup" (/nobackuppX), provide more than 90 petabytes (PB) of disk space and serve many thousands of cores. The /nobackup filesystems are Lustre-based filesystems managed under Lustre software version 2.x.

Using the Lustre /nobackup Filesystems

As the name suggests, these "nobackup" filesystems are for temporary use, and are not backed up. Lustre can handle many large files, but you cannot use the Lustre filesystems for long-term storage. If you want to save your files, move them to Lou.

Each Lustre filesystem is shared among many users. To learn how to achieve good I/O performance for your applications and avoid impeding the I/O operations of other users, read the related articles listed at the bottom of the page.

Which Lustre Filesystem is Assigned to Me?

Once you are granted a NAS account, one of the Lustre filesystems will be assigned to you. To find out which one, run the ls command as follows:

% ls -l /nobackup/your_username

In the output, look for the filesystem name (nobackuppX). For example, the following output shows that the user's assigned filesystem is /nobackupp17:

lrwxrwxrwx 1 root root 19 Sep 23  2021 /nobackup/your_username -> /nobackupp17/your_username

Default Quota and Policy on /nobackup Filesystems

Disk space and inode quotas are enforced on the /nobackup filesystems. The default soft and hard quota limits for inodes are 500,000 and 600,000, respectively. Quotas for the disk space are 1 terabyte and 2 terabytes, respectively. To check your disk space, inode usage, and quota on your /nobackup filesystem, run the lfs command as follows:

% lfs quota -h -u username /nobackup/username
Disk quotas for user username (uid nnnn):
        Filesystem      used       quota   limit   grace   files    quota   limit   grace
/nobackup/username        4k      1.024T  2.048T       -       1   500000  600000       -

It is expected that your data will occasionally exceed the soft limit. However, after a 14-day grace period, your access to the batch queues will be restricted and you will not be able to run new jobs until your data is reduced below the soft limit. If you reach the hard limit, your batch access will immediately be restricted, but any running jobs will continue.

Once you exceed your filesystem limits, PBS will return the following error and prevent you from submitting new jobs:

qsub: Filesystem quota exceeded! Please reduce your usage or request an increase before submitting.

If you anticipate a long-term need for higher quota limits, please send a justification via email to support@nas.nasa.gov. Your request will be reviewed by HECC management for approval.

For more information, see Quota Policy on Disk Space and Files.

Note: When a Lustre filesystem is full, the jobs writing to it will hang. A Lustre error with code -28 in the system log file indicates that the filesystem is full. NAS support staff will typically send emails to those using the most space, with a request to clean up their files.

Lustre Filesystem Configuration: Progressive File Layout

Lustre's Progressive File Layout (PFL) feature enables filesystems to dynamically change the stripe count for files based on their size. This means that the files are dynamically striped, therefore, you should not set custom stripe size or stripe counts.

The Lustre filesystems are called /nobackupp[17–19, 27–29]. They are equipped with a small number of solid state drives (SSDs) in addition to the hard disk drives (HDDs).

The HDD configurations are shown in the following table with the abbreviation nbpX. In the table, P = petabytes; T = terabytes.

Different PFL configurations are set among the SSD and HDD pools. To learn more, see Progressive File Layout with SSD and HDD Pools.