A Lustre filesystem is a highly scalable, high-performance shared filesystem for Linux clusters that is managed with Lustre software. It can support many thousands of client nodes, petabytes of storage, and hundreds of gigabytes per second of I/O throughput. The NAS Lustre filesystems are named "/nobackuppX."

Each Lustre filesystem is actually a set of many small filesystems, which are referred to as object storage targets (OSTs). The Lustre software presents the OSTs as a single unified filesystem.

For more information about OSTs and other Lustre filesystem components, see the Main Lustre Components section below.

Useful Lustre Commands

The examples in this section use /nobackupp17 as an example of a specific Lustre filesystem.

Listing Disk Usage and Quotas

To display disk usage and limits on your /nobackup directory:

pfe21% lfs quota -h -u username /nobackupp17

or

pfe21% lfs quota -h -u username /nobackup/username

To display usage on each OST, add the -v option:

pfe21% lfs quota -h -v -u username /nobackup/username

Listing Space Usage

To list space usage per OST and MDT in human-readable format for all Lustre filesystems or for a specific one:

pfe21% lfs df -h
pfe21% lfs df -h /nobackupp17

Listing Inode Usage

To list inode usage for all filesystems or for a specific one:

pfe21% lfs df -i
pfe21% lfs df -i /nobackupp17

Listing OSTs

To list all the OSTs for the filesystem:

pfe21% lfs osts /nobackupp17

Viewing Striping Information

To view the striping information for a specific file or directory:

pfe21% lfs getstripe filename 
pfe21% lfs getstripe -d directory_name

Note: Omitting the -d flag will display striping for all files in the directory.

File Striping

Files on the Lustre filesystems are striped automatically. This means they are transparently divided into chunks that are written or read simultaneously across a set of OSTs within the filesystem. The chunks are distributed among the OSTs using a method that ensures load balancing. Files larger than 100 gigabytes (GB) must be striped in order to avoid taking up too much space on any single OST, which might adversely affect the filesystem.

Striping benefits include:

• Allows one or more clients to read/write different parts of the same file at the same time, providing higher I/O bandwidth to the file because the bandwidth is aggregated over the multiple OSTs.
• Allows file sizes larger than the size of a single OST.

However, striping small files increases the number of objects in each file, resulting in increased overhead due to network operations—sometimes creating server contention. This is not likely to cause problems for files larger than 4 megabytes (MB), but in many cases, striping can cause noticeably slower read/write performance for files smaller than 4 MB.

Progressive File Layout

Lustre's progressive file layout (PFL) feature enables stripe settings for different segments of a file to be configured in such a way that a small stripe count is used for the beginning segment, and larger stripe counts are used for latter segments as the file grows. All of the NAS Lustre filesystems have been configured with NAS-selected PFL settings.

To learn more about PFL, see Progressive File Layout with SSD and HDD Pools.

Main Lustre Components

Lustre filesystem components include:

Metadata Server (MDS)

Service nodes that manage all metadata operations, such as assigning and tracking the names and storage locations of directories and files on the OSTs. There is 1 MDS per filesystem.

Object Storage Target (OST)

Storage devices where users' file data is stored. The size of each OST varies from approximately 7 terabytes (TB) to approximately 22 TB, depending on the Lustre filesystem. The capacity of a Lustre filesystem is the sum of the sizes of all OSTs. There are multiple OSTs per filesystem.

Metadata Target (MDT)

A storage device where the metadata (name, ownership, permissions and file type) are stored. There is 1 MDT per filesystem.

Object Storage Server (OSS)

Service nodes that run the Lustre software stack, provide the actual I/O service and network request handling for the OSTs, and coordinate file locking with the MDS. Each OSS can serve multiple OSTs. The aggregate bandwidth of a Lustre filesystem can approach the sum of bandwidths provided by the OSSes. There can be 1 or multiple OSSes per filesystem.

Lustre Clients

Compute nodes that mount the Lustre filesystem, and access/use data in the filesystem. There are typically thousands of Lustre clients per filesystem.