Hard disk drives (HDDs) and solid state drives (SSDs) are the two main storage options commonly deployed in data centers. With HDD, data are stored magnetically in spinning disks, and data retrieval is slow; therefore, HDD is practical only for data that does not need to be accessed frequently. SSDs are a newer technology where data is stored in integrated circuits, which dramatically reduces access time. Because SSDs are more expensive than HDDs, a hybrid solution provides a good compromise between price and performance.
The NAS Lustre filesystems (/nobackupp[17–19, 27–29]) are configured using Progressive File Layout (PFL) with HDDs and SSDs configured as different Lustre pools. This article provides information about these filesystems, using /nobackupp17 as an example.
SSD and HDD Pools
To view a list of a filesystem's object storage targets (OSTs) showing whether they are in the HDD pool or the SSD pool, use the lfs pool_list command. The OSTs are shown in hexadecimal format:
The available SSD space in each filesystem is much smaller than the HDD space. In the example below, the lfs df command shows that the size of each /nobackupp17 HDD OST is 565.0 terabytes (TB), while each SSD OST is only 30.3 TB. The output also shows both the hexadecimal (far left) and decimal (far right) labels of each OST.
The following table summarizes the configurations of /nobackupp[17–19, 27–29].
Filesystem
nbp17
nbp18
nbp19
nbp27
nbp28
nbp29
# of OSTs in HDD
32
(OST: 0-31)
32
(OST: 0-31)
16
(OST: 0-15)
32
(OST: 0-31)
32
(OST: 0-31)
16
(OST: 0-15)
Size/OST in HDD
565.0 T
565.0 T
565.0 T
565.0 T
565.0 T
565.0 T
Total Space in HDD
18.6 P
18.6 P
9.3 P
18.6 P
18.6 P
9.3 P
# of OSTs in SSD
32
(OST: 100-131)
16
(OST: 100-115)
16
(OST: 100-115)
32
(OST: 100-131)
16
(OST: 100-115)
16
(OST: 100-115)
Size/OST in SSD
30.3 T
60.5 T
29.6 T
30.3 T
60.5 T
29.6 T
Total Space in SSD
~ 1 P
~ 1 P
~ 0.5 P
~ 1 P
~ 1 P
~ 0.5 P
Progressive File Layout (PFL)
The Lustre filesystems take advantage of the Lustre PFL feature, which increases the stripe count of each file in a step-wise manner as the file grows in size. Specifically, PFL is built using composite layouts described by a series of components. Each component covers a non-overlapping portion of the file and has its own stripe setting in size, count, index and pool.
Default PFL Configurations
For each filesystem, a NAS-created default layout provides reasonable performance for a variety of file I/O patterns—and relieves users of the need to determine and set striping.
To find the default layout of each filesystem, use the lfs getstripe command as follows:
The output shows that /nobackupp17 is configured with three components:
The first component setting applies to the file segment from the beginning up to 268435456 B (256 MB), with a stripe count of 1 and stripe size of 16 MB; this component uses ssd-pool.
The second component setting applies to the file segment from 256 MB to 5 GB with a stripe count of 16 and stripe size of 16 MB; this component uses ssd-pool.
The last component setting applies to the file segment from 5 GB to the end of the file with a stripe count of 16 and stripe size of 16 MB; this component uses hdd-pool.
With this composite layout, I/O to and from the first 5 GB of files will benefit from the higher performance of SSDs.
Note: The default component layouts may be adjusted by NAS system administrators as needed.
Customized PFL Configurations
If you have experience using the PFL feature, and you want to have more control of your application's I/O, you can choose your own composite layouts using the lfs setstripe command with various options, such as --component-end (or -E), --stripe-size (or -S), --stripe-count (or -c), --overstripe-count (or -C), --pool (or -p). For example:
Note: If not explicitly set, files inherit layouts from their parent directory.
WARNING: SSD space is limited. Therefore, do not set striping for very large files to use only the SSD pool.
Mirroring and Migration of Data from SSD to HDD
As the limited SSD space fills up, data that is stored in SSDs is migrated over to HDDs to free up SSD space for I/O by new jobs. This automatic migration is accomplished through a mirroring mechanism.
After a file is closed, SSD data is queued to be mirrored to the HDD. The data will be in dual state—stored on both the SSD and the HDD—until SSD usage reaches a certain point; then, the SSD copy will be removed to free up space. Typically, the oldest files are removed first when SSD usage reaches 80%, and other files are removed more aggressively when usage reaches 90%. The mirroring and migration settings may be adjusted by NAS system administrators as needed.
Note: Mirrored data that exists in both SSDs and HDDs are counted twice against your quota.
To find out if a file is in dual state or has been migrated completely to HDDs, use the lfs getstripe filename command, as shown in the next two examples.
Output Showing Data in Dual State
If the lfs getstripe filename output shows entries with different values for lcme_mirror_id, the data is in dual state, as illustrated in this example:
Note: When your file is completely migrated to HDD (removed from SDD), the (ctime) of the file will change to the creation time of the mirrored copy on HDD, as described below.
Change Time (ctime) Example
When a file is mirrored, a second copy of the data is created. The file will retain both sets of data until you break the mirror, or when the filesystem automatically flushes the SSD pool—at which time, the (ctime) will be updated to reflect the time the mirror was created.
In the following example, the stat command is used to show the time metadata of a test file that was created at 15:48:45:
