Dear NAS Users,

Here is some information on the status of the HECC Lustre filesystems.

The problems that we have been having with the filesystems for the past couple of months are related to the removal of older Westmere racks and the installation of new Haswell racks, combined with hardware and software issues. The rack removal and installation changed the network topology and communication patterns, and the increase in compute capability provided by the 29 Haswell racks increased the load on the network and I/O systems. These changes triggered Lustre bugs, and bad hardware in the network exacerbated the problems.

We have taken a number of steps to resolve the problems. These steps include: moving the InfiniBand connections from older QDR switches to newer FDR switches for all the Lustre filesystems; replacing broken switches; and applying a patch from Intel, the vendor that provides our back-end Lustre support, to address an issue that caused the /nobackupp8 filesystem to hang. Since the patch was applied, the issue has not occurred again, and the patch has now been deployed to all of the Lustre filesystems. Finally, we also made tuning changes to the InfiniBand fabric. We believe that these changes have improved the reliability and stability of the filesystems.

To help us resolve any issues quickly, we would appreciate it if you would contact us if you have problems with the filesystems. Please contact the NAS Control Room (650-604-4444, 800-331-8737, or support@nas.nasa.gov), Cathy Schulbach (650-604-3180 or catherine.h.schulbach@nasa.gov), or me.

We sincerely regret the impact these problems have had on your work, and we are continuing to work with our vendors (SGI for systems, Intel for Lustre, and Mellanox for InfiniBand) to improve the filesystems and address any issues you identify.

Sincerely,
William Thigpen
HECC Deputy Project Manager
(650) 604-1061
William.W.Thigpen@nasa.gov