In 2014, the NAS Systems group worked with Mellanox, Intel, and SGI engineers to identify the sources of packet loss occurring on the InfiniBand network. We found and resolved problems in several areas including cable layout, system BIOS, switch and HCA firmware, subnet management, Lustre, and MPT libraries. All of the fixes are transparent to users, with one exception: users must change their runtime MPT library to version 2.11 (or later). No code recompilation is necessary.
MPT versions prior to 2.11 use a simplified packet retry mechanism that can cause excessive packet loss on the network under certain workloads. When this occurs, it can also affect other jobs on the system, causing them to slow down or fail. MPT version 2.11 includes several fixes that correct the problem, including significantly improving packet flow control and the retry mechanism. The fixes will not be backported to pre-2.11 versions, as this would involve too many dependencies and leave out other important updates.
In order to help maintain overall system stability, the following MPT modules on Pleiades were removed from the system on Monday, January 5, 2015:
If you have a critical need to access one or more of these modules, please contact the NAS Control Room at support@nas.nasa.gov, (800) 331-8737, or (650) 604-4444. We can provide access to retired modules under exceptional circumstances. We can also help you ensure that your applications can use a supported version of the library.