Note: This workaround should no longer be used. Please see information about the firmware fix in the InfiniBand QP Errors article.

Over the past several months, InfiniBand-related queue pair (QP) errors have caused some jobs on Pleiades to fail, and we have been working to find the cause and fix the problem. Recently, a dramatic increase in QP errors has made resolving this problem a top priority.

Working with SGI and Mellanox, we have now identified code that reproduces the problem. This will enable us to develop a permanent fix. While we do not yet have a solution, we now have a workaround.

To avoid the QP problem, set the MPI_USE_UD environment variable as follows:

• csh and tcsh users: setenv MPI_USE_UD true
• sh, bash, and ksh users: export MPI_USE_UD=true

Please note that this setting may cause some performance degradation. In our testing, the degradation was minimal, but your code may be affected differently.

After using this workaround, please let us know if you continue to get QP errors, have any questions, or experience other problems. Contact the NAS Control Room at (800) 331-8737, (650) 604-4444, or support@nas.nasa.gov.