Here is the current status of NAS computer systems.

At about 3:35 a.m. PT on Saturday, June 12, the chilled water pumps shut down, causing the chillers themselves to shut down. Given warning of the failure, NAS Control Room personnel shut down the systems on the computer floor, preventing the room from overheating and allowing the systems to be brought down gracefully. During this time, the condenser water pumps ramped up to attempt to cool the non-functioning chillers. Half of the controllers for those pumps failed catastrophically due to overheating. 

As a result, we currently have two of four chillers running. Until we are able replace the controllers for the two pumps or come up with a workaround, we will be running at reduced capacity with two chillers only.

Given the reduced capacity of the chillers, NAS systems personnel are bringing up only half of the Pleiades nodes, plus the archive system and the Lustre filesystems. The compute nodes have been booted up; and after some initial issues, the filesystem was stabilized. Our systems staff are working to diagnose the issues with the compute nodes and are running some user jobs to test the system. When user jobs are released to start running again, Pleiades will still be at one-half its capacity.

Endeavour and Merope are currently off-line. Once Pleiades is back in service (at half capacity), we will determine whether the facility can support bringing up Endeavour. Merope, which is located in a different building, will be brought up as soon as possible. In addition, our systems personnel are escalating the upgrade of Merope's hardware from Harpertowns to Westmeres and Nehalems.

We sincerely regret the impact this has on your work, and are doing our best to get things back to normal as soon as possible. We will also be working with mission directorate managers to assure that their critical work is prioritized.

Please check back here for daily updates. If you have concerns about job processing, please contact the NAS Control Room at (800) 331-8737, (650) 604-4444, or support@nas.nasa.gov, or contact Cathy Schulbach at 650-604-3180 or catherine.h.schulbach@nasa.gov.