Emergency PM

We are seeing some network issues on the cluster. In order to resolve these issues, we need to upgrade code on our infiniband infrastructure

UPDATE: We are currently experiencing unforeseen technical issues with the cluster. We are investigating and expect resolution and restoration of all Campus Cluster services by March 3rd 12PM

UPDATE2: We are still experiencing issues where the compute clients will not properly mount storage. We are engaged with vendor support and continue to work on the situation. Thank you for your patience. We have moved expected return to service to March 4th, 12PM

UPDATE3: Campus cluster is experiencing SLURM job failures in certain pods(sections) of the cluster. Investigations continue and there is a partial return to service with login nodes, storage, and data transfer services still operational. New full return of service date: Monday, March 7th, 12PM.

2022-03-02

1237

Emergency Work on iforge

Pause of scheduler at this time

We have identified some issues with gpfs and comms to a subset of the nodes. the scheduler is paused and we are working on the issue

GPFs is slow to respond and the Scheduler is paused so no new jobs will. Compute nodes are being rebooted, The login node may also need a reboot