Known resources impacted:

Granite: already planned to be offline for maintenance

iForge: cluster offline for the duration

Radiant: cluster online, without power redundancy

help@campuscluster.illinois.edu

help@campuscluster.illinois.edu

help@campuscluster.illinois.edu

Known resources impacted:

Granite: already planned to be offline for maintenance

iForge: cluster offline for the duration

Radiant: cluster online, without power redundancy

Emergency PM

We are seeing some network issues on the cluster. In order to resolve these issues, we need to upgrade code on our infiniband infrastructure

UPDATE: We are currently experiencing unforeseen technical issues with the cluster. We are investigating and expect resolution and restoration of all Campus Cluster services by March 3rd 12PM

UPDATE2: We are still experiencing issues where the compute clients will not properly mount storage. We are engaged with vendor support and continue to work on the situation. Thank you for your patience. We have moved expected return to service to March 4th, 12PM

UPDATE3: Campus cluster is experiencing SLURM job failures in certain pods(sections) of the cluster. Investigations continue and there is a partial return to service with login nodes, storage, and data transfer services still operational. New full return of service date: Monday, March 7th, 12PM.

2022-03-02 1237

2022-03-02 1715

Adding Ram
to improve performance

on-line maintenance

on-line maintenance

Had to restart rsyslog and Ldap after relocating /var/log

kimber7@illinois.edu

2022-02-09 1400

The NCSA VMWare cluster is experiencing storage performance issues.

-- Update: Adjustments have been made to storage used by the LDAP servers and other non-essential VM instances have been disabled. Testing is indicating that response times have improved and services are working normally again.

cerberus2.ncsa.illinois.edu, tg-kdc1.security.ncsa.illinois.edu, bwbh2.ncsa.illinois.edu

Info from Slack (https://status.slack.com/)

We've resolved the issue, and all impacted customers should now be able to access Slack. You may need to reload Slack (Cmd/Ctrl + Shift + R) to see the fix on your end. If that doesn't work, try clearing cache (Help > Troubleshooting > Clear Cache and Restart from the app menu). Thanks for bearing with us and we apologize for the disruption to your work day!

Feb 22, 1:24 PM CST

We're seeing signs of improvement. Please try reloading Slack, and if not a cache reset. We’re still monitoring the situation. We’ll confirm once this issue is fully resolved.

Feb 22, 11:07 AM CST

Slack is not loading for some users. We are continuing to investigate the cause and will provide more information as soon as it's available.

Feb 22, 9:23 AM CST

We're still working towards a full resolution. We'll be back with another update soon. Thank you for your patience.

Feb 22, 8:44 AM CST

We’re investigating the issue where Slack is not loading for some users. We’re looking into the cause and will provide more information as soon as it's available.

Feb 22, 8:25 AM CST

2022-02-18
2PM

Reboot to add ram/swap

This is to improve stability

The Lustre filesystem is not loading correctly. The support team has been contacted.

Still in progress. MDT0001 is partially recovered. Vendor is attempting to fully restore.

Near completion: Working with vendor on additional configuration changes. Hope to complete final validation and return to service by close of business 2022-02-18.

2022-02-18
2PM

Reboot to add ram/swap

This is to improve stability

2022-02-14

1PM

2022-02-14

4:15PM

2022-02-09

1000

2022-02-09

1200

LDAP (and dependent services, incl. Jira)

vSphere/ICI VMware

Authorization timeouts/failures in dependent services.

ICI staff are investigating.

LDAP (and dependent services, incl. Jira)

vSphere/ICI VMware

Cause of most severe issues was power fluctuations around 0555, but certain LDAP servers showed degraded slightly earlier.

MySQL database servers need to be synchronized to bring replicated database servers online.

NOTE: The MySQL database is back up, but users may experience issues due to an LDAP issue.

Wiki, JIRA, Savannah/Internal, Identity, and some web sites will stop working. More details are linked here.

22-02-08

3:15PM

Line card failure is affecting devices that are plugged into Neo-hpc-1 aggregation switch.  We've migrated links off the failed card, to other ports on the same switch.

No services are currently impacted.

help@campuscluster.illinois.edu

kimber7@illinois.edu

Kubernetes on NTS is not working properly after updates

LSST Quarterly Maintenance

2021-12-07

0930

2021-12-07

1030