status.ncsa.illinois.edu
Watch this page in the wiki to subscribe to automatic updates to this status page.
Please do not refer to any NCSA Industry Partners on this page. Please use the iforge nomenclature for all of the *forge infrastructure.
To see older events, see Archive of NCSA Status Home
Report a problem
Current Status
| Start | End | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
Upcoming Scheduled Maintenance
Listed below in chronological order.
| Start | End | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
Previous Outages or Maintenance
| Start | End | What System/Service was affected? | What happened? | What was affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
0700 | 2115 | HTC Condor | Cluster Relocation | All HTC Condor services unavailable | tbouvet@illinois.edu | COMPLETE |
0700 | 1915 | vForge | Quarterly Planned Maintenance | All vForge services unavailable. | help@ncsa.illinois.edu | COMPLETE |
0700 | 2135 | Illinois Campus Cluster Program (ICCP) | Quarterly Preventative Maintenance | All ICCP services unavailable. | help@campuscluster.illinois.edu | COMPLETE |
0700 | 1930 | ICRN | Planned Maintenance | All ICRN services (Jupyter, RStudio, and VS Code) unavailable | help@ncsa.illinois.edu | COMPLETE |
0800 | 1825 | Taiga | Semi-Annual Preventative Maintenance | All Taiga services will be completely unavailable for the duration of outage. FS will not be accessible via any endpoints | COMPLETE | |
0800 | 2000 | Harbor | Firmware and os update | All services will be available. The update is an online procedure. | COMPLETE | |
0800 | 1200 | Granite | Major Granite Library Upgrades | Access to the Granite archive | set@ncsa.illinois.edu | COMPLETE |
| 17:00 | vsphere Access and some VMs offline/inaccessible | Multiple datastores were down following power issues to NCSA Vsphere was offline. ~noon vsphere.ncsa.illlinois.edu was back up By 5:00pm all VMs were back online or in a state where they could be powered on sucessfully | Access to vsphere.ncsa.illlinois.edu Outages for multiple VMs (majority of which were test/dev) | help@ncsa.illinois.edu | COMPLETE |
| 2:10 PM | TGI RAILS | Not pingable/accessible admin jumphost not accepting logins to investigate RAILS network state vsphere not accessible to investigate jumphost state | RAILS was inaccessible. RAILS scheduler VM crashed RAILS logins required restart Some compute slurmd needed restart | jenos@illinois.edu | COMPLETE |
9:00am | 11:00am | NCSA 3003: | Correcting issues due to sudden power loss over weekend. Switch area59-mgmt primary partition is corrupted. Rebuilding at this time. | ISL & AREA59 networks. | neteng@ncsa.illinois.edu | COMPLETE |
0900 | 0930 | NPCF Wireless | Tech Services are swapping out the building router at NPCF. | Wireless and VoIP connectivity will be briefly interrupted while they move a couple of cables to the new switch. Should be pretty brief. | neteng@ncsa.illinois.edu | COMPLETE |
0600 | 0900 | wiki.ncsa.illinois.edu | Upgrade confluence | wiki will be unavailable while the updates are installed | pttran3 | COMPLETE |
0700 | 0800 | Final migration step to route all mail through campus servers | No impact is expected. | COMPLETE | ||
2025-06-18 0700 | Delta: 2025-06-19 1300 DeltaAI: 2025-06-18 22:23 | Delta/DeltaAI | system maintenance | all services will be unavailable note: the maintenance window on Delta was extended significantly due to issues on the Delta fabric that emerged due to changes made during the maintenance window
| help@ncsa.illinois.edu | COMPLETE |
0700 | 0900 | Jira | Upgrade | Jira will be down | cpitcel@illinois.edu | COMPLETE |
| 2025-05-14 1700 | 2025-05-15 0700 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares will be unavailable during maintenance. Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable. | help@ncsa.illinois.edu | COMPLETE |
| 2025-06-13 0900 | 2025-06-13 1100 | Facility Main UPS | Replace one battery | All systems on UPS power. The UPS will stay online at reduced capcity supporting load while the battery being replace | MO Rantissi | COMPLETE |
2025-06-10 0930 | 2025-06-12 1030 | CA FW | Planned Maintenance | Will apply patch on one firewall Tuesday, and the other on Thursday. During the update, traffic will be failed over to the other box. No interruptions expected | help+security@ncsa.illinois.edu | COMPLETE |
| 2025-06-03 0600 | 2025-06-03 0627 | NCSA GitLab git.ncsa.illinois.edu | GitLab was updated to latest version. | All GitLab services were unavailable for a few minutes. | help@ncsa.illinois.edu | COMPLETE |
2025-05-27 0930 | 2025-05-29 1500 | NCSA CWM | Planned Maintenance -Got unintentionally delayed due to troubleshooting a different issue | Will apply patch on one firewall Tuesday, and the other on Thursday. During the update, traffic will be failed over to the other box. No interruptions expected | help+security@ncsa.illinois.edu | COMPLETE |
2025-05-27 0930 | 2025-05-29 1500 | NPCF CWM | Planned Maintenance | Will apply patch on one firewall Tuesday, and the other on Thursday. During the update, traffic will be failed over to the other box. No interruptions expected | help+security@ncsa.illinois.edu | COMPLETE |
2025-05-27 0930 | 2025-05-29 1500 | ACHE FW | Planned Maintenance | Will apply patch on one firewall Tuesday, and the other on Thursday. During the update, traffic will be failed over to the other box. No interruptions expected | help+security@ncsa.illinois.edu | COMPLETE |
2025-05-22 0900 | 2025-05-22 1000 | Cerberus Bastions | Planned Maintenance | Cerberus bastion servers will be unavailable during a quick reboot. | help+security@ncsa.illinois.edu | COMPLETE |
2025-05-22 0900 | 2025-05-22 1000 | ACHE Bastions | Planned Maintenance | ACHE bastion servers will be unavailable during a quick reboot. | help+security@ncsa.illinois.edu | COMPLETE |
2025-05-22 0900 | 2025-05-22 1000 | Nightingale Bastions | Planned Maintenance | Nightingale bastion servers will be unavailable during a quick reboot. | help+security@ncsa.illinois.edu | COMPLETE |
| LastPass | Lastpass URL Encryption | Enrolled users may be prompted for their LastPass password to begin the URL Encryption process. This enhances the security of the LastPass database. However, during the encryption process, existing entries will not be editable. | help+security@ncsa.illinois.edu | COMPLETE | |
2025-05-21 0700 | 2025-05-21 1829 | Nightingale | Planned Maintenance | All Nightingale servers and services were unavailable | help@ncsa.illinois.edu | COMPLETE |
2025-05-15 1000 | 2025-05-15 1817 | HAL | Lustre filesystem upgrade | The Lustre filesystem will be upgraded from 6.3.0-1 to 6.3.3. This will be an online upgrade. Users may experience brief periods of slowdown while the services fail over and back during the upgrade. | Peter Hartman pehartma@illinois.edu | RESOLVED |
2025-05-14 1700 | 2025-05-15 0700 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares will be unavailable during maintenance. Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable. | help@ncsa.illinois.edu | RESOLVED |
2025-04-28 12:00PM | 2025-04-28 12:35PM | ICRN | Service unavailable Hub pod in crash loop, out of disk where the proxy runs | All ICRN apps | help@ncsa.illinois.edu | RESOLVED |
| | Jira / Help Desk | Scheduled maintenance | Jira and Help Desk will be unavailable | aloftus@illinois.edu | COMPLETE |
1000 | 1700 | ICRN | Rolling upgrades | 24 hour runtime limit will be enforced for rolling upgrades. Please refresh the web page if you experience interruptions. | help@ncsa.illinois.edu | COMPLETE |
1000 | 1400 | DNS2 | DNS2 needs to have some maintenance performed. | DNS2 will be unavailable while system updates are being performed. | help+neteng@ncsa.illinois.edu | COMPLETE |
2025-04-09 1000 | 2025-04-09 1101 | Harbor | Upgrade of the appliance to the latest release levels. | All harbor c-nodes and d-nodes will be upgraded in a rolling process. No downtime or service reduction is expected. | ckerner@illinois.edu | COMPLETE |
2025-04-02 1700 | 2025-04-03 0700 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares will be unavailable during maintenance. Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable. | help@ncsa.illinois.edu | COMPLETE |
0700 | 0800 | wiki.ncsa.illinois.edu | install software updates | wiki will be unavailable while the updates are installed | aloftus | COMPLETE |
| 2025-03-18 8:00 | 2025-03-18 08:10 | idp.ncsa.illinois.edu | The NCSA IdP was updated with the latest verion of Tomcat to address a vulnerability which was being exploited in the wild. | idp.ncsa.illinois.edu | help+idp@ncsa.illinois.edu | COMPLETE |
| 2025-03-17 10:00 | 2025-03-17 17:00 | All NPCF systems | Core networks power supply back up batteries will be disconnected from the core network power source for batteries replacement. | System operation should not be affected unless there is facility power disturbance taking out power to the network. | rantissi@illinois.edu | COMPLETE |
2025-03-13 08:00 | 2025-03-13 13:30 | Taiga/Granite | Online Upgrade of Taiga; and offline upgrade of Granite. | Access to Taiga encountered 5 ~5 minute failovers during its online upgrades which completed successfully. Granite was unavailable during this PM window for maintenance and upgrades from ScoutFS v2 to v3. | set@ncsa.illinois.edu | COMPLETE |
2025-03-10 10:00 | 2025-03-10 11:00 | LDAP Service ldap1, ldap2 | Update SSL certificates (5 minutes after service restart) | I will restart LDAP on each server to fully deploy cert. | tbouvet@illinois.edu bsper2@illinois.edu | COMPLETE |
| 2025-03-06 17:00 | 2025-03-06 17:15 | NCSA VPN | Cert is being updated for 2025 | Connections may be unavailable or disconnected. | neteng@ncsa.illinois.edu | COMPLETE |
| 2025-03-05 08:00 | 2025-03-05 10:30 | Facility infrastructure power | Facility power automatic transfer switches preventative maintenance. | Power flicker to facility office space and data center CRAHs switching to alternate power source. | rantissi@illinois.edu | COMPLETE |
| 2025-02-25 1545 | 2025-02-25 1600 | ldap service provided by ldap2.delta.ncsa.illinois.edu | stop ldap service and shutdown ldap2.delta, Increase memory to 8GB, increased cpus to 4, power up. Clients reconnecting | Any system that has sssd or ldap services pointing at ldap2.delta. Correctly configured clients will switch to secondary server in their list for the ldap service. | tbouvet@illinois.edu | COMPLETE |
| 2025-02-25 0930 | 2025-02-25 0945 | ldap service provided by ldap1. | stop ldap service and shutdown ldap1. Increase memory to 8GB, power up. Less than 10 minute outage suspected. | Any system that has sssd or ldap services pointing at ldap1.ncsa.illinois.edu. Correctly configured clients will switch to second server in their list for the ldap service. | tbouvet@illinois.edu | COMPLETE |
| 2025-02-18 0800 | 2025-02-18 0900 | idp.ncsa.illinois.edu | The servers hosting idp.ncsa.illinois.edu are moving to the 141.142.148.x subnet. The DNS entry for idp.ncsa.illinois.edu will be updated to point to the new servers. | idp.ncsa.illinois.edu | help+idp@ncsa.illinois.edu | COMPLETE |
1600 |
| Some IRST-hosted VMs | an IRST-run ESXi server is experiencing some IO issues, potentially causing issues for VMs on that server | some IRST-hosted VMs | help+security@ncsa.illinois.edu | RESOLVED |
2025-02-17 1700 | 2025-02-18 0700 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares will be unavailable during maintenance. Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable. | help@ncsa.illinois.edu | RESOLVED |
2025-02-17 1220 | 2025-02-17 1240 | ldap4 - LDAP service | Syetem short on memory 4GB almost used up. Took system down and increased memory to 8GB | System had rebooted daily because or out of memory condition. | tbouvet, Clausen | RESOLVED |
2025-02-14 1300 | 2025-02-14 1700 | ACHE SMTP Relay | This system will be down to be reinstalled to RHEL8 | Mail from ACHE systems will be queued up locally until the system is back online. | help+security@ncsa.illinois.edu | SCHEDULED |
12:45 | 19:45 | Duo device management | Multiple institutions reporting some issues with Duo administration and device management. | Duo Device Management errors may lead users to lock themselves out from Duo if multiple failed attempts to manage devices are recorded | help+security@ncsa.illinois.edu | RESOLVED |
31 Jan 2025 | 31 January 2025 | LastPass | unused accounts are disabled or deleted per policy | unused LastPass accounts | help+security@ncsa.illinois.edu | COMPLETE |
| 2025-01-27 0900 | 2025-01-27 0930 | idp.ncsa.illinois.edu | Update to latest Tomcat version | idp.ncsa.illinois.edu | help+idp@ncsa.illinois.edu | COMPLETE |
0005 | 0702 | NCSA DHCP | NCSA DHCP service was automatically restarted with some missing data. | NCSAnet wireless and office Ethernet connected computers were unable to get a IP address assigned. | help@ncsa.illinois.edu | COMPLETE |
0700 | 1945 | Radiant | Radiant Quarterly Maintenance | All Radiant services | help@ncsa.illinois.edu | COMPLETE |
2025-01-17 0900 | 2025-01-17 1700 | HAL HAL-DGX ISL-x86 Cluster | Planned Maintenance | All services including login, compute and Open Ondemand | help@ncsa.illinois.edu | COMPLETE |
2025-01-15 0700 | 2025-01-16 1314 | Nightingale | Quarterly Planned Maintenance | All Nightingale servers and services were unavailable | help@ncsa.illinois.edu | COMPLETE |
2025-01-13 0900 | 2025-01-16 0940 | ca-firewalls | Updating ca Firewalls fw1 Monday fw2 Thursday | No outages expected | security@ncsa.illinois.edu | COMPLETED |
2025-01-13 0900 | 2025-01-16 0940 | ache-firewalls | Updating ache Firewalls fw1 Monday fw2 Thursday | No outages expected | security@ncsa.illinois.edu | COMPLETED |
| 2025-01-01 2359 | 2025-01-15 XXXX | Campus Cluster | Campus Cluster is moving from ACB to NPCF | All Campus Cluster resources will be inaccessible during this period. | COMPLETED | |
2025-01-09 1700 | 2025-01-10 0700 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares will be unavailable during maintenance. Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable. | help@ncsa.illinois.edu | COMPLETED |
2025-01-07 0900 | 2025-01-09 0945 | NCSA-CWM firewalls | Updating NCSA-CWM Firewalls fw1 Tuesday fw2 Thursday | No outages expected | security@ncsa.illinois.edu | COMPLETED |
2025-01-07 0900 | 2025-01-09 0945 | NPCF-CWM firewalls | Updating NPCF-CWM Firewalls fw1 Tuesday fw2 Thursday | No outages expected | security@ncsa.illinois.edu | COMPLETED |
| 2025-01-02 1000 | 2025-01-02 1100 | idp.ncsa.illinois.edu | Update to latest Tomcat version | idp.ncsa.illinois.edu | help+idp@ncsa.illinois.edu | COMPLETED |
Legend:
IN PROGRESS
COMPLETE
RESOLVED
SCHEDULED
MONITORING