status.ncsa.illinois.edu
Watch this page in the wiki to subscribe to automatic updates to this status page.
Please do not refer to any NCSA Industry Partners on this page. Please use the iforge nomenclature for all of the *forge infrastructure.
To see older events, see Archive of NCSA Status Home
Report a problem
Current Status
| START | END | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|
Upcoming Scheduled Maintenance
Listed below in chronological order.
| Start | End | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
| 2021-04-08 0730 | 2021-04-08 0740 | NCSA Wiki | NCSA's Wiki service will restart | NCSA's Wiki service will restart to apply a new SSL certificate and renewed Confluence license. The wiki will not be available for about 5 minutes while it reloads. | help+service@ncsa.illinois.edu | SCHEDULED |
| 2021-04-08 0900 | 2021-04-08 1000 | WAN Link Migration | NCSA Neteng will be migrating the WAN link to ICCN Node 1 to new hardware. | Traffic will be automatically re-routed to redundant paths during the link outage. | help+neteng@ncsa.illinois.edu | SCHEDULED |
| 2021-04-10 0600 | 2021-04-10 0800 | CILogon hosted COmanage, Grouper, SATOSA, LDAP | On Saturday, April 10, the CILogon team will perform maintenance on the infrastructure used for hosted services. | As part of the maintenance all COmanage Registry, LDAP, Grouper, SAML proxy, SAML attribute authority, and MDQ services hosted by CILogon may experience brief outages. We do not expect that any specific service outage will last for more than a minute. | help@cilogon.org | SCHEDULED |
| 2021-04-12 1800 | 2021-04-13 0700 | File & Print Servers | Monthly Windows File & Print Server Maintenance | Windows File Shares such as HR, Business Office, Home, etc. and printing in the NCSA & NPCF buildings will be unavailable. | help+service@ncsa.illinois.edu | SCHEDULED |
| 2021-04-13 0800 | 2021-04-13 0830 | cilogon.org | Update to OA4MP v5.1.1. | The OAuth2/OIDC backend of the CILogon Service will be updated to OA4MP v5.1.1. | help@cilogon.org | SCHEDULED |
| 2021-04-15 0900 | 2021-04-08 1000 | WAN Link Migration | NCSA Neteng will be migrating the WAN link to ESnet to new hardware. | Traffic will be automatically re-routed to redundant paths during the link outage. | help+neteng@ncsa.illinois.edu | SCHEDULED |
| 2021-06-24 0800 | 2021-06-24 1200 | LSST | LSST Quarterly Maintenance
| All LSST services hosted at NCSA | lsst-admin@ncsa.illinois.edu | SCHEDULED |
| 2021-09-30 0800 | 2021-09-30 1200 | LSST | LSST Quarterly Maintenance
| All LSST services hosted at NCSA | lsst-admin@ncsa.illinois.edu | SCHEDULED |
| 2021-12-09 0800 | 2021-12-09 1200 | LSST | LSST Quarterly Maintenance
| All LSST services hosted at NCSA | lsst-admin@ncsa.illinois.edu | SCHEDULED |
Previous Outages or Maintenance
| Start | End | What System/Service was affected? | What happened? | What was affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
| 1st report 7:30am Monday | 8:19am Monday | NCSA LDAP2 | ldap2 is not responsive to authentication requests | NCSA Jira, any systems using LDAP2 as its only source. | help+service@ncsa.illinois.edu | RESOLVED |
2021-03-30 0800 | 2021-03-30 0845 | DNS1 | A software issue was causing BIND to fail. | DNS was not able to resolve during the period of time. DNS2 remained operational. | neteng+help@ncsa.illinois.edu | RESOLVED |
2021-03-23 2000 | 2021-03-23 2025 | NCSA VPN | The standby VPN hardware was replaced and transitioned into the current VPN cluster. Failover went as expected and firmware was upgraded on the primary after load was shifted to the new standby VPN. | Failover between the appliances occurred without issue and there was no impact to users. | neteng@ncsa.illinois.edu | RESOLVED |
| 2021-03-18 1230 | 1255 | Jira | Some functionality will be limited due to user limit being reached | Jira | help@service@ncsa.illinois.edu | RESOLVED |
| ~16:40 | 17:58 | AnyConnect VPN Service | An issue with SSL on the VPN service has caused an issue that has disconnected all users. Network engineering is looking into the issue. Due to a hardware failure and the VPN not failing over properly to the standby users were unable to connect to the VPN. This was due to an issue with syncing certificates. | During the outage, expect that you won't be able to connect/maintain a connection to the VPN | help+neteng@ncsa.illinois.edu | RESOLVED |
| 2021-03-16 0950 | 2021-03-16 1000 | CMDB | Will be applying updates per security vetting | CMDB, including web interface, will be down briefly during the update. | ncsagroup+org_itsm@ncsa.illinois.edu | RESOLVED |
2021-03-11 | 2021-03-11 | WAN Link Migration | NCSA Neteng migrated the link to ICCN to new hardware. | Traffic was automatically re-routed to redundant paths during the link outage. | help+neteng@ncsa.illinois.edu | RESOLVED |
2021-03-04 | 2021-03-04 | WAN Link Migration | NCSA Neteng migrated the 100G link to MREN to new hardware. | Traffic was automatically re-routed to redundant paths during the link outage. | help+neteng@ncsa.illinois.edu | RESOLVED |
| 2021-03-01 22:11 | 2021-03-01 22:47 | NCSA vSphere | About 40 VMs lost connection to their NFS storage. | Several VM-based services were timing out during the issue, including: vSphere management, a kerberos replica, a ldap replica, httpproxy, license servers, NCSA fileserver, Identity message queuing, monitoring. That triggered some of those VMs to switch to use read-only disk, needing to be rebooted later. | service@ncsa.illinois.edu | RESOLVED |
| 2021-02-25 0800 | 2021-02-25 1200 | LSST | LSST Quarterly Maintenance
| All LSST services hosted at NCSA | lsst-admin@ncsa.illinois.edu | COMPLETE |
2021-02- | 2021-02-25 | WAN Link Migration | NCSA Neteng migrated the 100G link to CARNE to new hardware. | The link to campus through the CARNE router was migrated to new hardware. Traffic was automatically re-routed to redundant paths during each link outage. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2021-2-18 2:30 pm | 2021-2-18 6pm | vsphere.ncsa.illinois.edu | logins were broken due to a cert caching issue on vsphere. | login to the administrative interface is available | service@ncsa.illinois.edu | COMPLETE |
| 2021-02-18 2pm | 2021-02-18 2:30pm | ldap | The certs on several ldap servers were set to expire tomorrow and next week, they were refreshed. | ldap server certs were refreshed prior to their expirations. | help+service@ncsa.illinois.edu | COMPLETE |
| 2021-02-17. 10:00 a.m. | 2021-02-17. 10:30 a.m. | Netdot maint | patching | Netdot may be unavailable during this time. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2021-02-16 0800 | 2021-02-16 1000 | cilogon.org | Update to OA4MP v5.1 had problems. | Several clients reported issues with OA4MP 5.1, so we reverted to OA4MP 4.4.5 at noon. | help@cilogon.org | CANCELED |
| 2021-02-15 1800 | 2021-02-17 00:40 | File & Print Servers | Monthly Windows File & Print Server Maintenance | Windows File Shares such as HR, Business Office, Home, etc. and printing in the NCSA & NPCF buildings will be unavailable. | help+service@ncsa.illinois.edu | COMPLETE |
| 2021-02-11 00:00 | 2021-02-11 04:00 | ICCP - MWT2 | “OmniPoP is doing maintenance for their hardware refresh on February 11 (600 W Chicago) between midnight and 4 a.m.CST. This will mean that the CARNE 100G OmniPoP connection will go down for a time during the Feb 11 window. Most of the traffic that would take this link will reroute to using other links. The only ICCP user that may be impacted is MWT2 because their primary path to UChicago is over this circuit. However, we do have a backup UChicago peering over the CARNE Internet2 100G circuit, so that path will be taken assuming that UChicago's backup path doesn't go through the 6WC OmniPoP switch. The tertiary path to UChicago would be through the ESnet LHCONE peering which goes over the CARNE I2 AL2S 100G. MWT2 shouldn’t need to do anything on their side to prepare for this work.” | All traffic should reroute during this maintenance, but MWT2 may experience brief connectivity issues to UChicago | neteng@ncsa.illinois.edu | COMPLETE |
2021-02-09 07:00 | 2021-02-09 15:35 | iforgeftp1 | public interface is down | S3 connection | iforge-admin@lists.ncsa.illinois.edu | COMPLETE |
2021-02-09 07:00 | 2021-02-09 17:00 | iForge cluster | Quarterly Maintenance | All systems unavailable | iforge-admin@lists.ncsa.illinois.edu | COMPLETE |
| 2021-01-28 10:00am | 2021-01-28 12:00pm | Radiant | Security updates and API endpoint hardening | The web interface to Radiant and the API interfaces will be unavailable during the maintenance period. | radiant-admin@lists.ncsa.illinois.edu | COMPLETE |
2021-01-27 8:00 am | 2021-01-27 8:00 pm | Open Storage Network POD | Updating Ceph to containerized implementation | See Previous Column | bdickin2@illinois.edu | COMPLETE |
| 2021-01-25 | 2021-01-26 | NCSA LDAP | Users with /bin/csh had their shells changed to /bin/bash | Users logging into systems that don't override the /bin/csh data already will find they are using /bin/bash when they login. | help+service@ncsa.illinois.edu | COMPLETE |
| 2021-01-26 13:45 | 2021-01-26 14:05 | administrative interface to vsphere.ncsa.illinosi.edu | administrative interface to vsphere.ncsa.illinosi.edu was upgraded to current patch level | Administrative interfaces to vm's were unavailable for about 20 minutes. | help+service@ncsa.illinois.edu | COMPLETE |
| 2021-01-20 08:00 | 2021-01-20 20:00 | ICCP | ICCP Quarterly Maintenance
| Cluster-wide outage | help@campuscluster.illinois.edu | COMPLETE |
| 2021-01-12 07:00 | 2021-01-12 8:00 | JIRA | JIRA Upgrade to 8.13.2 | All JIRA users | help+its@ncsa.illinois.edu | COMPLETE |
| 2021-01-04 09:14 | 2021-01-04 12:10 | Slack | Slack service issues | All Slack systems | https://status.slack.com/ | OUTAGE |
| 2020-12-21 18:00 | 2020-12-22 0700 | File & Print Servers | Monthly Windows File & Print Server Maintenance | Windows File Shares such as HR, Business Office, Home, etc. and printing in the NCSA & NPCF buildings were unavailable. | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-12-18 | 2020-12-18 | NCSA LDAP | New user accounts will have their shell in ldap set to /bin/bash | New users will have /bin/bash as their default shell | help+service@ncsa.illinois.edu | COMPLETE |
| 2020-12-10 0800 | 2020-12-10 1400 | LSST | LSST Quarterly Maintenance
| All LSST services hosted at NCSA | lsst-admin@ncsa.illinois.edu | COMPLETE |
| 2020-12-09 0800 | 2020-12-10 1000 | HAL | HAL Quarterly Maintenance | HAL cluster | help+isl@ncsa.illinois.edu | COMPLETED |
| 2020-11-24 10:30 | 2020-11-24 17:35 | Blue Waters compute | Boot RAID lost fiber channel connection for reasons not understood | Full system outage | jenos@illinois.edu | COMPLETED |
2020-11-19 11:00 a.m. | 2020-11-19 11:30 a.m. | DNS1 Migration | Neteng will be migrating DNS1 to a new switch. | We need to physical move the cable to DNS1 which will cause momentary outage for dns queries to DNS1. DNS2 will not be affected by the migration. | help+neteng@ncsa.illinois.edu | COMPLETED |
2020-11-18 09:00 | 2020-11-17 17:00 | UPS battery Monitor | Program and configure the new BMS | All UPS connected loads | rantissi@illinois.edu | COMPLETED |
| 2020-11-16 18:00 | 2020-11-16 21:30 | NCSA File & Print Servers | File & Print servers were offline for scheduled maintenance. Windows File Shares and printing were unavailable. | Windows File Shares such as HR, Business Office, Home, etc. and printing in the NCSA & NPCF buildings were unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-10-14 13:45 | 2020-10-14 14:04 | NCSA Wiki & Jira | NCSA's Wiki & Jira servers were restarted. | Wiki & Jira were offline while their servers reboot. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-11-13 12:00 | 2020-11-13 18:00 | Software Directorate VM Farm | Failing power supply of switch will be replaced, will use this to upgrade OS as well. | NCSA OpenSource, INCORE, etc (all machines running on 141.142.277.X). | kooper@illinois.edu | COMPLETED |
| 2020-11-12 9:55 | 2020-11-12 10:15 | cilogon.org | Update to OA4MP v5.0.2 was unsuccessful. | CILogon Service has been reverted to OA4MP 4.4.5. | help@cilogon.org | DELAYED |
| 2020-11-10 07:00 | 2020-11-10 21:05 | iForge cluster | Quarterly Maintenance Switching to GPFS 5 formatted filesystem. | All iForge nodes. | COMPLETE | |
| 2020-11-05 10:00 | 2020-11-05 12:24 | NPCF enterprise UPS | UPS maintenance, replace defective communication cards | Any rack (system) that is UPS power fed | rantissi@illinoise.edu |
|
| 2020-11-04 19:00 | 2020-110-4 21:00 | NCSA Building Router (2 of 2) | Software Updates | Software Updates will be applied to one of the NCSA building routers. Traffic will fall back to the seconds router. No network traffic should be affected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-11-02 08:00 | 2020-11-02 10:20 | NCSA GitLab git.ncsa.illinois.edu | LDAP authentication was disabled for NCSA GitLab. Users of the GitLab web interface are required to authenticate to NCSA through CILogon. | NCSA passwords can no longer access repositories. Use GitLab personal access tokens to authenticate against Git over HTTPS. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-11-02 09:00 | 2020-11-02 09:05 | NCSA Duo | The icon shown in the Duo app for NCSA will be updated to match the icon used in NCSA Slack. | NCSA Duo App pushes will show updated icon to match NCSA Slack. May need to restart phone/app to see updated icon. | help+duo@ncsa.illinois.edu | COMPLETED |
| 2020-10-30 08:00 | 2020-10-30 08:45 | SVN at subversion.ncsa.illinois.edu | Retired SVN Service at subversion.ncsa.illinois.edu | SVN is no longer be available. NCSA users are recommended to use one of our various Git repository options. | help+its@ncsa.illinois.edu | COMPLETED |
2020-10-29 9:00 | 2020-10-29 11:15 | iForge | maintenance to switch to GPFS version 5. | All nodes. | iforge-admin@ncsa.illinois.edu | COMPLETED |
| 2020-10-28 19:00 | 2020-10-28 21:00 | NCSA Building Router (1 of 2) | Software Updates | Software Updates will be applied to one of the NCSA building routers. Traffic will fall back to the seconds router. No network traffic should be affected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-10-27 22:00 | 2020-10-27 22:10 | NCSA VPN | Firmware Updates | Firmware updates were applied to the NCSA VPN. Any AnyConnect VPN sessions were reset during the maintenance and users may need to reconnect. Any IPSEC sessions failed over to the standby unit and were not affected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-10-27 09:00 | 2020-10-27 09:15 |
| The DNS CNAME entries for idp.ncsa.illinios.edu and crl.ncsa.illinois.edu will be changed from cilogon-web.ncsa.illinois.edu to shib-docker.security.ncsa.illinois.edu (141.142.149.33). NCSA Shib IdP v4.0.1 is currently up and running at 141.142.149.33. | help+idp@ncsa.illinois.edu | COMPLETED | |
| 2020-10-22 11am | 2020-10-22 1pm | security.ncsa.illinois.edu and grid.ncsa.illinois.edu | Cert replacement issue | Sites were down | cpitcel | RESOLVED |
| 2020 10-21 0900 | 2020 10-21 1400 | WAN Link Migration | ICCN Engineers will be migrating NCSA's 100G WAN links over to new optical cards. Below is the timetable for the moves: 10:00am CARNE (Node 1) to I2 (710 N Lakeshore Dr) 10:30am NCSA (Node 2) to MREN (710 N Lakeshore Dr) 11:30am CARNE (Node 2) to OmniPop (600 West Chicago) 12:00pm NCSA (Node 1) to I2 (600 West Chicago) 12:15pm NCSA (Node 1) to ESNet (600 West Chicago) | Individual links will be migrated one at a time, each taking roughly 15-20 minutes to complete, leaving redundant paths operational. Traffic will automatically be re-routed to these redundant paths during each link outage. There are exceptions where certain services won't failover in this way. In these cases, individual notifications have been sent out to affected parties. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-10-13 08:00 | 2020-10-13 09:00 | CILogon | Update to OA4MP v5.0 OAuth2/OIDC Libraries encountered issue with Syngenta IdP. Reverted to OA4MP v4.4.5. Will be addressed in future OA4MP update. | https://cilogon.org | help@cilogon.org | DELAYED |
| 2020 10-13 06:00 | 2020 10-13 06:15 | CILogon COmanage Registry at https://registry.cilogon.org | Service stack restart. | COmanage Registry and LDAP directory for the multi-tenant services. | help@cilogon.org | COMPLETED |
| 2020-10-12 18:00 | 2020-10-12 21:30 | NCSA File & Print Servers | Monthly Maintenance for Updates / Backup Checks | File & Print Servers were unavailable. Printing was offline, and fileserver shares were unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-10-01 06:00 | 2020-10-01 06:15 | NCSA VPN | The certificate for sslvpn.ncsa.illinois.edu was updated. | The SSL certificate has been updated. | neteng@ncsa.illinois.edu | COMPLETED |
| 2020-09-28 21:00 | 2020-09-29 16:45 | Nebula | Network card failed in network node, was replaced and network settings reconfigured | All non-virtual networking services for Nebula instances (north/south traffic) | nebula@ncsa.illinois.edu | COMPLETED |
2020-09-23 | 2020-09-24 | LSST | Monthly Maintenance:
| ALL LSST systems | lsst-admin@ncsa.illinois.edu | COMPLETED |
| 2020-09-22 | 2020-09-22 | CILogon multi-tenant COmanage Registry | Upgrade to version 3.3.0 | COmanage Registry service at https://registry.cilogon.org | help@cilogon.org | COMPLETED |
| 2020-09-14 18:00 | 2020-09-15 21:30 | NCSA File & Print Servers | Windows file and print servers were patched and unavailable during maintenance. | Access to Fileserver (Business Office, HR, Home, and Swap shared drives) was unavailable, printing was unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-09-08 08:00 | 2020-09-08 09:00 | CILogon | Update CILogon OIDC Client Admin API | https://cilogon.org | help@cilogon.org | COMPLETED |
| 2020-09-02 0800 | 2020-09-02 1600 | Core-East | Software Upgrades | No user impact expected | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-09-01 | 2020-09-01 | ldaps://ldap.cilogon.org | Restart of LDAP gateway service containers. | All LDAP services operated by CILogon. | help@cilogon.org | COMPLETED |