status.ncsa.illinois.edu
Watch this page in the wiki to subscribe to automatic updates to this status page.
Please do not refer to any NCSA Industry Partners on this page. Please use the iforge nomenclature for all of the *forge infrastructure.
To see older events, see Archive of NCSA Status Home
Report a problem
Current Status
| START | END | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
Upcoming Scheduled Maintenance?
| Start | End | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
| 2020 10-13 06:00 | 2020 10-13 06:15 | CILogon COmanage Registry at https://registry.cilogon.org | Service stack restart. | COmanage Registry and LDAP directory for the multi-tenant services. | help@cilogon.org | SCHEDULED |
| 2020 10-21 0900 | 2020 10-21 1400 | WAN Link Migration | ICCN Engineers will be migrating NCSA's 100G WAN links over to new optical cards. | Individual links will be migrated one at a time, leaving redundant paths operational which traffic will be re-routed. | help+neteng@ncsa.illinois.edu | SCHEDULED |
| 2020-10-30 08:00 | 2020-10-30 10:00 | SVN at subversion.ncsa.illinois.edu | Retiring SVN Service at subversion.ncsa.illinois.edu | SVN will no longer be available. NCSA users are recommended to use one of our various Git repository options. | help+its@ncsa.illinois.edu | SCHEDULED |
| 2020-11-02 08:00 | 2020-11-02 10:00 | NCSA GitLab git.ncsa.illinois.edu | LDAP authentication will be disabled for NCSA GitLab. Users of the GitLab web interface will be required to authenticate to NCSA through CILogon. | NCSA user passwords will no longer be able to access repositories via HTTPS. Instead users can also use GitLab personal access tokens to authenticate against Git over HTTPS. | help+its@ncsa.illinois.edu | SCHEDULED |
Previous Outages or Maintenance?
| Start | End | What System/Service was affected? | What happened? | What was affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
| 2020-10-01 06:00 | 2020-10-01 06:15 | NCSA VPN | The certificate for sslvpn.ncsa.illinois.edu was updated. | The SSL certificate has been updated. | neteng@ncsa.illinois.edu | COMPLETED |
| 2020-09-28 21:00 | 2020-09-29 16:45 | Nebula | Network card failed in network node, was replaced and network settings reconfigured | All non-virtual networking services for Nebula instances (north/south traffic) | nebula@ncsa.illinois.edu | COMPLETED |
2020-09-23 | 2020-09-24 | LSST | Monthly Maintenance:
| ALL LSST systems | lsst-admin@ncsa.illinois.edu | COMPLETED |
| 2020-09-22 | 2020-09-22 | CILogon multi-tenant COmanage Registry | Upgrade to version 3.3.0 | COmanage Registry service at https://registry.cilogon.org | help@cilogon.org | COMPLETED |
| 2020-09-14 18:00 | 2020-09-15 21:30 | NCSA File & Print Servers | Windows file and print servers were patched and unavailable during maintenance. | Access to Fileserver (Business Office, HR, Home, and Swap shared drives) was unavailable, printing was unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-09-08 08:00 | 2020-09-08 09:00 | CILogon | Update CILogon OIDC Client Admin API | https://cilogon.org | help@cilogon.org | COMPLETED |
| 2020-09-02 0800 | 2020-09-02 1600 | Core-East | Software Upgrades | No user impact expected | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-09-01 | 2020-09-01 | ldaps://ldap.cilogon.org | Restart of LDAP gateway service containers. | All LDAP services operated by CILogon. | help@cilogon.org | COMPLETED |
2020-08-28 1425 | 2020-08-28 1725 | NCSA Email from Campus Exchange | Campus Exchange could not deliver email addressed to "@ncsa.illinois.edu" addresses. A user requested a change to Exchange that broke delivery to NCSA. Any email you sent to "@ncsa.illinois.edu" addresses from campus Exchange will need to be resent by you. | Campus Exchange could not deliver email addressed to "@ncsa.illinois.edu" addresses. The change in Exchange was reverted around 1655, but it may take a bit of time for Office365 to update all of its servers. | help+its@ncsa.illinois.edu | RESOLVED |
| 2020-08-28 0800 | 2020-08-28 1000 | NPCF Networking DC Power System | Testing and maintenance of the DC power system and battery backup was performed. | No outage. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-08-26 0800 | 2020-08-26 1200 | Core-West | Replacing failed internal board. Software upgrades | No user impact expected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-08-17 0800 | 2020-08-22 19:00 | Code42 Crashplan Endpoint | One of the primary Code42 Crashplan Endpoints crashed and was not accepting backup requests | Code42 Crashplan clients that were assigned to backup to this server were unable to run backups. | crashplan@ncsa.illinois.edu | RESOLVED |
2020-08-18 06:00 | 2020-08-08 06:20 | NCSA VPN Service | Software upgrade was completed successfully. | All IPSEC sessions failed over successfully to the standby. Any users connected to the AnyConnect VPN were briefly disconnected and need to reconnect. An upgrade of the AnyConnect client was included with this upgrade and users will receive the upgrade when they reconnect to the VPN. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-08-18 06:00 | 2020-08-18 06:15 | registry.cilogon.org | CILogonPerson LDAP schema plugin was removed. | The COmanage Registry service. | help@cilogon.org | COMPLETED |
| 2020-08-11 07:00 | 2020-08-11 18:30 | iForge | Quarterly Maintenance | All systems will be unavailable during the maintenance | iforge-admin@ncsa.illinois.edu | COMPLETED |
| 2020-08-04 08:00 | 2020-08-04 08:30 | CILogon | https://go.ncsa.illinois.edu/CILogonServiceUpdate2020-08-04 | Remove <whitelisted> tag from idplist.xml file. Add "?initialidp=..." query parameter. | help@cilogon.org | COMPLETED |
| 2020-08-03 18:00 | 2020-08-03 23:00 | Fileserver & Printing | Monthly Windows Server Maintenance | NCSA Fileserver(s) and NCSA-Print were unavailable during the maintenance. Business Office, HR, home, swap, etc. shares were unavailable. Printing was unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-07-28 06:00 | 2020-07-28 06:05 | registry.cilogon.org | OA4MP plugin for creating/managing OIDC clients was updated | The COmanage Registry service was unavailable during the outage. | skoranda@illinois.edu | COMPLETED |
| 2020-07-24 0700 | 2020-07-24 0707 | Applied some minor configuration adjustments | Pages were unavailable for 5 minutes as the service restarted. | COMPLETED | ||
| 2020-07-22 0800 | 2020-07-22 0823 | git.ncsa.illinois.edu | GitLab was upgraded to the latest version | The git service was intermittently unavailable during this upgrade window. | COMPLETED | |
| 2020-07-18 0500 | 2020-07-18 0600 | NCSA WAN connectivity over Internet2 and ESnet links. | Emergency network maintenance performed by ICCN (our WAN provider) | Instability on WAN connectivity to Industry partners and research institutions using internet2 service provider network to connect to NCSA. General network connectivity will re-route accordingly. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-07-16 0700 | 2020-07-16 0800 | Core-West | A service alarm on Core-West switch requires a reset of an internal controller board. | Traffic will be failed over to Core-East while performing the maint. No impact expected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-07-15 1400 | 2020-07-15 1730 | Storage Condo | Perc Raid card failure in core server | Condo Services/GridFTP/NFS | ckerner@illinois.edu | COMPLETED |
| 2020-07-07 18:00 | 2020-07-07 21:30 | Fileserver & Printing | Monthly Windows Server Maintenance | NCSA Fileserver(s) and NCSA-Print were unavailable during the maintenance. Business Office, HR, home, swap, etc. shares were unavailable. Printing was unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-07-07 0800 | 2020-07-07 0900 | CILogon | Update Admin Client API | https://cilogon.org . No downtime is expected. | help@cilogon.org | COMPLETED |
2020-06-24 | 2020-06-25 Kubernetes updates completed at 1330 | LSST | Monthly Maintenance:
| ALL LSST systems | lsst-admin@ncsa.illinois.edu | COMPLETE |
| 2020-06-17 0600 | 2020-06-17 1600 | Blue Waters scratch filesystem | Disk failure during OST failover, both OSTs unavailable 14 drives offline, reassembly required | Some filesystem operations. | COMPLETED | |
| 2020-06-15 08:00 | 2020-06-15 11:50 | Software Directorate VM Farm
| Upgrade of servers, including vm servers | During this time all servers will be down, servers will be returned ASAP | COMPLETED | |
| 2020-06-10 09:00 | 2020-06-12 21:00 | hal.ncsa.illinois.edu | Quarterly PM | HAL System Service | dmu@illinois.edu | COMPLETE |
| 2020-06-08 18:00 | 2020-06-09 07:00 | Fileserver & Printing | Monthly Windows Server Maintenance | NCSA Fileserver(s) and NCSA-Print will be unavailable during the maintenance. Business Office, HR, home, swap, etc. shares will be unavailable. Printing will be unavailable. | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-06-04 09:30 | 2020-06-04 14:46 | vsphere.ncsa.illinois.edu | Updated vCenter SSL certificate and trust chain | Management of VM's was unavailable while updating SSL certificates | help+its@ncsa.illinois.edu | RESOLVED |
| 2020-05-30 08:00 | 2020-06-01 12:00 | linux.ncsa.illinois.edu public-linux.ncsa.illinois.edu | SSH password-based authentication were failing due to changes with intermediate certificates | SSH password-based authentication broke. Kerberos based auth continued to work. | RESOLVED | |
| 2020-5-28 07:00 | 2020-5-29 09:00 | NCSA Virtual Classroom | Nodes were added, network reconfigured, and updates were applied. | Student VM's were be unavailable | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-5-22 07:57 | 2020-5-22 14:00 | Blue Waters Compute | Mistaken cabinet removed from configuration causing unroutable configuration for HSN | All compute, all running jobs | COMPLETE | |
| 2020-05-20 10:00 | 2020-05-20 14:00 | DNS1/2 | Upgrades | DNS servers will be rebooted during this time. | help+neteng@ncsa.illinois.edu | COMPLETE |
2020-5-14 06:00 | 2020-5-14 08:00 | NCSA Wiki | Upgrade | Wiki pages will be unavailable | swrights@illinois.edu | COMPLETE |
| 2020-05-19 09:30 | 2020-05-19 10:40 | netact.ncsa.illinois.edu | Fixing a problem with apache | netact is down. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2020-05-12 07:00 | 2020-05-12 17:00 | iForge | Quarterly Maintenance | All systems will be unavailable during the maintenance | iforge-admin@ncsa.illinois.edu | COMPLETE |
| 2020-05-04 18:00 | 2020-05-04 22:30 | NCSA Fileservers & Print Servers | Monthly ITS Windows Server Maintenance | Fileserver Shares (HR, Business Office, Home, Swap, etc.) and shared printers on NCSA-Print | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-04-30 9:00 | 2020-04-30 11:00 | Systems connection to idds-prod | ITS will be updating firewall settings for idds-prod. | No impact is expected, but users should contact help+idds if issues occur. | help+idds@ncsa.illinois.edu | COMPLETE |
| 2020-04-22 8:00 | 2020-04-22 9:00 | CILogon | https://go.ncsa.illinois.edu/CILogonServiceUpdate2020-04-22 | CILogon will relax name and email attribute requirement for IdPs. | help@cilogon.org | COMPLETE |
| 2020-04-21 6:00 | 2020-04-21 6:05 | CILogon | AWS CILogon COmanage update | HTTP (80/443) and LDAP (389/636) ports will be unavailable | help@cilogon.org | COMPLETE |
| 2020-04-15 9:25 | 2020-04-15 10:01 | Campus Cluster user portal | Login access via UIUC Shibboleth was not working, while Shibboleth configurations were updated in support of new Shibboleth version | Login access via UIUC Shibboleth was not working. | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-04-14 11:08 | 2020-04-14 11:38 | vsphere.ncsa.illinois.edu | Vcenter was upgraded. | Management of VM's was unavailable for 30 minutes. | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-04-14 10:00 | 2020-04-14 12:00 | CILogon | https://go.ncsa.illinois.edu/CILogonServiceUpdate2020-04-14DELAYED | An incompatibility in the OIDC "getcert" endpoint was discovered. The update has been delayed. | help@cilogon.org | COMPLETE |
| 2020-04-13 13:55 | RSA Authentication Manager and SecurID | RSA Authentication Manager service was turned off. | Authentication using NCSA RSA tokens is no longer supported. If you are using RSA with other organizations it should continue to work. | otp@ncsa.illinois.edu | COMPLETE | |
2020-04-13 0900 | 2020-04-13 00930 | All Globus Services currently using RSA authentication | RSA authentication will be changed to DUO authentication | All Globus Services currently using RSA authentication | help+globus@ncsa.illinois.edu | COMPLETE |
| 2020-04-07 10:00 | 2020-04-07 13:00 | Cerberus Bastions, BWBH Bastions | These systems will be migrated from using RSA to DUO for their second factor. | SSH logins on the hosts: cerberus{1,2}..ncsa.illinois.edu bwbh{1,2}.ncsa.illinois.edu | help+security@ncsa.illinois.edu | COMPLETE |
| 2020-04-01 18:00 | 2020-04-01 22:00 | Windows Server Maintenance | NCSA Windows File & Print Servers were unavailable. Users were not be able to access data on fileserver, or print to any printers in the building while maintenance was completed. | NCSA File & Print Servers | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-04-01 9:02 | RSA SecurID self-service portal, https://otp.ncsa.illinois.edu/ | Portal was turned off. | If you need to change your PIN on activate a new phone you won't be able to. | otp@ncsa.illinois.edu | COMPLETE |