status.ncsa.illinois.edu
Watch this page in the wiki to subscribe to automatic updates to this status page.
Please do not refer to any NCSA Industry Partners on this page. Please use the iforge nomenclature for all of the *forge infrastructure.
To see older events, see Archive of NCSA Status Home
Report a problem
Current Status
| START | END | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
| 2020-12-21 18:00 | 2020-12-22 0700 | File & Print Servers | Monthly Windows File & Print Server Maintenance | Windows File Shares such as HR, Business Office, Home, etc. and printing in the NCSA & NPCF buildings are unavailable. | help+its@ncsa.illinois.edu | IN PROGRESS |
Upcoming Scheduled Maintenance
Listed below in chronological order.
| Start | End | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
| 2021-01-12 07:00 | 2021-01-12 8:00 | JIRA | JIRA Upgrade to 8.13.2 | All JIRA users | help+its@ncsa.illinois.edu | SCHEDULED |
| 2021-01-25 | 2021-01-25 | NCSA LDAP | Users with /bin/csh will have their shells changed to /bin/bash | Users logging into systems that don't override the /bin/csh data already will find they are using /bin/bash when they login. | help+service@ncsa.illinois.edu | SCHEDULED |
| 2021-02-25 0800 | 2021-02-25 1200 | LSST | LSST Quarterly Maintenance
| All LSST services hosted at NCSA | lsst-admin@ncsa.illinois.edu | SCHEDULED |
| 2021-06-24 0800 | 2021-06-24 1200 | LSST | LSST Quarterly Maintenance
| All LSST services hosted at NCSA | lsst-admin@ncsa.illinois.edu | SCHEDULED |
| 2021-09-30 0800 | 2021-09-30 1200 | LSST | LSST Quarterly Maintenance
| All LSST services hosted at NCSA | lsst-admin@ncsa.illinois.edu | SCHEDULED |
| 2021-12-09 0800 | 2021-12-09 1200 | LSST | LSST Quarterly Maintenance
| All LSST services hosted at NCSA | lsst-admin@ncsa.illinois.edu | SCHEDULED |
Previous Outages or Maintenance
?
| Start | End | What System/Service was affected? | What happened? | What was affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
| 2020-12-18 | 2020-12-18 | NCSA LDAP | New user accounts will have their shell in ldap set to /bin/bash | New users will have /bin/bash as their default shell | help+service@ncsa.illinois.edu | COMPLETE |
| 2020-12-10 0800 | 2020-12-10 1400 | LSST | LSST Quarterly Maintenance
| All LSST services hosted at NCSA | lsst-admin@ncsa.illinois.edu | COMPLETE |
| 2020-12-09 0800 | 2020-12-10 1000 | HAL | HAL Quarterly Maintenance | HAL cluster | help+isl@ncsa.illinois.edu | COMPLETED |
| 2020-11-24 10:30 | 2020-11-24 17:35 | Blue Waters compute | Boot RAID lost fiber channel connection for reasons not understood | Full system outage | jenos@illinois.edu | COMPLETED |
2020-11-19 11:00 a.m. | 2020-11-19 11:30 a.m. | DNS1 Migration | Neteng will be migrating DNS1 to a new switch. | We need to physical move the cable to DNS1 which will cause momentary outage for dns queries to DNS1. DNS2 will not be affected by the migration. | help+neteng@ncsa.illinois.edu | COMPLETED |
2020-11-18 09:00 | 2020-11-17 17:00 | UPS battery Monitor | Program and configure the new BMS | All UPS connected loads | rantissi@illinois.edu | COMPLETED |
| 2020-11-16 18:00 | 2020-11-16 21:30 | NCSA File & Print Servers | File & Print servers were offline for scheduled maintenance. Windows File Shares and printing were unavailable. | Windows File Shares such as HR, Business Office, Home, etc. and printing in the NCSA & NPCF buildings were unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-10-14 13:45 | 2020-10-14 14:04 | NCSA Wiki & Jira | NCSA's Wiki & Jira servers were restarted. | Wiki & Jira were offline while their servers reboot. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-11-13 12:00 | 2020-11-13 18:00 | Software Directorate VM Farm | Failing power supply of switch will be replaced, will use this to upgrade OS as well. | NCSA OpenSource, INCORE, etc (all machines running on 141.142.277.X). | kooper@illinois.edu | COMPLETED |
| 2020-11-12 9:55 | 2020-11-12 10:15 | cilogon.org | Update to OA4MP v5.0.2 was unsuccessful. | CILogon Service has been reverted to OA4MP 4.4.5. | help@cilogon.org | DELAYED |
| 2020-11-10 07:00 | 2020-11-10 21:05 | iForge cluster | Quarterly Maintenance Switching to GPFS 5 formatted filesystem. | All iForge nodes. | COMPLETE | |
| 2020-11-05 10:00 | 2020-11-05 12:24 | NPCF enterprise UPS | UPS maintenance, replace defective communication cards | Any rack (system) that is UPS power fed | rantissi@illinoise.edu |
|
| 2020-11-04 19:00 | 2020-110-4 21:00 | NCSA Building Router (2 of 2) | Software Updates | Software Updates will be applied to one of the NCSA building routers. Traffic will fall back to the seconds router. No network traffic should be affected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-11-02 08:00 | 2020-11-02 10:20 | NCSA GitLab git.ncsa.illinois.edu | LDAP authentication was disabled for NCSA GitLab. Users of the GitLab web interface are required to authenticate to NCSA through CILogon. | NCSA passwords can no longer access repositories. Use GitLab personal access tokens to authenticate against Git over HTTPS. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-11-02 09:00 | 2020-11-02 09:05 | NCSA Duo | The icon shown in the Duo app for NCSA will be updated to match the icon used in NCSA Slack. | NCSA Duo App pushes will show updated icon to match NCSA Slack. May need to restart phone/app to see updated icon. | help+duo@ncsa.illinois.edu | COMPLETED |
| 2020-10-30 08:00 | 2020-10-30 08:45 | SVN at subversion.ncsa.illinois.edu | Retired SVN Service at subversion.ncsa.illinois.edu | SVN is no longer be available. NCSA users are recommended to use one of our various Git repository options. | help+its@ncsa.illinois.edu | COMPLETED |
2020-10-29 9:00 | 2020-10-29 11:15 | iForge | maintenance to switch to GPFS version 5. | All nodes. | iforge-admin@ncsa.illinois.edu | COMPLETED |
| 2020-10-28 19:00 | 2020-10-28 21:00 | NCSA Building Router (1 of 2) | Software Updates | Software Updates will be applied to one of the NCSA building routers. Traffic will fall back to the seconds router. No network traffic should be affected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-10-27 22:00 | 2020-10-27 22:10 | NCSA VPN | Firmware Updates | Firmware updates were applied to the NCSA VPN. Any AnyConnect VPN sessions were reset during the maintenance and users may need to reconnect. Any IPSEC sessions failed over to the standby unit and were not affected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-10-27 09:00 | 2020-10-27 09:15 |
| The DNS CNAME entries for idp.ncsa.illinios.edu and crl.ncsa.illinois.edu will be changed from cilogon-web.ncsa.illinois.edu to shib-docker.security.ncsa.illinois.edu (141.142.149.33). NCSA Shib IdP v4.0.1 is currently up and running at 141.142.149.33. | help+idp@ncsa.illinois.edu | COMPLETED | |
| 2020-10-22 11am | 2020-10-22 1pm | security.ncsa.illinois.edu and grid.ncsa.illinois.edu | Cert replacement issue | Sites were down | cpitcel | RESOLVED |
| 2020 10-21 0900 | 2020 10-21 1400 | WAN Link Migration | ICCN Engineers will be migrating NCSA's 100G WAN links over to new optical cards. Below is the timetable for the moves: 10:00am CARNE (Node 1) to I2 (710 N Lakeshore Dr) 10:30am NCSA (Node 2) to MREN (710 N Lakeshore Dr) 11:30am CARNE (Node 2) to OmniPop (600 West Chicago) 12:00pm NCSA (Node 1) to I2 (600 West Chicago) 12:15pm NCSA (Node 1) to ESNet (600 West Chicago) | Individual links will be migrated one at a time, each taking roughly 15-20 minutes to complete, leaving redundant paths operational. Traffic will automatically be re-routed to these redundant paths during each link outage. There are exceptions where certain services won't failover in this way. In these cases, individual notifications have been sent out to affected parties. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-10-13 08:00 | 2020-10-13 09:00 | CILogon | Update to OA4MP v5.0 OAuth2/OIDC Libraries encountered issue with Syngenta IdP. Reverted to OA4MP v4.4.5. Will be addressed in future OA4MP update. | https://cilogon.org | help@cilogon.org | DELAYED |
| 2020 10-13 06:00 | 2020 10-13 06:15 | CILogon COmanage Registry at https://registry.cilogon.org | Service stack restart. | COmanage Registry and LDAP directory for the multi-tenant services. | help@cilogon.org | COMPLETED |
| 2020-10-12 18:00 | 2020-10-12 21:30 | NCSA File & Print Servers | Monthly Maintenance for Updates / Backup Checks | File & Print Servers were unavailable. Printing was offline, and fileserver shares were unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-10-01 06:00 | 2020-10-01 06:15 | NCSA VPN | The certificate for sslvpn.ncsa.illinois.edu was updated. | The SSL certificate has been updated. | neteng@ncsa.illinois.edu | COMPLETED |
| 2020-09-28 21:00 | 2020-09-29 16:45 | Nebula | Network card failed in network node, was replaced and network settings reconfigured | All non-virtual networking services for Nebula instances (north/south traffic) | nebula@ncsa.illinois.edu | COMPLETED |
2020-09-23 | 2020-09-24 | LSST | Monthly Maintenance:
| ALL LSST systems | lsst-admin@ncsa.illinois.edu | COMPLETED |
| 2020-09-22 | 2020-09-22 | CILogon multi-tenant COmanage Registry | Upgrade to version 3.3.0 | COmanage Registry service at https://registry.cilogon.org | help@cilogon.org | COMPLETED |
| 2020-09-14 18:00 | 2020-09-15 21:30 | NCSA File & Print Servers | Windows file and print servers were patched and unavailable during maintenance. | Access to Fileserver (Business Office, HR, Home, and Swap shared drives) was unavailable, printing was unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-09-08 08:00 | 2020-09-08 09:00 | CILogon | Update CILogon OIDC Client Admin API | https://cilogon.org | help@cilogon.org | COMPLETED |
| 2020-09-02 0800 | 2020-09-02 1600 | Core-East | Software Upgrades | No user impact expected | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-09-01 | 2020-09-01 | ldaps://ldap.cilogon.org | Restart of LDAP gateway service containers. | All LDAP services operated by CILogon. | help@cilogon.org | COMPLETED |
2020-08-28 1425 | 2020-08-28 1725 | NCSA Email from Campus Exchange | Campus Exchange could not deliver email addressed to "@ncsa.illinois.edu" addresses. A user requested a change to Exchange that broke delivery to NCSA. Any email you sent to "@ncsa.illinois.edu" addresses from campus Exchange will need to be resent by you. | Campus Exchange could not deliver email addressed to "@ncsa.illinois.edu" addresses. The change in Exchange was reverted around 1655, but it may take a bit of time for Office365 to update all of its servers. | help+its@ncsa.illinois.edu | RESOLVED |
| 2020-08-28 0800 | 2020-08-28 1000 | NPCF Networking DC Power System | Testing and maintenance of the DC power system and battery backup was performed. | No outage. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-08-26 0800 | 2020-08-26 1200 | Core-West | Replacing failed internal board. Software upgrades | No user impact expected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-08-17 0800 | 2020-08-22 19:00 | Code42 Crashplan Endpoint | One of the primary Code42 Crashplan Endpoints crashed and was not accepting backup requests | Code42 Crashplan clients that were assigned to backup to this server were unable to run backups. | crashplan@ncsa.illinois.edu | RESOLVED |
2020-08-18 06:00 | 2020-08-08 06:20 | NCSA VPN Service | Software upgrade was completed successfully. | All IPSEC sessions failed over successfully to the standby. Any users connected to the AnyConnect VPN were briefly disconnected and need to reconnect. An upgrade of the AnyConnect client was included with this upgrade and users will receive the upgrade when they reconnect to the VPN. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-08-18 06:00 | 2020-08-18 06:15 | registry.cilogon.org | CILogonPerson LDAP schema plugin was removed. | The COmanage Registry service. | help@cilogon.org | COMPLETED |
| 2020-08-11 07:00 | 2020-08-11 18:30 | iForge | Quarterly Maintenance | All systems will be unavailable during the maintenance | iforge-admin@ncsa.illinois.edu | COMPLETED |
| 2020-08-04 08:00 | 2020-08-04 08:30 | CILogon | https://go.ncsa.illinois.edu/CILogonServiceUpdate2020-08-04 | Remove <whitelisted> tag from idplist.xml file. Add "?initialidp=..." query parameter. | help@cilogon.org | COMPLETED |
| 2020-08-03 18:00 | 2020-08-03 23:00 | Fileserver & Printing | Monthly Windows Server Maintenance | NCSA Fileserver(s) and NCSA-Print were unavailable during the maintenance. Business Office, HR, home, swap, etc. shares were unavailable. Printing was unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-07-28 06:00 | 2020-07-28 06:05 | registry.cilogon.org | OA4MP plugin for creating/managing OIDC clients was updated | The COmanage Registry service was unavailable during the outage. | skoranda@illinois.edu | COMPLETED |
| 2020-07-24 0700 | 2020-07-24 0707 | Applied some minor configuration adjustments | Pages were unavailable for 5 minutes as the service restarted. | COMPLETED | ||
| 2020-07-22 0800 | 2020-07-22 0823 | git.ncsa.illinois.edu | GitLab was upgraded to the latest version | The git service was intermittently unavailable during this upgrade window. | COMPLETED | |
| 2020-07-18 0500 | 2020-07-18 0600 | NCSA WAN connectivity over Internet2 and ESnet links. | Emergency network maintenance performed by ICCN (our WAN provider) | Instability on WAN connectivity to Industry partners and research institutions using internet2 service provider network to connect to NCSA. General network connectivity will re-route accordingly. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-07-16 0700 | 2020-07-16 0800 | Core-West | A service alarm on Core-West switch requires a reset of an internal controller board. | Traffic will be failed over to Core-East while performing the maint. No impact expected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-07-15 1400 | 2020-07-15 1730 | Storage Condo | Perc Raid card failure in core server | Condo Services/GridFTP/NFS | ckerner@illinois.edu | COMPLETED |
| 2020-07-07 18:00 | 2020-07-07 21:30 | Fileserver & Printing | Monthly Windows Server Maintenance | NCSA Fileserver(s) and NCSA-Print were unavailable during the maintenance. Business Office, HR, home, swap, etc. shares were unavailable. Printing was unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-07-07 0800 | 2020-07-07 0900 | CILogon | Update Admin Client API | https://cilogon.org . No downtime is expected. | help@cilogon.org | COMPLETED |