status.ncsa.illinois.edu
Watch this page in the wiki to subscribe to automatic updates to this status page.
Please do not refer to any NCSA Industry Partners on this page. Please use the iforge nomenclature for all of the *forge infrastructure.
To see older events, see Archive of NCSA Status Home
Report a problem
Current Status
| START | END | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|
Upcoming Scheduled Maintenance?
| Start | End | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|
Previous Outages or Maintenance?
| Start | End | What System/Service was affected? | What happened? | What was affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
| 2020-09-08 08:00 | 2020-09-08 09:00 | CILogon | Update CILogon OIDC Client Admin API | https://cilogon.org | help@cilogon.org | COMPLETED |
| 2020-09-02 0800 | 2020-09-02 1600 | Core-East | Software Upgrades | No user impact expected | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-09-01 | 2020-09-01 | ldaps://ldap.cilogon.org | Restart of LDAP gateway service containers. | All LDAP services operated by CILogon. | help@cilogon.org | COMPLETED |
2020-08-28 1425 | 2020-08-28 1725 | NCSA Email from Campus Exchange | Campus Exchange could not deliver email addressed to "@ncsa.illinois.edu" addresses. A user requested a change to Exchange that broke delivery to NCSA. Any email you sent to "@ncsa.illinois.edu" addresses from campus Exchange will need to be resent by you. | Campus Exchange could not deliver email addressed to "@ncsa.illinois.edu" addresses. The change in Exchange was reverted around 1655, but it may take a bit of time for Office365 to update all of its servers. | help+its@ncsa.illinois.edu | RESOLVED |
| 2020-08-28 0800 | 2020-08-28 1000 | NPCF Networking DC Power System | Testing and maintenance of the DC power system and battery backup was performed. | No outage. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-08-26 0800 | 2020-08-26 1200 | Core-West | Replacing failed internal board. Software upgrades | No user impact expected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-08-17 0800 | 2020-08-22 19:00 | Code42 Crashplan Endpoint | One of the primary Code42 Crashplan Endpoints crashed and was not accepting backup requests | Code42 Crashplan clients that were assigned to backup to this server were unable to run backups. | crashplan@ncsa.illinois.edu | RESOLVED |
2020-08-18 06:00 | 2020-08-08 06:20 | NCSA VPN Service | Software upgrade was completed successfully. | All IPSEC sessions failed over successfully to the standby. Any users connected to the AnyConnect VPN were briefly disconnected and need to reconnect. An upgrade of the AnyConnect client was included with this upgrade and users will receive the upgrade when they reconnect to the VPN. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-08-18 06:00 | 2020-08-18 06:15 | registry.cilogon.org | CILogonPerson LDAP schema plugin was removed. | The COmanage Registry service. | help@cilogon.org | COMPLETED |
| 2020-08-11 07:00 | 2020-08-11 18:30 | iForge | Quarterly Maintenance | All systems will be unavailable during the maintenance | iforge-admin@ncsa.illinois.edu | COMPLETED |
| 2020-08-04 08:00 | 2020-08-04 08:30 | CILogon | https://go.ncsa.illinois.edu/CILogonServiceUpdate2020-08-04 | Remove <whitelisted> tag from idplist.xml file. Add "?initialidp=..." query parameter. | help@cilogon.org | COMPLETED |
| 2020-08-03 18:00 | 2020-08-03 23:00 | Fileserver & Printing | Monthly Windows Server Maintenance | NCSA Fileserver(s) and NCSA-Print were unavailable during the maintenance. Business Office, HR, home, swap, etc. shares were unavailable. Printing was unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-07-28 06:00 | 2020-07-28 06:05 | registry.cilogon.org | OA4MP plugin for creating/managing OIDC clients was updated | The COmanage Registry service was unavailable during the outage. | skoranda@illinois.edu | COMPLETED |
| 2020-07-24 0700 | 2020-07-24 0707 | Applied some minor configuration adjustments | Pages were unavailable for 5 minutes as the service restarted. | COMPLETED | ||
| 2020-07-22 0800 | 2020-07-22 0823 | git.ncsa.illinois.edu | GitLab was upgraded to the latest version | The git service was intermittently unavailable during this upgrade window. | COMPLETED | |
| 2020-07-18 0500 | 2020-07-18 0600 | NCSA WAN connectivity over Internet2 and ESnet links. | Emergency network maintenance performed by ICCN (our WAN provider) | Instability on WAN connectivity to Industry partners and research institutions using internet2 service provider network to connect to NCSA. General network connectivity will re-route accordingly. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-07-16 0700 | 2020-07-16 0800 | Core-West | A service alarm on Core-West switch requires a reset of an internal controller board. | Traffic will be failed over to Core-East while performing the maint. No impact expected. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2020-07-15 1400 | 2020-07-15 1730 | Storage Condo | Perc Raid card failure in core server | Condo Services/GridFTP/NFS | ckerner@illinois.edu | COMPLETED |
| 2020-07-07 18:00 | 2020-07-07 21:30 | Fileserver & Printing | Monthly Windows Server Maintenance | NCSA Fileserver(s) and NCSA-Print were unavailable during the maintenance. Business Office, HR, home, swap, etc. shares were unavailable. Printing was unavailable. | help+its@ncsa.illinois.edu | COMPLETED |
| 2020-07-07 0800 | 2020-07-07 0900 | CILogon | Update Admin Client API | https://cilogon.org . No downtime is expected. | help@cilogon.org | COMPLETED |
2020-06-24 | 2020-06-25 Kubernetes updates completed at 1330 | LSST | Monthly Maintenance:
| ALL LSST systems | lsst-admin@ncsa.illinois.edu | COMPLETE |
| 2020-06-17 0600 | 2020-06-17 1600 | Blue Waters scratch filesystem | Disk failure during OST failover, both OSTs unavailable 14 drives offline, reassembly required | Some filesystem operations. | COMPLETED | |
| 2020-06-15 08:00 | 2020-06-15 11:50 | Software Directorate VM Farm
| Upgrade of servers, including vm servers | During this time all servers will be down, servers will be returned ASAP | COMPLETED | |
| 2020-06-10 09:00 | 2020-06-12 21:00 | hal.ncsa.illinois.edu | Quarterly PM | HAL System Service | dmu@illinois.edu | COMPLETE |
| 2020-06-08 18:00 | 2020-06-09 07:00 | Fileserver & Printing | Monthly Windows Server Maintenance | NCSA Fileserver(s) and NCSA-Print will be unavailable during the maintenance. Business Office, HR, home, swap, etc. shares will be unavailable. Printing will be unavailable. | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-06-04 09:30 | 2020-06-04 14:46 | vsphere.ncsa.illinois.edu | Updated vCenter SSL certificate and trust chain | Management of VM's was unavailable while updating SSL certificates | help+its@ncsa.illinois.edu | RESOLVED |
| 2020-05-30 08:00 | 2020-06-01 12:00 | linux.ncsa.illinois.edu public-linux.ncsa.illinois.edu | SSH password-based authentication were failing due to changes with intermediate certificates | SSH password-based authentication broke. Kerberos based auth continued to work. | RESOLVED | |
| 2020-5-28 07:00 | 2020-5-29 09:00 | NCSA Virtual Classroom | Nodes were added, network reconfigured, and updates were applied. | Student VM's were be unavailable | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-5-22 07:57 | 2020-5-22 14:00 | Blue Waters Compute | Mistaken cabinet removed from configuration causing unroutable configuration for HSN | All compute, all running jobs | COMPLETE | |
| 2020-05-20 10:00 | 2020-05-20 14:00 | DNS1/2 | Upgrades | DNS servers will be rebooted during this time. | help+neteng@ncsa.illinois.edu | COMPLETE |
2020-5-14 06:00 | 2020-5-14 08:00 | NCSA Wiki | Upgrade | Wiki pages will be unavailable | swrights@illinois.edu | COMPLETE |
| 2020-05-19 09:30 | 2020-05-19 10:40 | netact.ncsa.illinois.edu | Fixing a problem with apache | netact is down. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2020-05-12 07:00 | 2020-05-12 17:00 | iForge | Quarterly Maintenance | All systems will be unavailable during the maintenance | iforge-admin@ncsa.illinois.edu | COMPLETE |
| 2020-05-04 18:00 | 2020-05-04 22:30 | NCSA Fileservers & Print Servers | Monthly ITS Windows Server Maintenance | Fileserver Shares (HR, Business Office, Home, Swap, etc.) and shared printers on NCSA-Print | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-04-30 9:00 | 2020-04-30 11:00 | Systems connection to idds-prod | ITS will be updating firewall settings for idds-prod. | No impact is expected, but users should contact help+idds if issues occur. | help+idds@ncsa.illinois.edu | COMPLETE |
| 2020-04-22 8:00 | 2020-04-22 9:00 | CILogon | https://go.ncsa.illinois.edu/CILogonServiceUpdate2020-04-22 | CILogon will relax name and email attribute requirement for IdPs. | help@cilogon.org | COMPLETE |
| 2020-04-21 6:00 | 2020-04-21 6:05 | CILogon | AWS CILogon COmanage update | HTTP (80/443) and LDAP (389/636) ports will be unavailable | help@cilogon.org | COMPLETE |
| 2020-04-15 9:25 | 2020-04-15 10:01 | Campus Cluster user portal | Login access via UIUC Shibboleth was not working, while Shibboleth configurations were updated in support of new Shibboleth version | Login access via UIUC Shibboleth was not working. | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-04-14 11:08 | 2020-04-14 11:38 | vsphere.ncsa.illinois.edu | Vcenter was upgraded. | Management of VM's was unavailable for 30 minutes. | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-04-14 10:00 | 2020-04-14 12:00 | CILogon | https://go.ncsa.illinois.edu/CILogonServiceUpdate2020-04-14DELAYED | An incompatibility in the OIDC "getcert" endpoint was discovered. The update has been delayed. | help@cilogon.org | COMPLETE |
| 2020-04-13 13:55 | RSA Authentication Manager and SecurID | RSA Authentication Manager service was turned off. | Authentication using NCSA RSA tokens is no longer supported. If you are using RSA with other organizations it should continue to work. | otp@ncsa.illinois.edu | COMPLETE | |
2020-04-13 0900 | 2020-04-13 00930 | All Globus Services currently using RSA authentication | RSA authentication will be changed to DUO authentication | All Globus Services currently using RSA authentication | help+globus@ncsa.illinois.edu | COMPLETE |
| 2020-04-07 10:00 | 2020-04-07 13:00 | Cerberus Bastions, BWBH Bastions | These systems will be migrated from using RSA to DUO for their second factor. | SSH logins on the hosts: cerberus{1,2}..ncsa.illinois.edu bwbh{1,2}.ncsa.illinois.edu | help+security@ncsa.illinois.edu | COMPLETE |
| 2020-04-01 18:00 | 2020-04-01 22:00 | Windows Server Maintenance | NCSA Windows File & Print Servers were unavailable. Users were not be able to access data on fileserver, or print to any printers in the building while maintenance was completed. | NCSA File & Print Servers | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-04-01 9:02 | RSA SecurID self-service portal, https://otp.ncsa.illinois.edu/ | Portal was turned off. | If you need to change your PIN on activate a new phone you won't be able to. | otp@ncsa.illinois.edu | COMPLETE | |
| 2020-03-29 12:02 | 2020-03-29 18:41 | Blue Waters compute service | High speed network out of service | All compute service, running jobs lost. | RESOLVED | |
| 2020-03-19 10:00 am | 2020-03-19 10:20 am | NAPS Application | NAPS upgrade complete | A set of planned changes including new features and improvements to existing ones were deployed to produciton. | Kimber Blum (kimber7@illinois.edu) or help+idds@ncsa.illinois.edu, Alina Banerjee(alinab@illinois.edu) | COMPLETE |
| 2020-3-16 8AM | 2020-3-16 4PM | Blue Waters compute | HSN issue | Compute was rebooted | COMPLETE | |
| 2020-03-16 10:00 am | 2020-03-16 01:00 pm | Main UPS/Critical Power | UPS annual maintenance | All production areas (no intended power interruptions, just loss of UPS functionality during the work) | rantissi@ | COMPLETE |
| 2020-03-11 15:00 | 2020-30-11 17:00 | netact | Upgrades | Netact will be down for system updates | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2020-03-09 19:08 | 2020-03-10 02:00 | VMware vSphere infrastructure for BW, iForge, ICC | vSphere data store failed | The NPCF data store failed. Optional NFS datastores are available to rebuild VMs. VMs used by Industry, ICC and BW needed to be recovered and rebuilt. | help+its@ncsa.illinois.edu | RESOLVED |
| 2020-03-10 10:00 | 2020-03-10 11:00 | CILogon, NCSA IdP, XSEDE IdP | Apache HTTPD SSL configuration change to require TLSv1.2 . | https:// connections to cilogon.org, demo.cilogon.org, ecp.cilogon.org, idp.ncsa.illinois.edu, and idp.xsede.org will use TLSv1.2 exclusively. Older clients may be impacted. | help@cilogon.org , help+idp@ncsa.illinois.edu | COMPLETE |
2020-03-05 06:00 | 2020-03-05 06:55 | NCSA VPN Service | Software Upgrades | All IPSEC sessions were seamlessly failed over. Any users connected to the AnyConnect VPN were disconnected and need to reconnect. | help+neteng@ncsa.illinois.edu | RESOLVED |
| 2020-03-04 11:14 | 2020-03-04 11:21 | NCSA Wiki | A virtual CPU became disabled and triggered a reboot | wiki.ncsa.illinois.edu was unavailable while it rebooted. | help+its@ncsa.illinois.edu | RESOLVED |
| 2020-03-03 17:00 | 2020-03-03 19:00 | DNS1/DNS2 | OS patching | Both DNS servers will be patched and rebooted. There may be some delays in DNS resolution during that time frame. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2020-03-03 11:00 | 2020-03-03 12:00 | NCSA-Print / Printing | Some users are experiencing issues with printing | Printing | help+its@ncsa.illinois.edu | RESOLVED |
| 2020-03-03 11:00 | 2020-03-03 11:03 | public-linux upgrade | The public-linux server was upgraded. | public-linux.ncsa.illinois.edu hostname now redirects to the new linux.ncsa.illinois.edu replacement server. | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-03-02 17:00 | 2020-03-02 22:30 | Windows Server Maintenance | Windows Servers such as Fileservers and Print Servers were upgraded/patched. | NCSA Windows File & Print Servers were unavailable. Users were not be able to access data on fileserver, or print to any printers in the building until maintenance was completed. | help+its@ncsa.illinois.edu | COMPLETE |
| 2020-02-27 08:00 | 2020-02-27 12:00 | LSST | Monthly Maintenance:
| ALL LSST systems will be updated, including:
| lsst-admin@ncsa.illinois.edu | COMPLETE |
| 2020-02-17 07:00 am | 2020-02-18 05:00 pm | Select clowder systems, users have been notified | Migration from NCSA to AWS | Select clowder systems | kooper@illinois.edu | COMPLETE |
| 2020-02-18 06:00 | 2020-02-18 06:30 | CILogon COmanage Registry | CILogon COmanage Registry AWS infrastructure update | https://registry.cilogon.org ports 80 and 443 will be unavailable for approx. 5 minutes. LDAP ports 389 and 636 will not be affected | help@cilogon.org | COMPLETE |
| 2020-02-17 18:00 | 2020-02-17 21:30 | NCSA LDAP | The primary LDAP server ran out of disk space, later causing intermittent outages with all LDAP replica servers. | All ITS managed LDAP servers, including:
| help+its@ncsa.illinois.edu | RESOLVED |
| 2020-02-17 12:00 pm | 2020-02-17 3:00 pm | OSN Pod | Ceph Update | All OSN Pod services | bdickin2@illinois.edu; sstevens@illinois.edu | COMPLETE |
| 2020-02-14 10:00 | 2020-02-14 14:00 | Wired networking in NCSA building | Some users reported they were unable to connect to the internet through their wired network connection. Wireless remained fully operational. | DHCP for NCSA building wired network. | help+neteng@ncsa.illinois.edu | Resolved |
| 2020-02-11 07:00 | 2020-02-11 1635 | iForge | Quarterly Maintenance | All systems will be unavailable during the maintenance | iforge-admin@ncsa.illinois.edu | Complete |
2020-02-04 10:00 | 2020-02-04 12:00 | CILogon upgrade | CILogon Service web front-end Bootstrap upgrade (http://bit.ly/36BvG57) | No downtime is expected. | help@cilogon.org | Primary production server upgraded. Secondary production server to be upgraded in a week. |
| 2020-02-03 10:00 | 2020-02-03 10:05 | Systems connection to idds-prod | ITS will be updating firewall settings for idds-prod. | No impact is expected, but users should contact help+idds if issues occur. | Complete | |
| 2020-01-30 13:00 | 2020-01-30 13:05 | Systems using acctd | IDDS will install triggers on the production database to support the new project-data message. | There are no changes that need to be made to current acctd implementations. The only impact acctd users may notice is the presence of project-data messages in acctd logs. | Complete | |
| 2020-01-30 8:00 | 2020-01-30 9:00 | LSST Firewalls | Firewall upgrade | No impact is expected. Traffic to/from 141.142.181.0/24 and 141.142.182.128/26 will be failed over from the primary firewall to the secondary firewall while the primary is upgraded, then failed back. Traffic between these subnets and the LSST storage network does not traverse the firewall. | help+security@ncsa.illinois.edu | COMPLETE |
| 2020-01-28 17:00 | 2020-01-28 19:00 | DHCP Upgrade | OS updates | DHCP server will be rebooted for all office and wireless networks. All connected clients will not be affected. Any new IP requests during the reboot will be delayed. This shouldn't be impacting for most users. | help+neteng@ncsa.illinois.edu | Completed |
| 2020-01-28 17:00 | 2020-01-28 19:00 | Exit-East Router | OS update | Most traffic will be sent via our second router. Some specific projects may be affected. Neteng will talk to those projects directly. | help+neteng@ncsa.illinois.edu | Completed |
| 2020-01-27 11:54 | 2020-01-28 09:36 | oa4mp.ncsa.illinois.edu | an automated CA certificate update caused authentication failures | NCSA RSA authentication to Globus was unavailable | help+idp@ncsa.illinois.edu | temporary work-around in place; proper fix scheduled for 2020-01-29 14:00 (note: oa4mp.ncsa.illinois.edu is scheduled for retirement on 2020-04-01) |
| 2020-01-21 0815 | 2020-01-21 0825 | ldap2 | ldap2 was returning ldap queries inconsistently so the service was restarted. | login to certain services was unusually slow for some users. Jira being the top problem. | help+its@ncsa.illinois.edu | ldap2 queries are working as expected after the restart. |
| 2020-01-16 : 1730 | 2020:01-16: 1748 | Condo NFS service | NFS exports are failing path resolutioin | NFS file system client mounts | Chad Kerner | Servers rebooted, mounts restored |
| 2020-01-15 08:00 | 2020-01-16 01:55 | ICCP | Quarterly Maintenance
| Total outage including export nodes (access to HTC will still available) | iccp-admins@campuscluster.illinois.edu | Complete |
| 2020-01-15 07:00 | 2020-01-15 12:00 | LSST NCSA Test Stand | Hardware repair in NCSA Test Stand | 21 servers in the NCSA Test Stand had their drive backplanes replaced by the vendor. | lsst-admin@ncsa.illinois.edu | COMPLETE |
2020-01-06 10:00 | 2020-01-08 14:30 | Code42 Crashplan Endpoints | The Code42 Crashplan servers start edpushing out Code42 Crashplan client updates | All users of CrashPlan will have their clients upgraded. | help+its@ncsa.illinois.edu | Complete |
2020-01-03 10:20 | 2020-01-03 11:20 | Code42 Crashplan was upgraded | Software updates to the CrashPlan Auth and Storage servers were applied | Backups were queued while the services restarted. | help+its@ncsa.illinois.edu | Complete |
| 2020-01-02 11:30 | 2020-01-02 17:39 | NCSA ITS vSphere vCenter | VCenter was upgraded to latest patch level. Due to some bugs it took longer to apply than expected. | The VMware administrative interface was unavailable during the update. | help+its@ncsa.illinois.edu | Complete |
2019-12-18 08:00 | 2019-12-18 10:00 | Facility infrastructure Electrical Transformer TX-5C | Replace defective temperature controller | "No Outage" Production projects on feeder C | MO Rantissi | Complete |
| 2019-12-17 06:00 | 2019-12-17 10:25 | JIRA | JIRA Upgrade from 7.6 to 8.5 | All JIRA users | help+its@ncsa.illinois.edu | COMPLETE |
| 2019-12-12 08:00 | 2019-12-12 14:00 | LSST | Monthly Maintenance:
| ALL LSST systems will be updated, including:
| lsst-admin@ncsa.illinois.edu | COMPLETE |
| 2019-12-12 10:02 | 2019-12-12 10:08 | internal.ncsa.illinois.edu | System memory was exhausted and OOM killer started killing https connections. | Savanna tools were unavailable | help+its@ncsa.illinois.edu | Memory resources for the server were doubled and service was brought back online. |
| 2019-12-10 13:45 | 2019-12-10 16:55 | Internet2 Connectivity | Internet2 Engineers isolated the issue to a malformed route update coming from an external peer to one of its nodes in Ashburn, VA. As this update was propagated throughout the Internet2 Network, it triggered a bug on the Internet2 routers and caused all internal BGP sessions of each router to rapidly flap, thus causing instability across the footprint. Engineers mitigated the issue by placing a filter on the specific peer to reject the malformed packet. The Major Incident has been resolved at this point. | Many different external resources, data transfers, sessions, etc. to various destinations. | help+neteng@ncsa.illinois.edu | Connectivity has stabilized. Please report any issues should they arise. |
| 2019-12-2 | 2019-12-2 afternoon | Wireless network | Tech Services reports they are having authentication issues affecting Wifi and VPN. Engineers are working on the problem. Tech Services Issue Description. | NCSAnet, IllinoisNet wireless are non functional at the moment. NCSA wired network remains available. IllinoisNet_guest is also functional. | help+neteng@ncsa.illinois.edu | Troubleshooting in progress |
| 2019-11-14 18:00 | 2019-11-14 19:00 | Exit-West Router | Software Upgrades | This should not be user impactful. All traffic will re-route via the other router. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2019-11-14 5:00 AM | 2019-11-14 3:30 PM | Nearline Endpoint | Issue with one storage library | Some Globus transfers were stalled for the period of the outage | bw+storage@ncsa.illinois.edu | COMPLETE |
| Nov 7 10:00 | Nov 7 14:00 | ICCP. All login nodes will be down. | Reroute some IB cables between Core switches and compute nodes. Changing topology on Subnet Manager. | Scheduler will be pause. No users access to login nodes. All running jobs will be kill. | help@campuscluster.illinois.edu | COMPLETE |
| 2019-11-05 07:00 | 2019-11-05 16:53 | iForge | Quarterly Maintenance | All systems will be unavailable during the maintenance | iforge-admin@ncsa.illinois.edu | COMPLETE |
| 2019-10-1 | 2019-11-1 | NCSA Windows Domain Controllers | ITS Migrated all Windows Systems to using the Campus Domain. The existing NCSA Windows Domain has been decommissioned and shutdown. | NCSA Windows Systems | help+its@ncsa.illinois.edu | COMPLETE |
2019-10-23 8 a.m. | 2019-10-23 12:00 p.m. | Core-West | Code upgrades will be performed on Core-West network switch. | This should not be user impacting. All traffic will flow through the redundant Core. | neteng+help@ncsa.illinois.edu | COMPLETE |
| 2019-10-22 06:12 | 2019-10-22 07:18 | Jira and Wiki | During reboots for system patches the wiki and Jira got stuck in a state that was not providing data to the users. | Only web access to these tools was impacted. | help+its@ncsa.illinois.edu | COMPLETE |
| 2019-10-16 08:00 | 2019-10-16 20:30 | ICC system wide | Quarterly maintenance | All services on ICC | help@campuscluster.illinois.edu | COMPLETE |
2019-10-16 8 a.m. | 2019-10-16 12:00 p.m. | Core-East | Code upgrades will be performed on Core-East network switch. | This should not be user impacting. All traffic will flow through the redundant Core. | neteng+help@ncsa.illinois.edu | COMPLETE |
| 2019-10-15 11:45am | 2019-10-15 11:56AM | npcf-exit-east | BGP peering flapped over I2 AL2S circuit | Traffic got re-routed but some WAN services were impacted as reported by users. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2019-10-10 07:00 | 2019-10-10 07:30 | mysql.ncsa.illinois.edu | Some table repairs broke replication; this maintenance will update the replicas with newer databases so the service will work as expected again. | Wiki, JIRA, and some web sites will stop working. Email forwarding to user accounts at NCSA will be delayed during the outage. | lindsey@ncsa.illinois.edu | COMPLETE |
2019-10-01 | 2019-10-03 | NCSA-Print & Building Printers | Some printers are having issues connecting to the NCSA Print Server. After updating drivers on the print server, public printers are working as expected. | Printing | help+its@ncsa.illinois.edu | COMPLETE |
| 2019-10-03 6AM | 2019-10-03 7:45AM | Jira and Wiki | During reboots for system patches the wiki and Jira got stuck in a state that was not providing data to the users. | Only web access to these tools was impacted. | help+its@ncsa.illinois.edu | COMPLETE |
| 2019-10-01 7AM | 2019-10-01 8:30PM | Blue Waters | NGA work load scheduled testing | scheduler testing for NGA workload | David King | COMPLETE |
| 2019-10-01 10AM | 2019-10-01 12:04PM | Blue Waters | EPO 4 racks lost xdp (cooling) CRAY warm swapped racks back into system successfully. | scheduler, some computes missing and Gemini was rerouted | COMPLETE | |
| 2019-10-01 07:00 | 2019-10-01 07:30 | mysql.ncsa.illinois.edu | MySQL servers needed to be synchronized to convert the server in NPCF back to a replicated host. | Wiki, JIRA, and some web sites stopped working. Email forwarding to user accounts at NCSA was delayed during the outage. | lindsey@ncsa.illinois.edu | COMPLETE |