| Atlassian has notified us of several critical security vulnerabilities in Confluence and Jira software. A mitigation has been applied to the Jira server and the Confluence server (wiki.ncsa.illinois.edu) will be patched. | There will be a brief outage to patch the Confluence server at 1600. The patching is expected to take 15-20 minutes but the entire hour is reserved as a precaution. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | Completed |
|---|
|
|
| 2023-12-06 0900 | 2023-12-06 1700 | Facility UPS | Preventive Maintenance _ Replace UPS capacitors. | All systems which are connected to UPS power. During the PM the systems will not lose power but will be unprotected. | MO Rantissi | UPS maintenance was halted due to damaged parts. Putting the UPS back together and rescheduling for a later date. The UPS is back online. |
| | IDDS database | Planned maintenance: postgresql upgrade | NCSA identity, group management, campus cluster user management page, TEM shift report tool, and naps | help+idds@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | Completed |
|---|
|
|
| 2023-11-09 1445 (vForge) 2023-11-09 1630 (license servers) | vForge / license servers | Quarterly Planned Maintenance | all vForge nodes and services (incl. related license servers/services) will be unavailable | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | Completed |
|---|
|
|
| | vForge / license servers | Quarterly Planned Maintenance | all vForge nodes and services (incl. related license servers/services) will be unavailable | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | Completed |
|---|
|
|
| | Radiant | Rebuilding rabbitmq service | Dashboard and API services were read-only during this time. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| | hub.ncsa.illinois.edu | private docker registry is down due to volumes in radiant in detaching state | hub.ncsa.illinois.edu is not reachable, and images stored are unreachable. Services that have their images local should continue to run, services that want push/pull images will get a 500 error. | Rob Kooper | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| | Confluence/Wiki | Upgrade the system | Confluence | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| 2023-10-31 09:30 | 2023-10-31 10:30 | NCSA OpenSource | upgrade Atlassian products | opensource confluence/jira/bamboo/bitbucket | | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| | HAL | Full System PM | All HAL services | | | Status |
|---|
| colour | Green |
|---|
| title | POSTPONED |
|---|
|
|
| | SSLVPN | New auth method was added to a new login profile, ncsa-vpn-saml-tunnelall. | There is now a test profile in place that isn't open to everyone. Please continue to use the profiles you were using before. If you notice and issue please report it. Our testing indicated logins were working as intended. | help+neteng@ncsa.illinois.edu | |
| | Radiant | OpenStack software update | The Radiant team will be conducting an OpenStack software update, from Victoria to Wallaby. This is a software stability update and does not include significant features or changes in functionality. The update will be done online and is not expected to impact running instances or system access. | help@ncsa.illinois.edu | |
| | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares will be unavailable during maintenance. Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable. | help@ncsa.illinois.edu | |
| | Taiga | Online, Rolling patch of Taiga servers | Taiga File System | set@ncsa.illinois.edu | |
| | HOLL-I | CS-2 Appliance Mode upgrade | All HOLL-I servers and services will be limited to internal testers | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | Completed |
|---|
|
|
| | HOLL-I | CS-2 Appliance Mode hardware installation | All HOLL-I servers and services will be unavailable | help@ncsa.illinois.edu | |
| | Confluence | Config will be applied to increase the period users can be logged in before logged out | Confluence will be down | help@ncsa.illinois.edu | |
| | sslvpn | testing new auth method | no user impact was observed | help+neteng@ncsa.illinois.edu | |
| 2023-10-10 0800 | 2023-10-10 1000 | cilogon.org | Moving to new compute infrastructure | cilogon.org, demo.cilogon.org, crl.cilogon.org | help@cilogon.org | |
| 2023-10-09 1215 | 2023-10-09 1532 | Taiga | Appliance has unmounted all of its OSTs. | Ability to do I/O to Taiga | set@ncsa.illinois.edu | |
| | Confluence | Confluence is being upgrade | Confluence will not be available for use | help@ncsa.illinois.edu | |
| 2023-10-04 0600 | 2023-10-04 1927 | Delta | Filesystem and OS patching | All Delta resources will be unavailable during the maintenance period
including:
+ Delta login nodes - unavailable
+ Delta compute nodes - unavailable
Delta services
+ Open OnDemand - unavailable
+ Delta Globus Online endpoint - unavailable | help@ncsa.illinois.edu | |
| 2023-10-04 0950 | 2023-10-04 1000 | Opensource Confluence | Patching confluece | opensource confluence will be down | | |
| 2023-09-27 1700 | 2023-09-28 0700 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | |
| 2023-09-26 1400 | 2023-09-26 1500 | Wireless NCSA building | Campus wireless outage. | NCSAnet and IllinoisNet users are experiencing connectivity issues. Tech Services is aware of the problem. | help+neteng@ncsa.illinois.edu | |
| 2023-09-28 1433 | 2023-09-28 1459 | cilogon.org | service outage due to AWS database issue | logins to cilogon.org were failing | help+cilogon@ncsa.illinois.edu | |
| 2023-09-26 7:30AM | 2023-09-26 8:00AM | Ldap Primary Server | Maintenance | Ldap updates will be disabled during maintenance | Timothy Bouvet | |
| 09/22/2023 8:00am | 9/22/2023 1:30pm | Wireless access | NCSANet is not authenticating users and denying connections. | Anyone attempting to connect to the wireless NCSANet ID. | neteng@ncsa.illinois.edu | |
| 2023-08-29 | 2023-09-21 - 1300 | opensource bitbucket | Bitbucket is not compatible with the deployed version of git, see https://jira.atlassian.com/browse/BSERV-14390 | opensource.ncsa.illinois.edu/bitbcket | | |
| 2023-Sep-19 - 0745 | 2023-Sep-19 - 0750 | LastPass | Rekey the LastPass/Duo Integration | LastPass users that utilize duo may not be able to authenticate until completed | James Eyrich | |
| 2023-Sep-18 - 1511 | 2023-Sep-18 - 1749 | Taiga | Outage due to failed MDS failover. | Taiga access was unavailable. | set@ncsa.illinois.edu | |
| 2023-09-14 0700 | 2023-09-14 2015 | vForge / license servers | Quarterly Planned Maintenance | all vForge nodes and services (and related license servers/services) will be unavailable | help@ncsa.illinois.edu | |
| 2023-09-14-0800 | 2023-09-14-2000 | Taiga & Granite Services | Semi-Annual Planned Maintenance | All Taiga and Granite services will be offline | set@ncsa.illinois.edu | |
| 2023-09-08 06:58 | 2023-09-08 09:50 | disruption to NPCF-DES-CORE, NPCF-CWMGMT-FW1 & 2, MForge VPN | NPCF-CORE-EAST has a DEAD linecard. Relocating affected links to other linecards with open ports while we work with vendor support for a replacement. | redundancy has been lost, access and activity remain normal. | | |
| 2023-09-06 09:48 | 2023-9-6 10:35 | NCSA Center Wide Management Network | the firewall protecting this network is showing offline | Centerwide management networks in NCSA building | (John) Walker | |
| 2023-9-6 09:50 | 2023-9-6 10:35 | The main switch in NCSA 3003 | In debugging a link problem between NCSA and NPCF the wrong fiber was inadvertently pulled | Networking in and out of 3003 was down for 35 mins | neteng@ncsa.illinois.edu | |
| 2023-08-26 23:06 | 2023-08-27 01:36 | CILogon | CILogon database replication error | CILogon OAuth/OIDC services unavailable | help+cilogon@ncsa.illinois.edu | |
| 2023-08-24 13:02 | 2023-08-25 14:15 | Taiga | Multiple SAS cable backend failure causing OSTs to go into write protect and unmount | Access to certain OSTs in Taiga | set@ncsa.illinois.edu | |
| 2023-08-16 1700 | 2023-08-17 0000 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | |
| 2023-08-16 0700 | 2023-08-16 1003 | Nightingale | Quarterly Planned Maintenance | All Nightingale servers and services were unavailable (other than the ngale-bastion* nodes) | help@ncsa.illinois.edu | |
| 2023-08-15 0400 | 2023-08-15 0500 | VMWare Gateway | VMWare is updating the Gateway OS | No expected effects | help@ncsa.illinois.edu | |
| 2023-08-14 0935 | 2023-08-14 2000 | NCSA VPN | Duo implemented new ssl checks that we were not passing | Users couldn't authenticate with DUO to establish new connections to the VPN. Existing VPN sessions remain connected. | Matthew Elliott | |
| 2023-07-27 1700 | 2023-07-27 1715 | HOLL-I | Live kernel patching | kernel was updated in response to recent security issue. | help@ncsa.illinois.edu | |
| 2023-07-25 0900 | 2023-07-25 1630 | Radiant | Changes to the OpenStack network configuration and network service node (increasing MTU on customer networks and adding a new dedicated network server) | These changes will impact project/instance networks and cause them to be unreachable for an extended period of time. Expect network timeouts and failure of NFS file system access. Systems may be unreachable for several hours - up to the entire planned time - but we are making every effort to minimze the downtime. | James Glasgow via help@ncsa.illlinois.edu | |
| | ICCP | ICCP Quarterly Maintenance | All ICCP services | help@campuscluster.illinois.edu | |
| 2023-07-19 0800 | 2023-07-19 0900 | u1carne router | scheduled maintenance | mForge, Magnus, and Access will have a brief outage as the routers reboot. | Michael Douglas via neteng@ncsa.illinois.edu | |
| 2023-07-12 1700 | 2023-07-12 2130 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | |
| 2023-07-07 1000 | 2023-07-07 1100 | NCSA Kerberos | Deleting out principals that were disabled on 2023-06-07 | Kerberos authentication should already be disabled for the planned hosts, so there should be zero notable effect. | help@ncsa.illinois.edu | |
| | HOLL-I | Transitioning CS-2 Execution Mode from Weight Streaming to Pipelined | Holl-I CS-2 | help@ncsa.illinois.edu | |
| 2023-06-30 0600 | 2023-06-30 0605 | NCSA GitLab | GitLab was updated to latest version | All GitLab services was unavailable for a few minutes. | help@ncsa.illinois.edu | |
| 2023-06-29 1849 | 2023-06-29 2000 | Delta | More Power fluctuations due to the severe weather have caused in all NCSA buildings. NCSA staff are working to restore all services to full functionality. | Delta Login, Openondemand and Scheduling. | help@ncsa.illinois.edu | |
| 2023-06-29 1316 | 2023-06-29 1530 | Most NCSA computer systems | Power fluctuations due to the severe weather have caused multiple system failures in all NCSA buildings. NCSA staff are working to restore all services to full functionality. | Virtually all systems have been impacted to some extent. Most NCSA compute resources have returned to service. | help@ncsa.illinois.edu | |
| 2023-06-21 1800 | 2023-06-21 1900 | DNS1 / DNS2 | BIND security patches | Due to a security issue with BIND, neteng will be rebooting both DNS servers (staggered) starting tonight at 1800. | neteng@ncsa.illinois.edu | |
| 2023-06-15 0600 | 2023-06-15 0605 | NCSA GitLab | GitLab updated to use new backup method | All GitLab services were unavailable for a few minutes. | help@ncsa.illinois.edu | |
| 2023-06-14 1700 | 2023-06-14 2200 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | |
| 2023-06-08 1100 | 2023-06-08 1205 | ICI Metrics | Major Upgrade to Grafana 9.5.x and Unified Alerting | Access to https://metrics.ncsa.illinois.edu and all alerting was paused | malone12@illinois.edu | |
| 2023-06-07 1300 | 2023-06-07 1500 | NCSA Kerberos | Disabling Kerberos Host Principals not in DNS | Kerberos Authentication for hosts may stop working. Please create a ticket if you think your host principal may have been disabled erroneously. | help@ncsa.illinois.edu | |
| 2023-05-23 1240 | 2023-05-23 1410 | Taiga | Failover events on tgio02 | I/O to and from Taiga for all services intermittently during this period | set@ncsa.illinois.edu | |
| 2023-05-23 0800 | 2023-05-23 1400 | Granite Tape Archive | Unplanned Library Maintenance due to component failure | Retrieval of data; | bdickin2@illinois.edu | |
| 2023-05-18 0800 | 2023-05-18 1400 | Granite Tape Archive | Library Preventative Maintenance | Retrieval of data; | bdickin2@illinois.edu | |
| 2023-05-17 0700 | 2023-05-17 2125 | Nightingale | Quarterly Planned Maintenance | All Nightingale servers and services were unavailable (other than the ngale-bastion* nodes) | help@ncsa.illinois.edu | |
| 2023-05-17 0530 | 2023-05-17 0600 | Wireless and VoIP (NCSA Building) | Router Upgrades | Wireless, VoIP and anything directly connected to the campus switches will be down, while they upgrade firmware on the router. | help+neteng@ncsa.illinois.edu | |
| 2023-05-16 0700 | 2023-05-16 1100 | HOLL-I | Quarterly Planned Maintenance | all HOLL-I nodes and services were unavailable | help@ncsa.illinois.edu | |
| 2023-05-15 0900 | 2023-05-15 1200 | HOLL-I | CS-2 CDU maintenance | the HOLL-I CS-2 was unavailable and there was a reservation in Slurm | help@ncsa.illinois.edu | |
| 2023-05-12 0600 | 2023-05-12 0615 | NCSA GitLab | GitLab was updated to latest version | All GitLab services was unavailable for a few minutes. | help@ncsa.illinois.edu | |
2023-05-11 0700 | 2023-05-11 1900 | vForge / license servers | Quarterly Planned Maintenance | all vForge nodes and services (and related license servers/services) will be unavailable | help@ncsa.illinois.edu | |
| 2023-May-03 0800 | 2023-April-25 0900 | ACHE FW Cluster Upgrade - Secondary | Upgrading ACHE Firewall member B | No outage expected | eyrich@illinois.edu | |
| | vSphere and hosts on it. | VMWare Licensing issues. Was forced to migrate to new vSphere. | LDAP, Wordpress Sites, various | help@ncsa.illinois.edu | |
2023-05-01
1530 | 2023-05-01
1605 | ICI VMware | Apply updates to address software issue. | | aloftus@ncsa.illinois.edu | |
| 2023-April-26 0800 | 2023-April-25 0900 | ACHE FW Cluster Upgrade - primary | Upgrading ACHE Firewall member A | No outage expected | eyrich@illinois.edu | |
| 2023-April-25 0800 | 2023-April-25 0900 | NPCF CWFM Cluster Upgrade secondary | Upgrading NPCF CW Firewall member B | No outage expected | eyrich@illinois.edu | |
| | NCSA VPN | The certificate on the NCSA VPN was replaced. | Users will be disconnected from the VPN and have to manually reconnect. | neteng@ncsa.illinois.edu | |
|
|
|
|
|
|
|
| 2023-April-20 0800 | 2023-April-20 0900 | NPCF CWFM Cluster Upgrade primary | Upgrading NPCF CW Firewall member A | No outage expected | eyrich@illinois.edu | |
| 2023-04-19 1800 | 2023-04-19 2300 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing were unavailable. | help@ncsa.illinois.edu | |
| | Delta | HSN and OS is being updated, | The entire system will be offline. | kingda@illinois.edu | |
| 04/17/23 0900 | 04/17/23 1400 | Granite Tape Archive | Upgrades to FS | Ingest or retrieval of data; | bdickin2@illinois.edu | |
| 2023-03-27 | 2023-04-16 | NCSA OpenSource BitBucket | incompatibility with git, only versions that can be installed are 2.25 or 2.40, and Bitbucket requires version 2.31 - 2.39
https://opensource.ncsa.illinois.edu/bitbucket is down until new version of BitBucket |
| Rob Kooper | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| 2023-04-03 | 2023-04-04 | NCSAnet, IllinoisNet, EDUroam | Tech Services is deploying a new certificate for all wireless networks. | Check #announce on NCSA Slack for more information, including links to download software that will update your wireless profiles. | help+neteng@ncsa.illinois.edu | |
| 2023-03-29 12:00 CDT | 2023-03-29 12:30 CDT | Primary Kerberos server | Configuration changes to match secondary KDCs | Password changes may have been delayed by ten minutes | Christopher Lindsey | |
| 2023-03-23 0843 | 2023-03-23 1030 | DHCP serving NCSAnet wireless and NCSA office wired wall jacks | The main NCSA DHCP server stopped answering queries and was restarted | If you didn't already have a DHCP lease your system would have been unable to connect to NCSAnet or register on an office wired wall jack. | neteng@ncsa.illinois.edu | |
| 2023-03-15 1800 | 2023-03-16 2300 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | |
| 2023-03-14 1100 | 2023-03-14 1150 | Authentication to vsphere.ncsa.illinois.edu and ache-vcenter will fail | Replacing SSL certs on Ldap1/2 | Ldap will be restarted on Ldap1/2 | tbouvet@illinois.edu | |
2023-03-09 0700 | 2023-03-09 17:20 | vForge / license servers | Quarterly Planned Maintenance | all nodes and services will be unavailable | help@ncsa.illinois.edu | |
| 03/09/2023 1713 | NCSA Taiga & Granite | Taiga Service Node Updates & Granite Upgrade | Taiga Public LNET router was upgraded and a second one added; access via public LNET was down from 0800 to 1100. Globus and NFS services were patched in a rolling/online fashion.
Granite experienced a short full downtime as we upgraded its software. | set@ncsa.illinois.edu | |
| 03/07/2023 8:30am | 03/07/2023 10:15am | Delta HSN | The HSN was dropping nodes and not allowing nodes to reconnect | High Speed Connectivity | help@ncsa.illinois.edu | |
| 2023-03-01: 1100 | 2023-03-01: 1115 | Radiant OpenStack Services | Changes to the OpenStack controller node to address networking performance issues | All OpenStack services were restarted to effect system configuration changes. The work was completed successfully and all services are available again | help@ncsa.illinois.edu | |
| | NCSA email | A mail loop caused routing and processing problems. | Mail routing and delivery was blocked. | help@ncsa.illinois.edu | |
| | HOLL-I | Quarterly Planned Maintenance | all nodes and services will be unavailable | help@ncsa.illinois.edu | |
| 2023-02-16 ~14:15 | 2023-02-16 ~14:25 | cerberus4 | mis-configuration caused roughly 50% of connections to be dropped | 50% of connections in and out dropped | help+security@ncsa | |
| 2023-02-10 0910 | 2023-02-10 0915 | users.ncsa.illinois.edu web site | restarting the system | no web pages from users.ncsa.illinois.edu will be available | help@ncsa.illinois.edu | |
| 02/08/2023 1800 | 02/09/2023 0000 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | |
| | Jira | Jira will be restarted to fix stuck notification emails. | Jira will unavailable during this time. | Andrew Loftus Also posted to #announce (Slack) | |
| | ICCP head node login and golub compute resources | Lost network connectivity for golub infrastructure | ICCP head node logins (ie cc-login.campuscluster.illinois.edu) and golub compute resources | help@campuscluster.illinois.edu | |
| | Jira | Jira offline for service restart to fix stuck emails. | Jira will unavailable during this time. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | Completed |
|---|
|
|
| 01/25/2023 0800 | 01/25/2023 0830 | NCSA LDAP | rolling LDAP restarts of redundant servers to deploy new schema file | Minimal impact for service restarts | | | Status |
|---|
| colour | Green |
|---|
| title | Completed |
|---|
|
|
| 2023-01-19 1310 | 2023-01-19 1330 | Jira | Jira offline for reboot to fix Boards. | Jira will unavailable during this time. | help@ncsa.illinois.edu | | Status |
|---|
| subtle | true |
|---|
| colour | Green |
|---|
| title | Complete |
|---|
|
|
| | ICCP | ICCP Quarterly Maintenance | All ICCP services | help@campuscluster.illinois.edu | |
| 2023-01-13 1200 | 2023-01-13 1230 | Jira | Jira offline for dashboard fixes. | Jira will unavailable during this time. | help@ncsa.illinois.edu | | Status |
|---|
| subtle | true |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
| 2023-01-12 0800 | 2023-01-13 1230 | Jira | Minor issues noticed in Jira likely caused by the upgrade yesterday evening. | Gadgets and dashboards are having issues. | | | Status |
|---|
| subtle | true |
|---|
| colour | Green |
|---|
| title | resolved |
|---|
|
|
| 2023-01-11 0700 | 2023-01-12 1200 | Nightingale | Quarterly Planned Maintenance | All Nightingale servers and services will be unavailable (other than the ngale-bastion* nodes) Maintenance has been extended until noon Thu, Jan 12 due to complications with firmware update on the Lustre storage appliance. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| 2023-01-12 0700 | 2023-01-12 0715 | NCSA VPN | Router Migration | The NCSA VPN was migrated to a different upstream router. Users were briefly disconnected. | help+neteng@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| | NCSA GitLab | GitLab upgrade | All GitLab services were unavailable for a few minutes while it upgraded to the latest version. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| 06:40 1/9/2023 | 2023-01-11 2100 | vSphere in 3003 | One of the storage appliances serving vsphere.ncsa.uiuc.edu started access issues. This has caused issues with 19vms. | crashplan has returned to service | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| 2023-01-11 1730 | 2023-01-11 1915 | Jira | Jira software upgrade | Jira will be unavailable while software upgrades are applied. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| 1/9/2023 6:40am | 1/11/2023 various | vSphere in 3003 | One of the storage appliances serving vsphere.ncsa.uiuc.edu had access issues. Data was moved to different storage for affected VMs. | digitalag.ncsa.illinois.edu, gecat, reu.ncsa.illinois.edu, ACIpartnership.org, astro, edream, caiiwp, brainstormhpcd.org, internal-dev, cmdb-dev-kimber7, reu-international.ncsa.illinois.edu, avl-test, mharp - ergo, infews-er.net, ncsa30, bluewaters - 2018-03-05. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| 12/23/2022 6:30pm | 12/27/2022 1:30pm | Taiga | Single OST is failing to re-mount following failover | File system is unavailable | set@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| | Wireless at NCSA building. | Router Upgrade | Tech Services will be upgrading their NCSA building router which will effect wireless at the NCSA building. Downtime will be estimated at 15 mins. | help+neteng@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| | Radiant | System maintenance | OpenStack: - "Minor system configuration changes will be made to increase system logging and optimize memory usage/allocation across nodes. No noticeable impact to end users is expected."
Networking:
- Swap fiber links to correct issue with security taps: In order to minimize user impact, we will swap one link at a time. User should see no impact however there is a slight possibility of a temporary network outage potentially lasting a few minutes however we currently do not anticipate this happening.
- Update Ethernet switch firmware: Switch reboots will be done in a rolling fashion and so are not expected to be disruptive to ongoing operations (due to switch/path redundancy).
| help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| 15 Dec 2022 0900 | 15 Dec 2022 0935 | NCSA Kerberos | NCSA's Read-Write KDC is being upgraded | Password changes and new accounts are being queued for completion after the upgrade. | help@ncsa.illnois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| | NCSA GitLab | GitLab was upgraded to latest version | All GitLab services was unavailable for a few minutes. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
| 12-01-2022 0600 | 12-01-2022 0700 | NCSA VPN | Software Upgrades | The appliances hosting the NCSA VPN were patched. Users experienced a brief disconnect as load is failed over between the appliances. The anyconnect client was upgraded at this time | neteng@ncsa.illinois.edu | |
| | NCSA identity password resets | The password reset process is not completing. | Users password resets were queued and then applied when the issue was fixed. Users who tried to change their password should find there password is now set to the password of their last attempt. | help@ncsa.illinois.edu | |
| | capnjack (license server) | Changes to IPTABLES | Unknown servers. Licenses affected are IDL, PGI, Intel, MATLAB, Abaqus, Sention LM, Luda, Ansys, CDL, Adaptive, Converge, CFD, RLM Type, rr_ld | meberger@illinois.edu re: SVCPLAN-1465 | | Status |
|---|
| colour | Green |
|---|
| title | Completed |
|---|
|
|
| 2022-11-16 1042 | 2022-11-16 1351 | CILogon | Docker Swarm failure | CILogon services were unavailable. See: https://cilogon.statuspage.io/incidents/2blf564965s0 | help@cilogon.org | |
| 2022-11-15 0700 | 2022-11-15 1700 | HOLL-I | Quarterly Planned Maintenance | all nodes and services will be unavailable | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
2022-11-10 0700 | 2022-11-10 1200 | vForge / license servers | Quarterly Planned Maintenance | all nodes and services will be unavailable | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | Completed |
|---|
|
|
| 2022-11-10 11:00 | 2022-11-10 11:50 | ASD Vsphere, specifically vm's using the tintri storage appliance. | Network connections were upgraded to 25G speed. | There was no disruption of service with this work. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
| 2022-11-07 0900 | 2022-11-07 0958 | set-analytics.ncsa.illinois.edu | Physical Machine Move from 3003 to NPCF | The SET managed Grafana/InfluxDB instance will be unavailable | set@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
| 2022-11-04 1900 | 2022-11-04 1930 | SET Taiga | SET caused a failover of tgio02 and then failed back. This fixed the mounting issue. | Clients with taiga currently mounted may experience slow or stopped IO during the failover. Failover completed properly and solved the mounting issue. | set@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
| 2022-11-03 1132 | 2022-11-04 1930 | Delta | Taiga filesystem (/taiga/ and /projects/) problem on dt-login01 and dt-login02 | The issue is limited to dt-login01 and dt-login02. Commands attempting to access /taiga/ or /projects/ on these nodes will hang. Users are advised to use dt-login03 or the login.delta.ncsa.illinois.edu "round robin" address UPDATE: dt-login01 and dt-login02 are fully functional again and back in the login.delta.ncsa.illinois.edu DNS "round robin". | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
| 2022-11-03 0048 | 2022-11-03 0106 | SET Taiga | tgio02 and tgio04 failed over | OSTs on the two nodes were inaccessible until the reboots were complete. This is a known issue with a vendor patch in progress. | set@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
| 2022-11-03 0600 | 2022-11-03 0615 | NCSA GitLab | GitLab was upgraded to latest version | All GitLab services was unavailable for a few minutes. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
| 2022-11-02 1700 | 2022-11-02 2000 | DNS Services | Patching for out of cycle security updates. | DNS1 and DNS2 will be patched and rebooted (staggered) to applied needed updates. | help+neteng@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
| 2022-11-01 1800 | 2022-11-02 0000 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
| | idp.ncsa.illinois.edu | Enable Duo Universal Prompt | NCSA Identity Provider will now use Duo Universal Prompt | help+idp@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| 2022-10-25 0800 | 2022-10-25 0900 | NCSA building 1st Floor Wifi / Security Cameras | Tech Services is replacing a networking switch on the 1st for of the NCSA building that powers the Access Points on the first floor. | This should be a short down time, but the access points will reboot while we migrate cables to the new switch. | help+neteng@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | completed |
|---|
|
|
| 2022-10-19 07:00 | 2022-10-20 07:15 | Some SSH Bastion Hosts | Out-of-Cycle reboot needed after failed patching.
Will reboot tomorrow at 07:00am | bwbh1.ncsa.illinois.edu
bwbh3.ncsa.illinois.edu
cerberus1.ncsa.illinois.edu
cerberus3.ncsa.illinois.edu
ache-bastion-1.ncsa.illinois.edu
ngale-bastion-1.ncsa.illinois.edu | help+security@ncsa.illinois.edu | | Status |
|---|
| colour | Green |
|---|
| title | COMPLETED |
|---|
|
|
| 2022-10-18 15:00 | 2022-10-18 15:30 | Radiant instance creation/management | system setting changes | No noticeable impact | pl@illinois.edu | |
| 2022-10-18 12:00 | 2022-10-18 12:05 | identity, email to NCSA addresses | system updates | 1 minute window to cause email delays and identity frontend unavailable | cpl@illinois.edu | |
| 2022-10-14 2200 | 2022-10-15 | NCSA office firewall upgrade | Upgrading code on the office firewall. | Office networks will be offline during this upgrade. | help+neteng@ncsa.illinois.edu | |
| 2022-10-13 1700 | 2022-10-13 1800 | SSLVPN Maintenance | The second member of the HA pair will be put back into service. | The second member was added with no outage. | help+neteng@ncsa.illinois.edu | |
| 2022-10-12 11:00 | 2022-10-12 12:00 | ASD Vsphere, specifically vm's using the tintri storage appliance. | Network connection on tintri storage box were switch to new hardware but their speed was unchanged. Additional work will need to be scheduled to complete the speed increase. | This had no service impact. | help@ncsa.illinois.edu | | Status |
|---|
| colour | Yellow |
|---|
| title | INCOMPLETE |
|---|
|
|
| 2022-10-10 0000 | 2022-10-10 1040 | NCSA VPN | The NCSA VPN had a member of the HA pair fail and licensing didn't fail over. | Users were unable to connect to the VPN until the licensing issue was resolved. | help+neteng@ncsa.illinois.edu | |
| 2022-10-03 0800 | 2022-10-03 0845 | HOLL-I | install security updates and reboot |
| help@ncsa.illinois.edu | |
| 2022-09-30 0600 | 2022-09-30 0615 | NCSA GitLab | GitLab was upgraded to latest version | All GitLab services were unavailable for a few minutes. | help@ncsa.illinois.edu | |
| 2022-09-27 1100 | 2022-09-28 1700 | odd numbered bastion hosts (cerberus1, cerberus3, ache-bastion-1, ngale-bastion-1, etc.) | puppet code refactoring for SSH configs | More changes were pushed out around 5p on 2022-09-28 and we believe the SSHD config issues are resolved. You can use the even numbered (cerberus2, cerebrus4) bastions as a work-around if any issues persist. | help+security@ncsa.illinois.edu | |
| 2022-09-28 0930 | 2022-09-28 1050 | Jira outgoing email | outgoing email degraded | Jira failed to send some/most outgoing email during this time frame. | help@ncsa.illinois.edu | |
| 2022-09-24 1445 | 2022-09-25 1045 | Granite | Building power outage caused Disk Storage Unit to power cycle | Any user operations on the cluster were interrupted and unavailable until resolution. | bdickin2@illinois.edu | |
| 2022-09-21 0800 | 2022-09-21 0930 | HOLL-I | Change CS-2 execution mode to Pipelined | Execution mode of the CS-2 was changed from Weight Streaming to Piplined. | help@ncsa.illinois.edu | |
| 2022-09-08 0800 | 2022-09-10 1100 | Granite | Granite Bi-annual Maintenance (now back in service) | Any ingest or retrieval to/from the Archive | bdickin2@illinois.edu slack-id: briandi
set@ncsa.illinois.edu | |
| 2022-09-09 0943 | 2022-09-09 1457 | Jira | outgoing email degraded | Jira failed to send some/most outgoing email during this time frame. | help@ncsa.illinois.edu | |
2022-09-08
0700 | 2022-09-08 1010: license servers 2022-09-09
0230: vForge | vForge / license servers | Quarterly Planned Maintenance | all nodes and services will be unavailable | help@ncsa.illinois.edu | |
| | ASD VM services net | Routing in the switch stacks is being swiched from NCSA 3003 to NPCF | All systems on the 141.142.192.x network will be unreachable for up to 5 minutes. | help@ncsa.illinois.edu | |
| 2022-08-31 1800 | 2022-09-01 0700 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were be unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | |
| | Jira | Jira service will be restarted | Jira will not be available | help@ncsa.illinois.edu | |
| 08-24-22 1830 | 08-26-22 0800 | Granite Tape Archive | FS crash and lockup | A few files that were transferred into the archive shortly before the crash needed to be re-transferred. | bdickin2@illinois.edu slack-id: briandi
set@ncsa.illinois.edu
| |
2022-08-17
1200 | n/a | All LSST hosts at NCSA | Servers will be shutoff and retired. | All LSST servers and services at NCSA. | lsst-admin@ncsa.illinois.edu | |
| 2022-08-17 0700 | 2022-08-17 1320 | Nightingale | Quarterly Planned Maintenance | All Nightingale servers and services will be unavailable (other than the ngale-bastion* nodes) | help@ncsa.illinois.edu | |
| 2022-08-16 0700 | 2022-08-17 1305 | HOLL-I | Quarterly Planned Maintenance | All HOLL-I servers and services will be unavailable 2022-08-16 1505 - HOLL-I cluster return to service, but CS-2 remains offline for further work; CS-2 expected return to service by 2022-08-17 1000 2022-08-17 1305 - HOLL-I CS-2 is returned to service | help@ncsa.illinois.edu | |
| 2022-08-09 2000 | 2022-08-09 2300 | Office Networks on 2nd Floor | Code updates on office network switches. | Office ports will be offline as switches reboot. | help+neteng@ncsa.illinois.edu | |
| 2022-08-10 2000 | 2022-08-10 2300 | Office Networks on 3rd Floor | Code updates on office network switches. | Office ports will be offline as switches reboot. | help+neteng@ncsa.illinois.edu | |
| 2022-08-11 2000 | 2022-08-11 2300 | Office Networks on 4th Floor | Code updates on office network switches. | Office ports will be offline as switches reboot. | help+neteng@ncsa.illinois.edu | |
2022-08-03
0900 | 2022-08-03
1000 | NPCF Center-wide management firewalls | Secondary firewall will be upgraded | No impact to services is anticipated. Traffic will flow normally through the primary firewall as the secondary is upgraded. | help+security@ncsa.illinois.edu | |
2022-07-27
0940 | 2022-07-28
15:36 | ACHE, Nightingale | Several accounts have had their Covered Entity status revoked | Affected users/accounts will not be able to access resources that requires Covered Entity enrollment | help+hippa@ncsa.illinois.edu | |
2022-07-27
0900 | 2022-07-27
1000 | NPCF Center-wide management firewalls | Primary firewall will be upgraded | No impact to services is anticipated. Traffic will be failed over to the secondary firewall, the primary will be updated, and then traffic will be moved back to the primary. | help+security@ncsa.illinois.edu | |
| | Jira | Additional LDAP group will be added for exclusion to sync with LDAP users. | In theory, nothing. | help@ncsa.illinois.edu | |
| | ICC | ICC Quarterly Maintenance | All ICC services | help@campuscluster.illinois.edu | |
| 2022-07-19 0700 | 2022-07-19 0900 | Radiant | Victoria Update | Minimally disruptive, brief interruptions to OpenStack services, such as the Horizon dashboard | radiant-admin@ncsa.illinois.edu | |
2022-07-14
2345 | 2022-07-14
2359 | Wiki | The service will be restarted in order to increase the login timeout. | Wiki will be unavailable for about 5 mins while it restarts. | | |
2022-07-08
1700 | 2022-07-11
0800 | LSST hosts in NCSA 3003 | Due to a full building power outage at NCSA on Sunday, 10 July, some LSST servers will be unavailable over the weekend. Servers will be shutdown at COB on Friday and returned to service on Monday morning. | lsst-dbb-fts1
lsst-dbb-rucio
lsst-demo
lsst-dm-monitor
lsst-int-monitor
lsst-mon-dev
lsst-pup
lsst-test5
lsst-xfer
l1-cl-arctl
l1-cl-fault
l1-cl-header
nts-ccamfwdr1
nts-acamfwdr2
nts-acamfwdr1 | lsst-admin@ncsa.illinois.edu | |
| 2022-07-11 08:30 | 2-22-07-11 9:30 | All ITSM (CMDB) VMs | All ITSM VMs are currently down. Ticket has been created to get them brought back up. | Production CMDB service (openDCIM) is not available | kimber7@illinois.edu | |
2022-07-10 0700 | 2022-07-10 1430 | NCSA building power | Building power feed work for multiple campus Buildings | AVL, LSST, ISL and Software standard services were down from Friday afternoon until Monday morning. | Daniel Lapine | |
2022-07-8 1600 | 2022-07-11 0900 | cerberus2 and cerberus4 | Campus is doing work on a common feed that affects multiple buildings, include the NCSA Building. Work is scheduled from 0700-1700, but may finish early | VM hosts running these 2 bastions will be down for the weekend due to the scheduled power work at NCSA | help+security@ncsa.illinois.edu | |
| 2022-07-06 1730 | 2022-07-06 2030 | Wiki (wiki.ncsa.illinois.edu) | Confluence and MySQL upgrades | wiki will be down during the upgrade | | |
| 2022-07-05 1800 | 2022-07-05 2130 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | |
| 2022-07-05 1800 | N/A | iForge | end of service | iForge was removed from service. Operations have moved to the new vForge virtual cluster. | help+industry@ncsa.illinois.edu | |