...
| Start | End | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2022-09-08 -31 18000800 | 2022-09-01 0700 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares will be unavailable during maintenance. Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable. | 08 2000 | Granite | Granite Bi-annual Maintenance | Any ingest or retrievel to/from the Archive | bdickin2@illinois.edu slack-id: briandi set@ncsa.illinois.edu help@ncsa.illinois.edu |
| ||||||||||
| 2022-09-08 0800 | 2022-09-08 2000 | Granite | Delta, HOLL-I, Radiant, DES, vForge | Taiga Granite Bi-annual Planned Maintenance. | Taiga mounts will be unavailable during maintenance. No data can be read or written Any ingest or retrievel to/from Taiga during the ArchivePM window. | Christopher Hellerbdickin2@illinois.edu slack-id: briandi set@ncsa.illinois.edu |
| |||||||||||||
2022-09-08 | 0800
| 2022-09-08 | 2000
| vForge / license servers | Quarterly Planned Maintenance | all nodes and services will be unavailable | help@ncsa.illinois.edu | Delta, HOLL-I, Radiant, DES, vForge | Taiga Bi-annual Planned Maintenance. | Taiga mounts will be unavailable during maintenance. No data can be read or written to/from Taiga during the PM window. | Christopher Heller | |||||||||
| Status |
|
| ||||||||||||||||||
-09-08 0800 | -09-08 | vForge / license servers | Quarterly Planned Maintenance | all nodes and services will be unavailable | 2000 | ICC | ICC Quarterly Maintenance | ICC Cluster nodes only | help@ncsa |
| ||||||||||
|
| ICC | ICC Quarterly Maintenance | ICC Cluster nodes only |
|
Previous Outages or Maintenance
Previous Outages or Maintenance
| Start | End | Start | End | What System/Service was affected? | What happened? | What was affected? | Contact Person | Status | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2022-08-31 1800 | 2022-09-01 0700 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were be unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu |
| ||||||||
1730 | 1830 | Jira | Jira service will be restarted | Jira will not be available | help@ncsa.illinois.edu |
| ||||||||
| 08-24-22 1830 | 08-26-22 0800 | Granite Tape Archive | FS crash and lockup | A few files that were transferred into the archive shortly before the crash needed to be re-transferred. | bdickin2@illinois.edu slack-id: briandi |
| ||||||||
| 2022-08-17 1200 | n/a | All LSST hosts at NCSA | Servers will be shutoff and retired. | All LSST servers and services at NCSA. | lsst-admin@ncsa.illinois.edu |
| ||||||||
| 2022-08-17 0700 | 2022-08-17 1320 | Nightingale | Quarterly Planned Maintenance | All Nightingale servers and services will be unavailable (other than the ngale-bastion* nodes) | help@ncsa.illinois.edu |
| ||||||||
| 2022-08-16 0700 | 2022-08-17 1305 | HOLL-I | Quarterly Planned Maintenance | All HOLL-I servers and services will be unavailable 2022-08-16 1505 - HOLL-I cluster return to service, but CS-2 remains offline for further work; CS-2 expected return to service by 2022-08-17 1000 2022-08-17 1305 - HOLL-I CS-2 is returned to service | help@ncsa.illinois.edu |
| ||||||||
| 2022-08-09 2000 | 2022-08-09 2300 | Office Networks on 2nd Floor | Code updates on office network switches. | Office ports will be offline as switches reboot. | help+neteng@ncsa.illinois.edu |
| ||||||||
| 2022-08-10 2000 | 2022-08-10 2300 | Office Networks on 3rd Floor | Code updates on office network switches. | Office ports will be offline as switches reboot. | help+neteng@ncsa.illinois.edu |
| ||||||||
| 2022-08-11 2000 | 2022-08-11 2300 | Office Networks on 4th Floor | Code updates on office network switches. | Office ports will be offline as switches reboot. | help+neteng@ncsa.illinois.edu |
| ||||||||
| 2022-08-03 0900 | 2022-08-03 1000 | NPCF Center-wide management firewalls | Secondary firewall will be upgraded | No impact to services is anticipated. Traffic will flow normally through the primary firewall as the secondary is upgraded. | help+security@ncsa.illinois.edu |
| ||||||||
| 2022-07-27 0940 | 2022-07-28 15:36 | ACHE, Nightingale | Several accounts have had their Covered Entity status revoked | Affected users/accounts will not be able to access resources that requires Covered Entity enrollment |
| |||||||||
| 2022-07-27 0900 | 2022-07-27 1000 | NPCF Center-wide management firewalls | Primary firewall will be upgraded | No impact to services is anticipated. Traffic will be failed over to the secondary firewall, the primary will be updated, and then traffic will be moved back to the primary. | help+security@ncsa.illinois.edu |
| ||||||||
0900 | 0915 | Jira | Additional LDAP group will be added for exclusion to sync with LDAP users. | In theory, nothing. | help@ncsa.illinois.edu |
| ||||||||
0800 | 2000 | ICC | ICC Quarterly Maintenance | All ICC services |
| |||||||||
| 2022-07-19 0700 | 2022-07-19 0900 | Radiant | Victoria Update | Minimally disruptive, brief interruptions to OpenStack services, such as the Horizon dashboard | radiant-admin@ncsa.illinois.edu |
| ||||||||
| 2022-07-14 2345 | 2022-07-14 2359 | Wiki | The service will be restarted in order to increase the login timeout. | Wiki will be unavailable for about 5 mins while it restarts. | Andrew Loftus |
| ||||||||
| 2022-07-08 1700 | 2022-07-11 0800 | LSST hosts in NCSA 3003 | Due to a full building power outage at NCSA on Sunday, 10 July, some LSST servers will be unavailable over the weekend. Servers will be shutdown at COB on Friday and returned to service on Monday morning. | lsst-dbb-fts1 lsst-dbb-rucio lsst-demo lsst-dm-monitor lsst-int-monitor lsst-mon-dev lsst-pup lsst-test5 lsst-xfer l1-cl-arctl l1-cl-fault l1-cl-header nts-ccamfwdr1 nts-acamfwdr2 nts-acamfwdr1 | lsst-admin@ncsa.illinois.edu |
| ||||||||
| 2022-07-11 08:30 | 2-22-07-11 9:30 | All ITSM (CMDB) VMs | All ITSM VMs are currently down. Ticket has been created to get them brought back up. | Production CMDB service (openDCIM) is not available | kimber7@illinois.edu |
| ||||||||
2022-07-10 0700 | 2022-07-10 1430 | NCSA building power | Building power feed work for multiple campus Buildings | AVL, LSST, ISL and Software standard services were down from Friday afternoon until Monday morning. | Daniel Lapine |
| ||||||||
2022-07-8 1600 | 2022-07-11 0900 | cerberus2 and cerberus4 | Campus is doing work on a common feed that affects multiple buildings, include the NCSA Building. Work is scheduled from 0700-1700, but may finish early | VM hosts running these 2 bastions will be down for the weekend due to the scheduled power work at NCSA | help+security@ncsa.illinois.edu |
| ||||||||
| 2022-07-06 1730 | 2022-07-06 2030 | Wiki (wiki.ncsa.illinois.edu) | Confluence and MySQL upgrades | wiki will be down during the upgrade |
| |||||||||
| 2022-07-05 1800 | 2022-07-05 2130 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu |
| ||||||||
| 2022-07-05 1800 | N/A | iForge | end of service | iForge was removed from service. Operations have moved to the new vForge virtual cluster. | help+industry@ncsa.illinois.edu |
| ||||||||
| 2022-06-28 0700 | 2022-06-28 1900 | Radiant | Unexpected complications during Radiant Maintenance |
Longer than expected outages of controller service. Instances that had floating IPs had no networking connectivity. Horizon dashboard and API was down (cannot launch new instances, etc). | radiant-admin@ncsa.illinois.edu |
| ||||||||
| 06-11-22 1400 | 06-14-2022 1630 | Granite Tape Archive | FS was locked up due to a bug alert setting; | Ingest or retrieval of data from the cluster | bdickin2@illinois.edu slack-id: briandi |
| ||||||||
| 2022-06-02 1800 | 2022-06-07 1830 | NCSA Wiki Servcie | Due to a critical security vulnerability announced by Atlassian we have been forced to restrict access to the NCSA Wiki to NCSA internal networks. This restriction will remain in place until Atlassian is able to provide a patch or mitigation for the vulnerability. | No remote access is allowed to the NCSA Wiki. Use the NCSA VPN for remote access. More information about using the VPN can be found here: https://users.ncsa.illinois.edu/clausen/NCSA_VPN_instructions_202206.pdf | help@ncsa.illinois.edu |
| ||||||||
| 2022-06-22 1430 | 2022-06-22 1900 | NCSA LDAP1 replica is down | LDAP1 database server is failed. The IAM team is investigating. | Only servers using ldap1 and should use ldap2 | tbouvet@illinois.edu |
| ||||||||
| 2022-06-22 1430 | 2022-06-22 1600 | NCSA LDAP central replicas (ldap2-3) and any services that rely on them. | LDAP database servers are failed. The IAM team is investigating. | Any service, such as the internal web server and Jira and Confluence servers, that rely on LDAP for user identification data may be affected. | help@ncsa.illinois.edu |
| ||||||||
1700 | 1830 | Confluence (Wiki) | Patching to address a security flaw | Confluence will not be accessible | help+service@ncsa.illinois.edu |
| ||||||||
| 2022-06-02 0600 | 2022-06-02 0615 | NCSA GitLab | GitLab was updated to latest version | All GitLab services were unavailable for a few minutes. | help+service@ncsa.illinois.edu |
| ||||||||
1700 | 1900 | Jira | Upgrade | Jira will not be available | help+service@illinois.edu |
| ||||||||
| 2022-06-01 0900 | 2022-06-01 1015 | Facility UPS | Replace two batteries, | All system with UPS feed, the UPS will stay online supporting loads but at reduced capacity and no outage expected. | rantissi@illinois.edu |
| ||||||||
| 2022-05-25 2230 | 2022-05-26 16:15 | Delta | 3 HSN switches were experiencing problems switches were updated and reconfigured |
| help@ncsa.illinois.edu |
| ||||||||
| 2022-05-25 1800 | 2022-05-25 2230 | Taiga - CenterWide FS | Partial outage. Some projects asked to temporary unmount /taiga | delta | Christopher Heller |
| ||||||||
| 2022-05-18 0700 | 2022-05-18 1400 | Nightingale | Nightingale Planned Maintenance | All Nightingale Services | help@ncsa.illinois.edu |
| ||||||||
| 2022-05-12 1700 | 2022-05-12 1800 | Jira & Wiki | Change to puppet configs | Downtime expected on each system for 1 to 5 minutes | help+service@ncsa.illinois.edu |
| ||||||||
2022-05-10 0700 | 2022-05-10 1900 | iForge / vForge / license servers | Quarterly Planned Maintenance | all nodes and services will be unavailable | help@ncsa.illinois.edu |
| ||||||||
| 2022-05-10 0800 | 2022-05-10 0815 | cilogon.org | Update to OA4MP v5.2.6 | Improvements in the back-end service | help@cilogon.org |
| ||||||||
| 2022-05-09 1800 | 2022-05-09 2130 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help+service@ncsa.illinois.edu |
| ||||||||
| 2022-05-04 1000 | 2022-05-04 1015 | IDDS Accounting Services | Planned Maintenance | All IDDS services (APIs, acctd, etc) | help+idds@ncsa.illinois.edu, tolbert@illinois.edu |
| ||||||||
| 2022-05-04 0600 | 2022-05-04 0622 | NCSA GitLab | GitLab was updated to latest version | All GitLab services were unavailable for a few minutes. | help+service@ncsa.illinois.edu |
| ||||||||
| 2022-04-19 12:00 | 2022-04-19 12:01 | Radiant | Restarted the AMQP service to put in some performance changes | New instance or virtual network changes that were submitted during the five-second restart may have failed | radiant-admin@ncsa.illinois.edu |
| ||||||||
| 2022 04-16 0600 | 2022 04-16 0630 | CILogon | Several cilogon.org services will be updated | https://cilogon.org , https://crl.cilogon.org , https://demo.cilogon.org , ldaps://ldap.cilogon.org | help@cilogon.org |
| ||||||||
| 2022-04-14 2100 | 0915 | Jira | New tickets cannot be created due to the user license limit being reached | Creation of new tickets. | https://www.ncsa.illinois.edu/expertise/user-services/user-support/ |
| ||||||||
2022 04-14 0800 | 2022 04-14 0830 | Wifi, VoIP, CCTV and FS networks at NCSA. | Tech services will be replacing their building router at NCSA. They expect a 10 mins outage. | Services may see a temporary interruption as cables are being changed. | help+neteng@ncsa.illinois.edu |
| ||||||||
2022 04-09 0600 | 2022 04-09 0700 | Internet2 / ESnet WAN connections. | During a few minute outage, some of our WAN circuits will be migrated. Traffic will be automatically re-routed. | help+neteng@ncsa.illinois.edu |
| |||||||||
| 2022-03-17 0900 | 2022-04-12 1030 | jira | ldap auths have been sporadically failing. This service is being monitored to determine a root cause. | Jira logins break | help+service@ncsa.illinois.edu |
| ||||||||
| 2022-04-12 0900 | 2022-04-12 0930 | vsphere.ncsa.illinois.edu | vcenter security updates are being installed | vm management interface will be unavailable for 15 mins. | help@ncsa.illinois.edu |
| ||||||||
| 2022-04-07 1900 | 2022-04-07 1950 | NCSA VPN | Software Upgrades / SSL Certificate | The appliances hosting the NCSA VPN were patched and receive an updated SSL certificate. Users will experience a brief disconnect as load is failed over between the appliances. | neteng@ncsa.illinois.edu |
| ||||||||
| 2022-04-06 2200 | 2022-04-07 0000 | Some office ports on the second floor. | Once of the switches on the second floor is experiencing a software problem and is currently down. Code updates are being applied. | One of the six switches on the second floor is down. Users who are connected to this port, might not receive link. | help+neteng@ncsa.illinois.edu |
| ||||||||
| 2022-04-06 1530 | 2022-04-07 0630 | All systems which mount/utilize Taiga | A bug involving the multirail functionality caused constant reboots with one of the metadata servers. This resulted in cluster de-stabilization and loss of function. | All lustre/NFS mountpoints to Taiga, Globus to Taiga. | help@ncsa.illinois.edu |
| ||||||||
| 2022-04-04 0930 | 2022-04-04 1000 | NCSA LDAP | Instantiation of Delta resource OU branch in the NCSA LDAP database with replication testing. | No impacts to properly configured systems or searches is expected. | help@ncsa.illinois.edu |
| ||||||||
| 2022-04-01 0600 | 2022-04-01 0700 | NCSA GitLab | GitLab was updated to latest version | All GitLab services was unavailable for a few minutes. | help+service@ncsa.illinois.edu |
|
...