status.ncsa.illinois.edu
Watch this page in the wiki to subscribe to automatic updates to this status page.
Please do not refer to any NCSA Industry Partners on this page. Please use the iforge nomenclature for all of the *forge infrastructure.
To see older events, see Archive of NCSA Status Home
Report a problem
Current Status
| Start | End | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|
Upcoming Scheduled Maintenance
Listed below in chronological order.
| Start | End | What System/Service is affected | What is happening? | What will be affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
|
|
| ||||
2022-11-10 | 2022-11-10 1900 | vForge / license servers | Quarterly Planned Maintenance | all nodes and services will be unavailable | help@ncsa.illinois.edu | SCHEDULED |
| 2022-11-15 0700 | 2022-11-15 1700 | HOLL-I | Quarterly Planned Maintenance | all nodes and services will be unavailable | help@ncsa.illinois.edu | SCHEDULED |
Previous Outages or Maintenance
| Start | End | What System/Service was affected? | What happened? | What was affected? | Contact Person | Status |
|---|---|---|---|---|---|---|
2022-11-10 | 2022-11-10 1200 | vForge / license servers | Quarterly Planned Maintenance | all nodes and services will be unavailable | help@ncsa.illinois.edu | COMPLETED |
| 2022-11-10 11:00 | 2022-11-10 11:50 | ASD Vsphere, specifically vm's using the tintri storage appliance. | Network connections were upgraded to 25G speed. | There was no disruption of service with this work. | help@ncsa.illinois.edu | COMPLETED |
| 2022-11-07 0900 | 2022-11-07 0958 | set-analytics.ncsa.illinois.edu | Physical Machine Move from 3003 to NPCF | The SET managed Grafana/InfluxDB instance will be unavailable | set@ncsa.illinois.edu | COMPLETED |
| 2022-11-04 1900 | 2022-11-04 1930 | SET Taiga | SET caused a failover of tgio02 and then failed back. This fixed the mounting issue. | Clients with taiga currently mounted may experience slow or stopped IO during the failover. Failover completed properly and solved the mounting issue. | set@ncsa.illinois.edu | COMPLETED |
| 2022-11-03 1132 | 2022-11-04 1930 | Delta | Taiga filesystem (/taiga/ and /projects/) problem on dt-login01 and dt-login02 | The issue is limited to dt-login01 and dt-login02. Commands attempting to access /taiga/ or /projects/ on these nodes will hang. Users are advised to use dt-login03 or the login.delta.ncsa.illinois.edu "round robin" address UPDATE: dt-login01 and dt-login02 are fully functional again and back in the login.delta.ncsa.illinois.edu DNS "round robin". | help@ncsa.illinois.edu | COMPLETED |
| 2022-11-03 0048 | 2022-11-03 0106 | SET Taiga | tgio02 and tgio04 failed over | OSTs on the two nodes were inaccessible until the reboots were complete. This is a known issue with a vendor patch in progress. | set@ncsa.illinois.edu | COMPLETED |
| 2022-11-03 0600 | 2022-11-03 0615 | NCSA GitLab | GitLab was upgraded to latest version | All GitLab services was unavailable for a few minutes. | COMPLETED | |
| 2022-11-02 1700 | 2022-11-02 2000 | DNS Services | Patching for out of cycle security updates. | DNS1 and DNS2 will be patched and rebooted (staggered) to applied needed updates. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2022-11-01 1800 | 2022-11-02 0000 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | COMPLETED |
0800 | 0830 | idp.ncsa.illinois.edu | Enable Duo Universal Prompt | NCSA Identity Provider will now use Duo Universal Prompt | help+idp@ncsa.illinois.edu | COMPLETED |
| 2022-10-25 0800 | 2022-10-25 0900 | NCSA building 1st Floor Wifi / Security Cameras | Tech Services is replacing a networking switch on the 1st for of the NCSA building that powers the Access Points on the first floor. | This should be a short down time, but the access points will reboot while we migrate cables to the new switch. | help+neteng@ncsa.illinois.edu | COMPLETED |
| 2022-10-19 07:00 | 2022-10-20 07:15 | Some SSH Bastion Hosts | Out-of-Cycle reboot needed after failed patching. Will reboot tomorrow at 07:00am | bwbh1.ncsa.illinois.edu bwbh3.ncsa.illinois.edu cerberus1.ncsa.illinois.edu cerberus3.ncsa.illinois.edu ache-bastion-1.ncsa.illinois.edu ngale-bastion-1.ncsa.illinois.edu | help+security@ncsa.illinois.edu | COMPLETED |
| 2022-10-18 15:00 | 2022-10-18 15:30 | Radiant instance creation/management | system setting changes | No noticeable impact | pl@illinois.edu | COMPLETE |
| 2022-10-18 12:00 | 2022-10-18 12:05 | identity, email to NCSA addresses | system updates | 1 minute window to cause email delays and identity frontend unavailable | cpl@illinois.edu | COMPLETE |
| 2022-10-14 2200 | 2022-10-15 | NCSA office firewall upgrade | Upgrading code on the office firewall. | Office networks will be offline during this upgrade. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2022-10-13 1700 | 2022-10-13 1800 | SSLVPN Maintenance | The second member of the HA pair will be put back into service. | The second member was added with no outage. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2022-10-12 11:00 | 2022-10-12 12:00 | ASD Vsphere, specifically vm's using the tintri storage appliance. | Network connection on tintri storage box were switch to new hardware but their speed was unchanged. Additional work will need to be scheduled to complete the speed increase. | This had no service impact. | help@ncsa.illinois.edu | INCOMPLETE |
| 2022-10-10 0000 | 2022-10-10 1040 | NCSA VPN | The NCSA VPN had a member of the HA pair fail and licensing didn't fail over. | Users were unable to connect to the VPN until the licensing issue was resolved. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2022-10-03 0800 | 2022-10-03 0845 | HOLL-I | install security updates and reboot | help@ncsa.illinois.edu | COMPLETE | |
| 2022-09-30 0600 | 2022-09-30 0615 | NCSA GitLab | GitLab was upgraded to latest version | All GitLab services were unavailable for a few minutes. | COMPLETE | |
| 2022-09-27 1100 | 2022-09-28 1700 | odd numbered bastion hosts (cerberus1, cerberus3, ache-bastion-1, ngale-bastion-1, etc.) | puppet code refactoring for SSH configs | More changes were pushed out around 5p on 2022-09-28 and we believe the SSHD config issues are resolved. You can use the even numbered (cerberus2, cerebrus4) bastions as a work-around if any issues persist. | help+security@ncsa.illinois.edu | RESOLVED |
| 2022-09-28 0930 | 2022-09-28 1050 | Jira outgoing email | outgoing email degraded | Jira failed to send some/most outgoing email during this time frame. | help@ncsa.illinois.edu | RESOLVED |
| 2022-09-24 1445 | 2022-09-25 1045 | Granite | Building power outage caused Disk Storage Unit to power cycle | Any user operations on the cluster were interrupted and unavailable until resolution. | bdickin2@illinois.edu | COMPLETE |
| 2022-09-21 0800 | 2022-09-21 0930 | HOLL-I | Change CS-2 execution mode to Pipelined | Execution mode of the CS-2 was changed from Weight Streaming to Piplined. | help@ncsa.illinois.edu | COMPLETE |
| 2022-09-08 0800 | 2022-09-10 1100 | Granite | Granite Bi-annual Maintenance (now back in service) | Any ingest or retrieval to/from the Archive | bdickin2@illinois.edu slack-id: briandi set@ncsa.illinois.edu | COMPLETE |
| 2022-09-09 0943 | 2022-09-09 1457 | Jira | outgoing email degraded | Jira failed to send some/most outgoing email during this time frame. | help@ncsa.illinois.edu | COMPLETE |
2022-09-08 | 2022-09-08 1010: license servers 2022-09-09 | vForge / license servers | Quarterly Planned Maintenance | all nodes and services will be unavailable | help@ncsa.illinois.edu | COMPLETE |
0500 | 0600 | ASD VM services net | Routing in the switch stacks is being swiched from NCSA 3003 to NPCF | All systems on the 141.142.192.x network will be unreachable for up to 5 minutes. | help@ncsa.illinois.edu | COMPLETE |
| 2022-08-31 1800 | 2022-09-01 0700 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were be unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | COMPLETE |
1730 | 1830 | Jira | Jira service will be restarted | Jira will not be available | help@ncsa.illinois.edu | COMPLETE |
| 08-24-22 1830 | 08-26-22 0800 | Granite Tape Archive | FS crash and lockup | A few files that were transferred into the archive shortly before the crash needed to be re-transferred. | bdickin2@illinois.edu slack-id: briandi | COMPLETE |
| 2022-08-17 1200 | n/a | All LSST hosts at NCSA | Servers will be shutoff and retired. | All LSST servers and services at NCSA. | lsst-admin@ncsa.illinois.edu | COMPLETE |
| 2022-08-17 0700 | 2022-08-17 1320 | Nightingale | Quarterly Planned Maintenance | All Nightingale servers and services will be unavailable (other than the ngale-bastion* nodes) | help@ncsa.illinois.edu | COMPLETE |
| 2022-08-16 0700 | 2022-08-17 1305 | HOLL-I | Quarterly Planned Maintenance | All HOLL-I servers and services will be unavailable 2022-08-16 1505 - HOLL-I cluster return to service, but CS-2 remains offline for further work; CS-2 expected return to service by 2022-08-17 1000 2022-08-17 1305 - HOLL-I CS-2 is returned to service | help@ncsa.illinois.edu | COMPLETE |
| 2022-08-09 2000 | 2022-08-09 2300 | Office Networks on 2nd Floor | Code updates on office network switches. | Office ports will be offline as switches reboot. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2022-08-10 2000 | 2022-08-10 2300 | Office Networks on 3rd Floor | Code updates on office network switches. | Office ports will be offline as switches reboot. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2022-08-11 2000 | 2022-08-11 2300 | Office Networks on 4th Floor | Code updates on office network switches. | Office ports will be offline as switches reboot. | help+neteng@ncsa.illinois.edu | COMPLETE |
| 2022-08-03 0900 | 2022-08-03 1000 | NPCF Center-wide management firewalls | Secondary firewall will be upgraded | No impact to services is anticipated. Traffic will flow normally through the primary firewall as the secondary is upgraded. | help+security@ncsa.illinois.edu | COMPLETE |
| 2022-07-27 0940 | 2022-07-28 15:36 | ACHE, Nightingale | Several accounts have had their Covered Entity status revoked | Affected users/accounts will not be able to access resources that requires Covered Entity enrollment | RESOLVED | |
| 2022-07-27 0900 | 2022-07-27 1000 | NPCF Center-wide management firewalls | Primary firewall will be upgraded | No impact to services is anticipated. Traffic will be failed over to the secondary firewall, the primary will be updated, and then traffic will be moved back to the primary. | help+security@ncsa.illinois.edu | COMPLETE |
0900 | 0915 | Jira | Additional LDAP group will be added for exclusion to sync with LDAP users. | In theory, nothing. | help@ncsa.illinois.edu | COMPLETE |
0800 | 2000 | ICC | ICC Quarterly Maintenance | All ICC services | COMPLETE | |
| 2022-07-19 0700 | 2022-07-19 0900 | Radiant | Victoria Update | Minimally disruptive, brief interruptions to OpenStack services, such as the Horizon dashboard | radiant-admin@ncsa.illinois.edu | COMPLETE |
| 2022-07-14 2345 | 2022-07-14 2359 | Wiki | The service will be restarted in order to increase the login timeout. | Wiki will be unavailable for about 5 mins while it restarts. | COMPLETE | |
| 2022-07-08 1700 | 2022-07-11 0800 | LSST hosts in NCSA 3003 | Due to a full building power outage at NCSA on Sunday, 10 July, some LSST servers will be unavailable over the weekend. Servers will be shutdown at COB on Friday and returned to service on Monday morning. | lsst-dbb-fts1 lsst-dbb-rucio lsst-demo lsst-dm-monitor lsst-int-monitor lsst-mon-dev lsst-pup lsst-test5 lsst-xfer l1-cl-arctl l1-cl-fault l1-cl-header nts-ccamfwdr1 nts-acamfwdr2 nts-acamfwdr1 | lsst-admin@ncsa.illinois.edu | COMPLETE |
| 2022-07-11 08:30 | 2-22-07-11 9:30 | All ITSM (CMDB) VMs | All ITSM VMs are currently down. Ticket has been created to get them brought back up. | Production CMDB service (openDCIM) is not available | kimber7@illinois.edu | RESOLVED |
2022-07-10 0700 | 2022-07-10 1430 | NCSA building power | Building power feed work for multiple campus Buildings | AVL, LSST, ISL and Software standard services were down from Friday afternoon until Monday morning. | Daniel Lapine | COMPLETE |
2022-07-8 1600 | 2022-07-11 0900 | cerberus2 and cerberus4 | Campus is doing work on a common feed that affects multiple buildings, include the NCSA Building. Work is scheduled from 0700-1700, but may finish early | VM hosts running these 2 bastions will be down for the weekend due to the scheduled power work at NCSA | help+security@ncsa.illinois.edu | COMPLETE |
| 2022-07-06 1730 | 2022-07-06 2030 | Wiki (wiki.ncsa.illinois.edu) | Confluence and MySQL upgrades | wiki will be down during the upgrade | COMPLETE | |
| 2022-07-05 1800 | 2022-07-05 2130 | NCSA File & Print Servers | Scheduled Windows Server Maintenance | File & Print Shares were unavailable during maintenance. Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable. | help@ncsa.illinois.edu | COMPLETE |
| 2022-07-05 1800 | N/A | iForge | end of service | iForge was removed from service. Operations have moved to the new vForge virtual cluster. | help+industry@ncsa.illinois.edu | RESOLVED |
Legend:
IN PROGRESS
RESOLVED
SCHEDULED
MONITORING