Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Report a problem 

Current Status  

START
Start
END
EndWhat System/Service is affectedWhat is happening?What will be affected?Contact PersonStatus

Upcoming Scheduled Maintenance

Listed below in chronological order.

StartEndWhat System/Service is affectedWhat is happening?What will be affected?Contact PersonStatus

 0600

 0900

WikiUpgrade to next versionWiki will be unavailablehelp+service@ncsa
2024-05-15 07002024-05-15 19:00NightingaleQuarterly Planned MaintenanceAll Nightingale servers and services were unavailable (other than the ngale-bastion* nodes)help@ncsa.illinois.edu

Status
colour

Blue

Grey
title

scheduled2021-09-14 00002021-09-14 0600Internet2 WAN circuitInternet2 will be migrating our WAN circuit to new hardware. Traffic over that path will reroute while the change happens.  We anticipate the migration to take less than 30 mins.

SCHEDULED


Previous Outages or Maintenance

StartEndWhat System/Service was affected?What happened?What was affected?

Contact Person

Status

10:30

11:00

wiki.ncsa.illinois.eduwiki.ncsa.illinois.edu has slow load time or completely times out (ticketed as SVC-24573)Viewing/Editing Confluence (Wiki) pages.help@ncsa
help+neteng@ncsa
.illinois.edu

Status
colour

Blue

Green
title

scheduled

resolved

2021

2024-

09-30
08002021-09-30
1200LSST

LSST Quarterly Maintenance

  • TBD
All LSST services hosted at NCSA

04-25 17:00

2024-04-25 17:45

sslvpn.ncsa.illinois.eduAn update for an active 0day attack was installedvpn connections dropped for users and they had to reconnect.help+neteng@ncsa
lsst-admin@ncsa
.illinois.edu

Status
colour

Blue

Green
title

scheduled2021-10-20
08002021-10-20
2000ICCP

ICCP Quarterly Maintenance

  • TBD
ICCP Cluster nodes only

complete

 
1700


1700

NCSA WikiServer, DB, and Application upgradeswiki.ncsa.illinois.edu will be unavailable during this timehelp@ncsa
help@campuscluster
.illinois.edu

Status
colour

Blue

Green
title

scheduled

complete

2021

2024-

12-09
08002021-12-09
1200LSST

LSST Quarterly Maintenance

  • TBD
All LSST services hosted at NCSAlsst-admin@ncsa.illinois.edu StatuscolourBluetitlescheduled

Previous Outages or Maintenance

04-19 08:00

2024-04-19 09:15

DHCP leases for NCSANet and NCSA wired networkDHCP server stopped renewing IP address leases after a reboot for patchingDHCP lease renewals ceased, causing loss of connectivity for systems that tried to renew during the window. Already established connections were unaffected
StartEndWhat System/Service was affected?What happened?What was affected?

Contact Person

Status2021-09-09 06002021-09-09 0700NCSA VPNSoftware UpgradesThe appliances hosting the NCSA VPN will be patched. Users will experience a brief disconnect as load is failed over between the appliances
.help+neteng@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2021

2024-

09-08 13002021-09-08 1400Group prod_b Bastion hostsOut of cycle patchingBastion hosts in group prod_b will be patched and rebooted. (see MOTD for group assignment)

04-17 13:15

2024-04-17 13:16

ache-repo.ncsa.illinois.eduA filesystem is being fsck'ed.The data filesystem will be offline so packages will be unavailable.help+security@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2021

2024-

09

04-16 08

09002021-09-08 1000Group prod_a Bastion hostsOut of cycle patchingBastion hosts in group prod_a will be patched and rebooted. (see MOTD for group assignment)

:45

2024-04-16 09:15

TaigaRolling Failovers on Taiga's MDT0 applianceAccess to FS was interrupted

set@ncsa

help+security@ncsa

.illinois.edu

Status
colourGreen
title

COMPLETE

complete

2021

2024-

09

04-

02 9:30 AM

04 12:35

2024-04-04 16:35

Subset of Taiga native (Lustre) clientsIB Link on tgio11 began failing RDMA traffic causing some I/O interrupt issues on clients leveraging 3 LNET routers.  Access to the file system via these LNET routers is periodically timing out; suspect is bad IB cable.  Confirming with vendor. 

set@ncsa

2021-09-02 1PMPDU in rack AA81We are replacing a PDU in NPCF rack AA81All systems in the rack have redundant power connections.  No service outages are expected from this workhelp+service@ncsa

.illinois.edu

Status
colourGreen
title

COMPLETE2021-09-01 07002021-09-01 0800cilogon.orgUpdate to OA4MP v5.2.1Device Authorization Grant Flow transactions will be stored in database rather than in memoryhelp@cilogon.org

Mitigated

 
0700


0730 

VMware migrationsVMware hosts are migrating to a new license modelAll VM guest machines and all services should remain operational and accessible. No downtimes are expected.

help@ncsa

StatuscolourGreentitleCOMPLETE

 1200

 1205

WikiSecurity patch is being appliedWiki will be downhelp+service@ncsa

.illinois.edu


Status
colour

Blue

Green
title

scheduled

complete

2021

2024-

08-25 9:00am2021-08-25 6:45pmBlue Waters System reboot due to blade fallout coinciding with HSN reroute and SMW not recovering.All jobs interrupted

04-03 1800

2024-04-04 0700

NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares will be unavailable during maintenance.  Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable.help@ncsa.illinois
jenos@illinois
.edu

Status
colourGreen
title

Complete

complete

2021

2024-

08-19 05382021-08-19 0700IRST systems hosted on IRST Node 2Storage controller failure, all VMs taken offlinesome prod_b systems, and non-redundant services.

04-02 11:00

2024-04-02 11:30

IRST services, including systems run on IRST VMWare clustersmoving to upgraded switches/routers

Systems run by IRST, and any systems on the IRST-run VMWare cluster. Outage is expected to last < 5min.

help+security@ncsa.illinois
eyrich@illinois
.edu

Status
colourGreen
title

RESOLVED

complete

2021



2024-

08-19 5:342021-08-19 6:20cilogon.orgStorage controller failure in IRST VM farmcilogon.org was unreachable until we initiated fail-over to our backup servers at NICS.

02-28

2024-04-01

SSLVPNSSLVPN will start using CILogon for authentication and DUO integration. Four new profiles have been created (duplicating the existing four) but with the name "cilogon" in the name.  These new profiles will use the new authentication method.   After a few weeks of testing, if no issues are found, we will remove the old profiles on March 20. help+neteng@ncsa.illinois.edu
help@cilogon.org

Status
colourGreen
title

COMPLETE2021-08-18 11362021-08-18 1156NCSA WikiTest instance caused interference.NCSA Wiki

complete

 
2330

 
2345

NCSA GitLab,
NCSA Windows File & Print Servers
Web Redirect Server

VMs migrating to a new clusterAffected services will be unavailable for a few minutes.help@ncsa
help+service@ncsa
.illinois.edu

Status
colourGreen
title

COMPLETE2021-08-17 05002021-08-17 0700NCSA/NPCF Wide Area NetworkBetween 5:00AM and 7:00 AM CDT on 08/17/2021, Campus ICCN Engineers will be upgrading firmware on the ICCN router 710rtr at the Starlight facility in Chicago.Our peerings with MREN and OmniPoP will go down. All traffic destined for those peerings will reroute via other peerings, so no production impact

complete

 

 

NPCF WifiTech Services will be replacing the AP at NPCF.

Tech Services will be replacing the Access Points at NPCF.  No user impact is expected.

help+neteng@ncsa.illinois.edu

Status
colourGreen
title

COMPLETE2021-08-16 18002021-08-17 0000NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares will be unavailable during maintenance.  Users will not able to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable.help+service@ncsa.illinois.edu 

complete


2100 hrs


2200 hrs

Jira, Wiki, internal.ncsa.illinois.edu, identity.ncsa.illinois.eduVMs will migrate to a new cluster.Services will be unavailable for a few minutes (<5 mins) while the VM is shutdown and moved.help@ncsa.illinois.edu

Status
colourGreen
title

COMPLETE2021-08-12 9:542021-08-12 1012JiraAttempted snapshot of Jira in vSphere was too intensive for the systemJirahelp+service@illinois.edu

Status
colourGreen
titleCOMPLETE

2021-08-10
20002021-08-011
0000Radiant API and Web access
Radiant cluster name change.During this time access to the API endpoints and the Horizon web dashboard will be intermittently unavailable.  Instances will continue to run and be available over the network with no interruptions.

complete

 

 

NCSA WifiTech Services will be replacing the AP at the NCSA building. 

Tech Services will be replacing the Access Points at the NCSA building.  No user impact is expected.

help+neteng@ncsa
radiant-admin@ncsa
.illinois.edu

Status
colourGreen
title

COMPLETE

complete

2021

2024-

08-10 07:002021-08-10 17:10iForgeQuarterly MaintenanceAll systems unavailable

03-14 0700

2024-03-15  2140

vForge / license serversQuarterly Planned Maintenanceall vForge nodes and services (incl. related license servers/services) will be unavailablehelp@ncsa
iforge-admin@lists.ncsa
.illinois.edu

Status
colourGreen
titlecomplete

2021-08-09 14212021-08-09 1440NCSA WikiDB conflict configuration with Wiki & Wiki-TestNCSA Wiki was unaccessible


03/14/2024 080003/15/2024 2125
  • Extended outage due to a problematic upgrade solution. 
  • Vendor engineers involved
Taiga/GraniteSemi-Annual Maintenance

All Taiga & Granite Storage Services

set@ncsa
help+service@ncsa
.illinois.edu

Status
colourGreen
title

complete2021-08-05 10002021-08-05 1030NPCF Core Router - Linecard RebootA problem was identified on one of the line cards in our core router requiring a reboot of the linecard. The linecard was successfully rebooted and we will continue monitoring the hardware for further issues.All connections to this linecard are redundant and there was no impact to users.neteng@ncsa.illinois.edu

COMPLETED

1200

1500

vsphere.ncsa.illinois.edu consoleUpgradeThe vsphere.ncsa.illinois.edu web console. VMs should not be affectedhelp@ncsa
StatuscolourGreentitlecomplete2021-08-05
08002021-08-05
1000LSST

LSST Emergency OS Patching

LSST services hosted at NCSA except:

  • NTS will remain up (has already been patched)
lsst-admin@ncsa
.illinois.edu

Status
colourGreen
title

complete

completed

2021
2024-
08
03-
04
0800
072024-03
2021
-08
-04
1700Radiant API and Web access
Installation of new Radiant cluster
Cluster name changes are starting at 1100; This will make the horizon dashboard unreachable.
NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares will be unavailable during maintenance.  Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable.help@ncsa
During this time access to the API endpoints and the Horizon web dashboard will be intermittently unavailable.  Instances will continue to run and be available over the network with no interruptions.
radiant-admin@ncsa
.illinois.edu

Status
colourGreen
title

COMPLETED

completed

2021
2024-
08
02-
04 0700
22 0640
2021
2024-
08
02-
04 0800cilogon.orgUpdate to OA4MP v5.2.0Added support for Device Authorization Grant Flow (RFC 8628)
22 0648NCSA GitLabGitLab being updated to latest versionAll GitLab services will be unavailable for a few minutes.help@ncsa.illinois.edu
help@cilogon.org

Status
colourGreen
title

COMPLETED2021-08-03
08002021-08-03
1700Radiant API and Web access
Installation of new Radiant cluster
During this time access to the API endpoints and the Horizon web dashboard will be intermittently unavailable.  Instances will continue to run and be available over the network with no interruptions.

completed

1500

1600

ACHE vSphere is being upgradedACHE vSphere is being upgraded

ACHE vSphere will not be accessible

help@ncsa.illinois.edu

Status
colourGreen
titlecompleted

1600

 

1630

sslvpn.ncsa.illinois.edussl cert is refreshed

Users may need to manually reconnect if the system drops their session

neteng@ncsa
radiant-admin@ncsa
.illinois.edu

Status
colourGreen
title

COMPLETED

completed

2021-08-03 9:00 am2021-08-03 11:30 amRadiant ClusterA change was made to the firewall that unintentionally restricted access for instances and other internal cluster communication.Access to instances and workload
02/08/2024 103002/08/2024 1330NCSA Backbone Network Battery BackupNPCF Network DC Battery Maintenance Network Engineering is taking the battery back-up servicing NPCF networking equipment offline for periodic maintenance. This will be non-service impacting, as all core networking equipment still has two independent power feeds. neteng@ncsa
radiant-admin@ncsa
.illinois.edu

Status
colourGreen
title

resolved

completed

2021
2024-
07
01-
31 0600
29 1600
2021
2024-
07
02-
31 0630CILogon hosted servicesInfrastructure maintenanceDuring this time each service hosted by CILogon including COmanage Registry, LDAP, Grouper, SAML proxy, and MDQ will become unavailable for a short time. Each individual service outage will last less than 5 minutes. Services that will not be impacted include: * OIDC clients that do not query LDAP for resolving attributes * X.509 certificate issuance and certificate revocation lists * LIGO and GW-Astronomy serviceshelp@cilogon.org
05UIUC NetworkComplete / partial network outageWhile NCSA network is up and not impacted, much of the UIUC network is currently offline.  This could be affecting a broad range of services such as wireless, facility networks, campus websites, etc.  No current ETA, as engineers are still troubleshooting the problem. neteng@ncsa.illinois.edu

Status
colourGreen
title

COMPLETE2021-07-29 13002021-07-29 1400IRST-run bastion hosts (pool B)Security patchingHosts managed by IRST will be patched and rebooted. Only hosts in pool B will be patched at this time

completed

1730

1830

vCenter Server ApplianceCritical patches are being applied

The vcenter.internal.ncsa.edu site will not be accessible. Operating VMs should not be affected

help@ncsa
help+security@ncsa
.illinois.edu

Status
colourGreen
title

COMPLETE2021-07-29 09002021-07-29 1000IRST-run bastion hosts (pool A)Security patchingHosts managed by IRST will be patched and rebooted. Only hosts in pool A will be patched at this time

completed

0800

1400

HOLL-IHOLL-I will enter a shuttered/standby modeAll HOLL-I servers and services will no longer be available after standby mode is activated.help@ncsa
help+security@ncsa
.illinois.edu

Status
colourGreen
title

COMPLETE

completed

2021

2024-

07

01-

28 1000

24 1800

2021

2024-

07-28 1050LSSTOS Updates on only NCSA Test Stand (NTS)Only the LSST NCSA Test Stand (NTS) services hosted at NCSA

01-25 0700

NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares will be unavailable during maintenance.  Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable.help@ncsa
lsst-admin@ncsa
.illinois.edu
Status
colourGreen
title
COMPLETE2021-07-27 06002021-07-27 0900JiraUpgradeJira will be unavailable
completed


0900

0915

LastPassyearly audit performed. Users disabled or deleted per policyAccounts that had been disabled for over a year were deleted. Accounts that were unused for a year were disabledhelp+security@ncsa
help+serverice@ncsa
.illinois.edu

Status
colourGreen
title

COMPLETE

completed

2021


2024-

07

01-

26 1800

22 0900

2021

2024-

07

01-

27 0000

22 1000

idp.ncsa.illinois.eduConfiguration update for sslvpnConfiguration for the NCSA Identity Provider will be updated with login support for sslvpn.ncsa.illinois.edu.help+idp@ncsa.illinois.edu
NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were unavailable during maintenance.  Users were not able to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable.help+service@ncsa.illinois.edu 

Status
colourGreen
title

complete2021-07-21
08002021-07-21
2900ICCP

ICCP Quarterly Maintenance

  • TBD
All ICCP services

completed

1700

1745

Wiki service upgradeUpgrade version to address recently announce security vulnerabilities.

Wiki will be down during upgrade and testing.

help@ncsa
help@campuscluster
.illinois.edu

Status
colourGreen
title

complete2021-07-21 15:242021-07-21 21:50ASD Vshpere cluster in 3003One of the 4 hypervisors in the cluster paniced.  Unscheduled preventative maintenance is being preformed on it and the other 3 nodes in the cluster.after the initial outage at 15:24, there should be no additional outages.

completed

  2200

  2330

Jira service upgradeUpgrade version to address recently announce security vulnerabilities.

Jira will be down during upgrade and testing.

help@ncsa
help+service@ncsa
.illinois.edu

Status
colourGreen
title

complete

completed

2021
2024-
07
01-
13 0700
17 0500
2021
2024-
07
01-
13 0800cilogon.orgUpdate to OA4MP v5.1.4.The OAuth2/OIDC backend of the CILogon Service will be updated to OA4MP v5.1.4.
17 0700Wireless connectivity on 2nd, 3rd and 4 floors. Tech Services will be replacing some network components in switches that provide connectivity for wireless. 

Each floor (wireless) will lose connectivity for a few mins while the cards are replaced. 

neteng+help@ncsa.illinois.edu
help@cilogon.org

Status
colourGreen
title

complete

completed

2021
2024-
07
01-
08 0800
16 0900
2121
2024-
07-08 1000OpenAFSThe remaining OpenAFS database servers were upgraded.No service impacts were seen
01-16 23:30Facility UPSSecond attempt, Preventive Maintenance_Replace UPS capacitorsAll systems which are connected to UPS power. During the PM the systems will not lose power but will be unprotected.rantissi@illinois.edu
help+service@ncsa.illinois.efu


Status
colourGreen
title

complete

COMPLETED

2021
2024-
07
01-
07 0600
16 2200
2021
2024-
07
01-
07 0800CILogon AWS Hosted ServicesUpgrading AWS RDS Aurora MySQL v5.6 to v5.7COmanage Registry and Grouper services hosted by CILogon will be unavailable
17 0300Waster leak in Node 1 on campus.Node1 ( located on campus) has a water leak that may require full power down to address.  This will take out several devices that provide connectivity to NCSA WAN. No power outage was needed to repair the leakneteng@ncsa.illinois.edu
help@cilogon.org

Status
colourGreen
title

complete

COMPLETED

2021
2024-
07
01-
01

2140

2021-07-01

1430

Horizon dashboard access was down for the entire period. Cluster networking was down from 1200 to1430.Investigations into Horizon  dashboard accessibility issues resulted in the application of an incorrect default network gateway for the cluster around noon. This was corrected and networking functionality restored around 1400. Instances began recovering soon thereafter.Radiant admins believe running instances have recovered on their own but we advise everyone to check their systems and report any issues they see to the help desk.
10 07002024-01-10 12:09NightingaleQuarterly Planned MaintenanceAll Nightingale servers and services were unavailable (other than the ngale-bastion* nodes)help@ncsa.illinois.edu

Status
colourGreen
title

resolved

COMPLETED

2021
2024-
07
01-
01

0247

2021-07-01

1300

Various systems in NPCF, ACB, NCSA

There was a power event in the Champaign-Urbana area at around 2:47AM today. Details about the cause are currently unknown.  This event caused disruptions to systems at the NCSA building, NPCF and ACB. Known issues have generally been resolved but there may be unidentified issues lingering. If you encounter any problems, please notify NCSA help desk staff (help@ncsa.illinois.edu).

Multiple systems/services were impacted. All have been recovered and return to normal operations is complete.NCSA help desk
09 21002024-01-10 0400WifiPerforming a Code upgrade that will affect the Wi-Fi Environment. The majority of the system will be online and functional while individual Access Points will be upgraded. This upgrade is expected to gracefully migrate clients to adjacent access points to minimize any interruption. 

<----


This will more than likely also impact NCSAnet.

neteng+help@ncsa.illinois.edu

Status
colourGreen
title

resolved2021-07-01 02:58 CDT2021-07-01 06:00 CDTACHE and NGALE bastion hostsLoss of power.All ache-* services, ngale bastion hosts

COMPLETED

08 Jan 2024 0700

08 Jan 2024 1600

vforgeRadiant upgradeEntire cluster is shut downjlong@ncsa.illinois.edu
Status
colourGreen
titleCompleted


0700

1430

RadiantThe Radiant cluster was be upgraded from OpenStack Wallaby to Yoga.The web dashboard and API endpoints were unavailable; networking for instances may have been intermittent.help@ncsa.illinois.edu

Status
colourGreen
title

resolved

Completed

2021
2024-
06-29 22:00

2021-06-29 23:59

NCSA 4th Floor Office networkRebooting one or more of the office switches on the NCSA Building 4th floor to resolve a phone issue.Office port connectivity will be intermittent during the maintenance window.

Matt Kollross

help+neteng@ncsa.illinois.edu

01-05 04002024-01-19 1200WifiUpgrading the code used for the Authentication on the Wi-Fi system and VPN. There will be an interruption to the IllinoisNet_Guest device registration and the IllinoisNet_Guest self-registration portal; both are expected to be back online before regular business hours. Regular authentication and traffic flow for the Wi-Fi and VPN is not expected to be interrupted.

<----


This will more than likely also impact NCSAnet.

neteng+help@ncsa
StatuscolourGreentitleresolved2021-06-24
08002021-06-24
1345LSST
  • Updates are being applied on Prod/Stable k8s, rebuild of some ingress nodes
Prod/Stable K8Slsst-admin@ncsa
.illinois.edu

Status
colourGreen
title

resolved

COMPLETED

2021
2023-
06
12-
24
08002021-06-24
1200LSST

LSST Quarterly Maintenance

  • OS updates on all servers

All LSST services hosted at NCSA

EXCEPT Prod/Stable K8S

19  04002023-12-19 0800WifiUpgrading the core campus Wi-Fi hardware. There will be an interruption to Campus Wi-Fi (including IllinoisNet, IllinoisNet_Guest, and eduroam), IllinoisNet_Guest device registration, and the IllinoisNet_Guest self-registration portal.

<----


This will more than likely also impact NCSAnet.

neteng+help@ncsa
lsst-admin@ncsa
.illinois.edu

Status
colourGreen
title

complete

Completed

2021

2023-

06

12-

22 0000

13 1800

2021

2023-

06

12-

22 0400Internet2 WAN linkInternet2 will be migrating NCSA's physical port to their new next generation infrastructure.During the maintenance, our I2 connection will be down.  Traffic will reroute to other connections.  Some point to point connections maybe unavailable for period of time.  The maintenance window is not expected to take all 4 hours.

14 0700

NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares will be unavailable during maintenance.  Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable.help@ncsa

Matt Kollross

help+neteng@ncsa
.illinois.edu
Status
colourGreen
title
complete
Completed
2021-06-21 18002021-06-22 0000NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were unavailable during maintenance.  Users were not able to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable.help+service@ncsa


  1600

1630

wiki.ncsa.illinois.edu and jira.ncsa.illinois.edu
  StatuscolourGreentitlecomplete2021-06-17-07002021-06-17-0820OpenAFSThe OpenAFS database server kaskaskia was upgradedNo service outages were observed or reported.
Atlassian has notified us of several critical security vulnerabilities in Confluence and Jira software. A mitigation has been applied to the Jira server and the Confluence server (wiki.ncsa.illinois.edu) will be patched.There will be a brief outage to patch the Confluence server at 1600. The patching is expected to take 15-20 minutes but the entire hour is reserved as a precaution.help@ncsa.illinois.edu
help+service@ncsa.illinois.exdu

Status
colourGreen
title

complete

Completed

2021
2023-12-06 09002023-12
22002021
-06
-15 1500LSST FirewallThe NPCF secondary firewall was offline due to a hard drive failure.No impact occurred to production services as the primary firewall stayed online.Kay Avila
1700Facility UPSPreventive Maintenance _ Replace UPS capacitors.All systems which are connected to UPS power. During the PM the systems will not lose power but will be unprotected.MO Rantissi

Status
colour

Green

Red
title

RESOLVED2021-06-14 17002021-06-15 0958NCSA GitLabAttempt to fix an authentication bug for a particular user accidentally broke all authentication through the web interface,Authentication through the web interface did not work.

INCOMPLETE

UPS maintenance was halted due to damaged parts.  Putting the UPS back together and rescheduling for a later date.

The UPS is back online.

2023-11-10
0800

2023-11-10
0900

IDDS databasePlanned maintenance: postgresql upgradeNCSA identity, group management, campus cluster user management page, TEM shift report tool, and napshelp+idds@ncsa
help+service@ncsa
.illinois.edu

Status
colourGreen
title

RESOLVED

Completed

2021

2023-

06

11-

11

09 0700

2021

2023-

06-11 0905NCSA JiraJira email problemJira is not accepting issues via email, you can still create issue directly via Jira GUICameron Pitcel

11-09 1445 (vForge)

2023-11-09 1630 (license servers)

vForge / license serversQuarterly Planned Maintenanceall vForge nodes and services (incl. related license servers/services) will be unavailablehelp@ncsa.illinois.edu

Status
colourGreen
title

RESOLVED

Completed

2021

2023-

06

11-

10

09 0700

2021

2023-

06-10 0800cilogon.orgUpdate to OA4MP v5.1.3.The OAuth2/OIDC backend of the CILogon Service will be updated to OA4MP v5.1.3.

11-09 14:30

vForge / license serversQuarterly Planned Maintenanceall vForge nodes and services (incl. related license servers/services) will be unavailablehelp@ncsa.illinois.edu
help@cilogon.org

Status
colourGreen
title

complete

1000

1030

Completed

2023-11-08 1200

2023-11-08 1230

RadiantRebuilding rabbitmq serviceDashboard and API services were read-only during this time. help@ncsa
Jira.ncsa
.illinois.edu
Configuration change to address a vulnerabilityThere should not be any service interruption, but as with all things, it is possible

Status
colourGreen
titleCOMPLETED

2023-11-03

2023-11-07

hub.ncsa.illinois.eduprivate docker registry is down due to volumes in radiant in detaching statehub.ncsa.illinois.edu is not reachable, and images stored are unreachable. Services that have their images local should continue to run, services that want push/pull images will get a 500 error.Rob Kooper 
help+service@ncsa.illinois.edu

Status
colourGreen
title

Resolved2021-06-022021-06-02NetdotNetdot web access now requires 2FA via SSL VPN, or Cerberus proxy. Security requested that Netdot require 2FA, in order to access the web interface.  To accommodate that request, the Netdot firewall has limited web access to the VPN subnet or via proxy from the Cerberus jump hosts. 

COMPLETED

1700

1830

Confluence/WikiUpgrade the systemConfluencehelp@ncsa

Matt Kollross

help+neteng@ncsa
.illinois.edu

Status
colourGreen
title

RESOLVED

COMPLETED

2021
2023-
05-252021-05-26vcenters for ache and ASDemergency security updates were applied.the administrative interface was off-line for about 20 minutes as the updates were installed.
10-31 09:302023-10-31 10:30NCSA OpenSourceupgrade Atlassian productsopensource confluence/jira/bamboo/bitbucket

Status
colourGreen
titleCOMPLETED

 

 

HALFull System PMAll HAL services
help+service@ncsa.illinois.edu

Status
colourGreen
title

RESOLVED

POSTPONED

2021-05-26

1000

2021-05-26

1030

VoIP phones at NPCFMigrating the VoIP networks to a campus IP to enable future migrations by tech services.After the networks are migrated, a reboot all phones at the NPCF building will be performed.


1330

14:30

SSLVPNNew auth method was added to a new login profile, ncsa-vpn-saml-tunnelall. There is now a test profile in place that isn't open to everyone. Please continue to use the profiles you were using before. If you notice and issue please report it. Our testing indicated logins were working as intended. help+neteng@ncsa.

Matt Kollross

neteng+help@ncsa.
illinois.edu

Status
colourGreen
title

RESOLVED

2021-05-21

1800

2021-05-21

1900

VoIP phones at the NCSA buildingMigrating the VoIP networks to a campus IP to enable future migrations by tech services.After the networks are migrated, a reboot all phones at the NCSA building will be performed.

Matt Kollross

neteng+

COMPLETE

0900

1700

RadiantOpenStack software updateThe Radiant team will be conducting an OpenStack software update, from Victoria to Wallaby. This is a software stability update and does not include significant features or changes in functionality. The update will be done online and is not expected to impact running instances or system access.
help@ncsa.illinois.edu

Status
colourGreen
title

RESOLVED2021-05-20 05:402021-05-20 08:45LSST

ESXi host outage causing degradation of select services.

Degradation of select services:

  • data backbone gateway (lsst-dbb-gw01 down)
  • HTCondor (Central Manager nodes down for Prod & DAC)
  • login (lsst-login01 is down)

Also loss of redundancy for some underlying services, including auth/access & k8s head nodes.

COMPLETE

  1700

  0700

NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares will be unavailable during maintenance.  Users will be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing will be unavailable.help@ncsa
lsst-admin@ncsa
.illinois.edu

Status
colourGreen
title

RESOLVED2021-05-15
06002021-05-15
0800CILogon hosted services including COmanage Registry, LDAP, SAML proxy, SAML AA, MDQMaintenanceAll CILogon hosted services were temporarily unavailable.

complete

0930

1100

TaigaOnline, Rolling patch of Taiga serversTaiga File Systemset@ncsa.illinois.edu
help@cilogon.org

Status
colourGreen
title

complete

2021-05-12 07:00

2021-05-12 08:00

COMPLETE

0800

0800

HOLL-ICS-2 Appliance Mode upgradeAll HOLL-I servers and services will be limited to internal testershelp@ncsa
internal.ncsa
.illinois.edu
NCSA Internal Web Server Upgrade
(aka Savannah or MIS Tools)Updates were made that will affect the availability of the NCSA internal website and Savannah system. The system was be unavailable during this time.

help+service@ncsa.illinois.edu

Status
colourGreen
titleCompleted

0800

0800

HOLL-ICS-2 Appliance Mode hardware installationAll HOLL-I servers and services will be unavailablehelp@ncsa.
StatuscolourGreentitlecomplete

2021-05-11

07:00

2021-05-11

19:00

iForgeQuarterly MaintenanceAll systems unavailableiforge-admin@lists.ncsa.
illinois.edu

Status
colourGreen
title

complete2021-05-06 09002021-05-06 0945WAN Link MigrationNCSA Neteng migrated the WAN link to Internet 2 to new hardware.Traffic was automatically re-routed to redundant paths during the link outage. Any connections relying on layer-2 connections over AL2S saw a brief blip as the connection is cut over. Affected parties were contacted in advance.

COMPLETE

1800

1815

ConfluenceConfig will be applied to increase the period users can be logged in before logged outConfluence will be downhelp@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

  1700

  1800

sslvpntesting new auth methodno user impact was observedhelp+neteng@ncsa.illinois.edu

Status
colourGreen
title

complete

COMPLETE

2021
2023-
05
10-
03
06002021-05-03
0630CILogon Multi-tenant COmanage RegistryUpgrade to version 3.3.2The service at https://registry.cilogon.org  was unavailable
10 08002023-10-10 1000cilogon.orgMoving to new compute infrastructurecilogon.org, demo.cilogon.org, crl.cilogon.orghelp@cilogon.org

Status
colourGreen
title

complete

COMPLETE

2021
2023-
04
10-
29 1600
09 1215
2021
2023-
04
10-
29 1700
  • HTCondor Prod
  • HTcondor DAC
Add new nodes into Condor service pools
  • HTCondor Prod
  • HTcondor DAC
09 1532TaigaAppliance has unmounted all of its OSTs. Ability to do I/O to Taigaset@ncsa
lsst-admin@ncsa
.illinois.edu

COMPLETE

2021-04-21 08:002021-04-21 20:00ICCPICCP Quarterly MaintenanceThe scheduler will be down.  All compute nodes will be converted to rhel7.9 with RedHat IB.

iccp-admins@campuscluster.illinois.edu

COMPLETE

2021-04-15 16002021-04-15 1700NCSA OpensourceUpgrade of OS on all machines related to opensourcejira, wiki, git etc hosted at https://opensource.ncsa.illinois.edu/kooper@illinois.edu

COMPLETE

2021-04-15

12:25

2021-04-15

14:45

ICI vmware

Several hosts on the vmware service were experiencing timeouts

  • bluewaters
  • bluewaters-test
  • internal
  • its-nagios
  • ldap1
  • vcenter
no or intermittent connectivity to these hostshelp+service@ncsa.illinois.edu

RESOLVED

Root cause is still being investigated.

2021-04-15
09002021-04-15
0942CMDBApplying new certificates and restarting servicesCMDB, including web interface, will be down briefly during the update.ncsagroup+org_itsm@ncsa.illinois.edu

RESOLVED

2021-04-15 09002021-04-15 0920WAN Link MigrationNCSA Neteng will migrated the WAN link to ESnet to new hardware.Traffic was automatically re-routed to redundant paths during the link outage.help+neteng@ncsa.illinois.edu

RESOLVED

2021-04-14 15:002021-04-14 15:00git.ncsa.illinois.eduUsers can no longer access repositories from git clients over HTTPS using their NCSA password.

NCSA passwords can not access repositories with Git clients. Instead use ssh keys over SSH or personal access tokens over HTTPS.

We thought this went into effect during git changes on Nov 2, 2020 but discovered it was still working until we made changes to GitLab to fully remove LDAP functionality.

help+service@ncsa.illinois.edu

COMPLETE

2021-04-13 14152021-04-13 1845git.ncsa.illinois.eduThe GitLab website at git.ncsa.illinois.edu was having issues with authentication. The LDAP server that it uses was timing out.
  • Login to the Git web interface was timing out.
  • Access from git clients continued to work during the outage.
help+service@ncsa.illinois.edu

RESOLVED

2021-04-13 0800

2021-04-13 0830

cilogon.orgUpdate to OA4MP v5.1.1.The OAuth2/OIDC backend of the CILogon Service will be updated to OA4MP v5.1.1.help@cilogon.org

COMPLETE

2021-04-12 18002021-04-12 2245File & Print ServersMonthly Windows File & Print Server MaintenanceWindows File Shares such as HR, Business Office, Home, etc. and printing in the NCSA & NPCF buildings were unavailable.help+service@ncsa.illinois.edu 

COMPLETE

2021-04-10
06002021-04-10
0800CILogon hosted COmanage, Grouper, SATOSA, LDAPOn Saturday, April 10, the CILogon team will perform maintenance on the infrastructure used for hosted services.As part of the maintenance all COmanage Registry, LDAP, Grouper, SAML proxy, SAML attribute authority, and MDQ services hosted by CILogon may experience brief outages. We do not expect that any specific service outage will last for more than a minute.help@cilogon.org

COMPLETE

2021-04-08 09002021-04-08 1045WAN Link MigrationNCSA Neteng migrated the WAN link to ICCN Node-1 to new hardware.Traffic was automatically re-routed to redundant paths during the link outage. Issues were noticed by users during the outage and are currently being investigated in cooperation with our upstream provider.help+neteng@ncsa.illinois.edu

COMPLETE

2021-04-08 07302021-04-08 0734NCSA WikiNCSA's Wiki service was restartedNCSA's Wiki service was restarted to apply a new SSL certificate and renewed Confluence license. The wiki was not available for 4 minutes while it reloaded.help+service@ncsa.illinois.edu 

COMPLETE

2021-04-07 1610

2021-04-07 1733Internal Savannah/MIS websiteThe Savannah/MIS website would not load due to a corrupted MySQL database table referenced across all of the Savannah tools.Internal/Savannahhelp+service@ncsa.illinois.edu

RESOLVED

1st report 7:30am Monday8:19am MondayNCSA LDAP2ldap2 is not responsive to authentication requestsNCSA Jira, any systems using LDAP2 as its only source.help+service@ncsa.illinois.edu

RESOLVED

2021-03-30

0800

2021-03-30

0845

DNS1A software issue was causing BIND to fail. DNS was not able to resolve during the period of time.  DNS2 remained operational. neteng+help@ncsa.illinois.edu

RESOLVED

2021-03-23

2000

2021-03-23

2025

NCSA VPNThe standby VPN hardware was replaced and transitioned into the current VPN cluster. Failover went as expected and firmware was upgraded on the primary after load was shifted to the new standby VPN.Failover between the appliances occurred without issue and there was no impact to users.neteng@ncsa.illinois.edu

RESOLVED

2021-03-18 12301255JiraSome functionality will be limited due to user limit being reachedJirahelp@service@ncsa.illinois.edu

RESOLVED

~16:4017:58AnyConnect VPN Service

An issue with SSL on the VPN service has caused an issue that has disconnected all users. Network engineering is looking into the issue.

Due to a hardware failure and the VPN not failing over properly to the standby users were unable to connect to the VPN. This was due to an issue with syncing certificates.

During the outage, expect that you won't be able to connect/maintain a connection to the VPNhelp+neteng@ncsa.illinois.edu

RESOLVED

2021-03-16 09502021-03-16 1000CMDBWill be applying updates per security vettingCMDB, including web interface, will be down briefly during the update.ncsagroup+org_itsm@ncsa.illinois.edu

RESOLVED

2021-03-11
0900

2021-03-11
0930

WAN Link MigrationNCSA Neteng migrated the link to ICCN to new hardware.Traffic was automatically re-routed to redundant paths during the link outage.help+neteng@ncsa.illinois.edu

RESOLVED

2021-03-04
0900

2021-03-04
0905

WAN Link MigrationNCSA Neteng migrated the 100G link to MREN to new hardware.Traffic was automatically re-routed to redundant paths during the link outage.help+neteng@ncsa.illinois.edu

RESOLVED

2021-03-01 22:112021-03-01 22:47NCSA vSphereAbout 40 VMs lost connection to their NFS storage.Several VM-based services were timing out during the issue, including: vSphere management, a kerberos replica, a ldap replica, httpproxy, license servers, NCSA fileserver, Identity message queuing, monitoring. That triggered some of those VMs to switch to use read-only disk, needing to be rebooted later.service@ncsa.illinois.edu

RESOLVED

...

Status
colourGreen
titleCOMPLETE

0600

0700

ConfluenceConfluence is being upgradeConfluence will not be available for use

help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-10-04 06002023-10-04 1927DeltaFilesystem and OS patchingAll Delta resources will be unavailable during the maintenance period
including:
+ Delta login nodes - unavailable
+ Delta compute nodes - unavailable
Delta services
+ Open OnDemand - unavailable
+ Delta Globus Online endpoint - unavailable
help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-10-04 09502023-10-04 1000Opensource ConfluencePatching conflueceopensource confluence will be down

Status
colourGreen
titleCOMPLETE

2023-09-27 17002023-09-28 0700NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were unavailable during maintenance.  Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable.help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-09-26 14002023-09-26 1500Wireless NCSA buildingCampus wireless outage.NCSAnet and IllinoisNet users are experiencing connectivity issues.   Tech Services is aware of the problem. help+neteng@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-09-28 14332023-09-28 1459cilogon.orgservice outage due to AWS database issuelogins to cilogon.org were failinghelp+cilogon@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-09-26 7:30AM2023-09-26 8:00AMLdap Primary  ServerMaintenanceLdap updates will be disabled during maintenanceTimothy Bouvet 

Status
colourGreen
titleCOMPLETE

09/22/2023 8:00am9/22/2023 1:30pmWireless accessNCSANet is not authenticating users and denying connections.Anyone attempting to connect to the wireless NCSANet ID.neteng@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-08-292023-09-21 - 1300opensource bitbucketBitbucket is not compatible with the deployed version of git, see https://jira.atlassian.com/browse/BSERV-14390opensource.ncsa.illinois.edu/bitbcket

Status
colourGreen
titleCOMPLETE

2023-Sep-19 - 07452023-Sep-19 - 0750LastPassRekey the LastPass/Duo IntegrationLastPass users that utilize duo may not be able to authenticate until completedJames Eyrich

Status
colourGreen
titleCOMPLETE

2023-Sep-18 - 15112023-Sep-18 - 1749TaigaOutage due to failed MDS failover.Taiga access was unavailable.set@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-09-14 07002023-09-14 2015vForge / license serversQuarterly Planned Maintenanceall vForge nodes and services (and related license servers/services) will be unavailablehelp@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-09-14-08002023-09-14-2000Taiga & Granite ServicesSemi-Annual Planned MaintenanceAll Taiga and Granite services will be offline 

set@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-09-08  06:582023-09-08 09:50disruption to NPCF-DES-CORE, NPCF-CWMGMT-FW1 & 2, MForge VPN

NPCF-CORE-EAST has a DEAD linecard.

Relocating affected links to other linecards with open ports while we work with vendor support for a replacement.

redundancy has been lost, access and activity remain normal.

Status
colourGreen
titleCOMPLETE

2023-09-06 09:482023-9-6 10:35NCSA Center Wide Management Networkthe firewall protecting this network is showing offlineCenterwide management networks in NCSA building(John) Walker 

Status
colourGreen
titleCOMPLETE


2023-9-6 09:502023-9-6 10:35The main switch in NCSA 3003In debugging a link problem between NCSA and NPCF the wrong fiber was inadvertently pulledNetworking in and out of 3003 was down for 35 minsneteng@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-08-26 23:062023-08-27 01:36CILogonCILogon database replication errorCILogon OAuth/OIDC services unavailablehelp+cilogon@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-08-24 13:022023-08-25 14:15TaigaMultiple SAS cable backend failure causing OSTs to go into write protect and unmountAccess to certain OSTs in Taiga

set@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-08-16 17002023-08-17 0000NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were unavailable during maintenance.  Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable.help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-08-16 07002023-08-16 1003NightingaleQuarterly Planned MaintenanceAll Nightingale servers and services were unavailable (other than the ngale-bastion* nodes)help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2023-08-15 04002023-08-15 0500VMWare GatewayVMWare is updating the Gateway OSNo expected effects

help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-08-14 09352023-08-14 2000NCSA VPNDuo implemented new ssl checks that we were not passingUsers couldn't authenticate with DUO to establish new connections to the VPN. Existing VPN sessions remain connected.Matthew Elliott 

Status
colourGreen
titleComplete

2023-07-27 17002023-07-27 1715HOLL-ILive kernel patchingkernel was updated in response to recent security issue.help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-07-25 09002023-07-25 1630RadiantChanges to the OpenStack network configuration and network service node (increasing MTU on customer networks and adding a new dedicated network server)These changes will impact project/instance networks and cause them to be unreachable for an extended period of time. Expect network timeouts and failure of NFS file system access. Systems may be unreachable for several hours - up to the entire planned time - but we are making every effort to minimze the downtime.

James Glasgow via help@ncsa.illlinois.edu

Status
colourGreen
titleComplete

2023-07-19 0800

2023-07-19 2000

ICCPICCP Quarterly MaintenanceAll ICCP services

help@campuscluster.illinois.edu

Status
colourGreen
titleComplete

2023-07-19 08002023-07-19 0900u1carne routerscheduled maintenancemForge, Magnus, and Access will have a brief outage as the routers reboot.

Michael Douglas via neteng@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-07-12 17002023-07-12 2130NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were unavailable during maintenance.  Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable.help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-07-07 10002023-07-07 1100NCSA KerberosDeleting out principals that were disabled on 2023-06-07Kerberos authentication should already be disabled for the planned hosts, so there should be zero notable effect. help@ncsa.illinois.edu

Status
colourGreen
titleComplete

1345

1450

HOLL-ITransitioning CS-2 Execution Mode from Weight Streaming to PipelinedHoll-I CS-2help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-06-30 06002023-06-30 0605NCSA GitLabGitLab was updated to latest versionAll GitLab services was unavailable for a few minutes.help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-06-29 18492023-06-29 2000DeltaMore Power fluctuations due to the severe weather have caused in all NCSA buildings. NCSA staff are working to restore all services to full functionality.Delta Login, Openondemand and Scheduling.help@ncsa.illinois.edu

Status
colourGreen
titleRESOLVEd

2023-06-29 13162023-06-29 1530Most NCSA computer systemsPower fluctuations due to the severe weather have caused multiple system failures in all NCSA buildings. NCSA staff are working to restore all services to full functionality.

Virtually all systems have been impacted to some extent.

Most NCSA compute resources have returned to service.

help@ncsa.illinois.edu

Status
colourGreen
titleRESOLVEd

2023-06-21 18002023-06-21 1900DNS1 / DNS2 BIND security patchesDue to a security issue with BIND, neteng will be rebooting both DNS servers (staggered) starting tonight at 1800.  neteng@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-06-15 06002023-06-15 0605NCSA GitLabGitLab updated to use new backup methodAll GitLab services were unavailable for a few minutes.help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-06-14 17002023-06-14 2200NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were unavailable during maintenance.  Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable.help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-06-08 11002023-06-08 1205ICI MetricsMajor Upgrade to Grafana 9.5.x and Unified AlertingAccess to https://metrics.ncsa.illinois.edu and all alerting was pausedmalone12@illinois.edu

Status
colourGreen
titleComplete

2023-06-07 13002023-06-07 1500NCSA KerberosDisabling Kerberos Host Principals not in DNSKerberos Authentication for hosts may stop working. Please create a ticket if you think your host principal may have been disabled erroneously. help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-05-23 12402023-05-23 1410TaigaFailover events on tgio02I/O to and from Taiga for all services intermittently during this periodset@ncsa.illinois.edu

Status
colourGreen
titleRESOLVEd

2023-05-23 08002023-05-23 1400Granite Tape ArchiveUnplanned Library Maintenance due to component failureRetrieval of data;bdickin2@illinois.edu

Status
colourGreen
titleRESOLVEd

2023-05-18 08002023-05-18 1400Granite Tape ArchiveLibrary Preventative MaintenanceRetrieval of data;bdickin2@illinois.edu

Status
colourGreen
titleCOMPLETE

2023-05-17 07002023-05-17 2125NightingaleQuarterly Planned Maintenance

All Nightingale servers and services were unavailable (other than the ngale-bastion* nodes)

help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-05-17 05302023-05-17 0600Wireless and VoIP (NCSA Building)Router UpgradesWireless, VoIP and anything directly connected to the campus switches will be down, while they upgrade firmware on the router.help+neteng@ncsa.illinois.edu

Status
colourBlue
titleSCHEDULED

2023-05-16 07002023-05-16 1100HOLL-IQuarterly Planned Maintenanceall HOLL-I nodes and services were unavailablehelp@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-05-15 09002023-05-15 1200HOLL-ICS-2 CDU maintenancethe HOLL-I CS-2 was unavailable and there was a reservation in Slurmhelp@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-05-12 06002023-05-12 0615NCSA GitLabGitLab was updated to latest versionAll GitLab services was unavailable for a few minutes.help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-05-11 0700

2023-05-11 1900

vForge / license serversQuarterly Planned Maintenance

all vForge nodes and services (and related license servers/services) will be unavailable

help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-May-03 08002023-April-25 0900ACHE FW Cluster Upgrade - SecondaryUpgrading ACHE Firewall member BNo outage expectedeyrich@illinois.edu

Status
colourGreen
titleCOMPLETE

13:00

16:30

vSphere and hosts on it.VMWare Licensing issues. Was forced to migrate to new vSphere.LDAP, Wordpress Sites, varioushelp@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2023-05-01
1530
2023-05-01
1605
ICI VMwareApply updates to address software issue.


Expand
titleVMs that could experience a short outage...

campuscluster
congo-vm.ncsa.uiuc.edu
internal-test
ldap-mg1.ncsa.illinois.edu
ldap-radiant1
ldap1.delta
ldap2
ldap2.ngale.ncsa.illinois.edu
ldap3.ncsa.illinois.edu
manage - uillinois.edu AD
metrics01
metrics02
midwestbigdatahub.org
rad-adm01
studentcluster.ncsa.illinois.edu
svna-build
tintri-global-center
asd-backup01.internal.ncsa.edu
asd-log.internal.ncsa.edu
asd-pup01.internal.ncsa.edu
hli-pup01.internal.ncsa.edu
jlongtest
mlong-agent1
oncall-test


aloftus@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2023-April-26 08002023-April-25 0900ACHE FW Cluster Upgrade - primaryUpgrading ACHE Firewall member ANo outage expectedeyrich@illinois.edu

Status
colourGreen
titleCOMPLETE

2023-April-25 08002023-April-25 0900NPCF CWFM Cluster Upgrade secondaryUpgrading NPCF CW Firewall member BNo outage expectedeyrich@illinois.edu

Status
colourGreen
titleCOMPLETE

 

1700

 

1715

NCSA VPNThe certificate on the NCSA VPN was replaced.Users will be disconnected from the VPN and have to manually reconnect.neteng@ncsa.illinois.edu 

Status
colourGreen
titleCOMPLETE








2023-April-20 08002023-April-20 0900NPCF CWFM Cluster Upgrade primaryUpgrading NPCF CW Firewall member ANo outage expectedeyrich@illinois.edu
Status
colourGreen
titleComplete


2023-04-19 18002023-04-19 2300NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were unavailable during maintenance.  Users were be unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing were unavailable.help@ncsa.illinois.edu

Status
colourGreen
titleComplete

  0800

  2000

DeltaHSN and OS is being updated,The entire system will be offline.kingda@illinois.edu

Status
colourGreen
titleComplete

04/17/23 090004/17/23 1400Granite Tape ArchiveUpgrades to FSIngest or retrieval of data;bdickin2@illinois.edu

Status
colourGreen
titleComplete

2023-03-27


2023-04-16NCSA OpenSource BitBucketincompatibility with git, only versions that can be installed are 2.25 or 2.40, and Bitbucket requires version 2.31 - 2.39
https://opensource.ncsa.illinois.edu/bitbucket is down until new version of BitBucket

Rob Kooper 

Status
colourGreen
titleCOMPLETED

2023-04-032023-04-04NCSAnet, IllinoisNet, EDUroamTech Services is deploying a new certificate for all wireless networks.Check #announce on NCSA Slack for more information, including links to download software that will update your wireless profiles. help+neteng@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-03-29 12:00 CDT2023-03-29 12:30 CDTPrimary Kerberos serverConfiguration changes to match secondary KDCsPassword changes may have been delayed by ten minutesChristopher Lindsey 

Status
colourGreen
titleComplete

2023-03-23 08432023-03-23 1030DHCP serving NCSAnet wireless and NCSA office wired wall jacksThe main NCSA DHCP server stopped answering queries and was restartedIf you didn't already have a DHCP lease your system would have been unable to connect to NCSAnet or register on an office wired wall jack.neteng@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-03-15 18002023-03-16 2300NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were unavailable during maintenance.  Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable.help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-03-14 11002023-03-14 1150Authentication to vsphere.ncsa.illinois.edu and ache-vcenter will failReplacing SSL certs on Ldap1/2Ldap will be restarted on Ldap1/2tbouvet@illinois.edu

Status
colourGreen
titleComplete

2023-03-09 0700

2023-03-09 17:20

vForge / license serversQuarterly Planned Maintenance

all nodes and services will be unavailable

help@ncsa.illinois.edu

Status
colourGreen
titleComplete


03/09/2023 0800

03/09/2023 1713NCSA Taiga & GraniteTaiga Service Node Updates & Granite UpgradeTaiga Public LNET router was upgraded and a second one added; access via public LNET was down from 0800 to 1100.  Globus and NFS services were patched in a rolling/online fashion.  

Granite experienced a short full downtime as we upgraded its software.  
set@ncsa.illinois.edu 

Status
colourGreen
titleComplete

03/07/2023 8:30am03/07/2023 10:15amDelta HSNThe HSN was dropping nodes and not allowing nodes to reconnectHigh Speed Connectivityhelp@ncsa.illinois.edu
Status
colourGreen
titleComplete
2023-03-01: 1100

2023-03-01: 1115

Radiant OpenStack ServicesChanges to the OpenStack controller node to address networking performance issues

All OpenStack services were restarted to effect system configuration changes. The work was completed successfully and all services are available again

help@ncsa.illinois.edu

Status
colourGreen
titleComplete

2023-02-25

2023-02-27 0930

NCSA email A mail loop caused routing and processing problems.Mail routing and delivery was blocked.help@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2023-02-21 0700

2023-02-21 1640

HOLL-IQuarterly Planned Maintenanceall nodes and services will be unavailablehelp@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2023-02-16 ~14:152023-02-16 ~14:25cerberus4mis-configuration caused roughly 50% of connections to be dropped50% of connections in and out droppedhelp+security@ncsa

Status
colourGreen
titlecomplete

2023-02-10 09102023-02-10 0915users.ncsa.illinois.edu web siterestarting the systemno web pages from users.ncsa.illinois.edu will be available help@ncsa.illinois.edu

Status
colourGreen
titlecomplete

02/08/2023 180002/09/2023 0000NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were unavailable during maintenance.  Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable.help@ncsa.illinois.edu

Status
colourGreen
titlecomplete

1215

1230

JiraJira will be restarted to fix stuck notification emails.Jira will unavailable during this time.

Andrew Loftus 

Also posted to #announce (Slack)

Status
colourGreen
titlecomplete

 11:18

15:02

ICCP head node login and golub compute resourcesLost network connectivity for golub infrastructureICCP head node logins (ie cc-login.campuscluster.illinois.edu) and golub compute resources

help@campuscluster.illinois.edu

Status
colourGreen
titleresolved

1200

1300

JiraJira offline for service restart to fix stuck emails.Jira will unavailable during this time.help@ncsa.illinois.edu

Status
colourGreen
titleCompleted

01/25/2023 0800

01/25/2023 0830

NCSA LDAProlling LDAP restarts of redundant servers to deploy new schema file

Minimal impact for service restarts

Status
colourGreen
titleCompleted

2023-01-19 13102023-01-19 1330JiraJira offline for reboot to fix Boards.Jira will unavailable during this time.help@ncsa.illinois.edu

Status
subtletrue
colourGreen
titleComplete

  0800

1700

ICCPICCP Quarterly MaintenanceAll ICCP services

help@campuscluster.illinois.edu

Status
colourGreen
titleresolved

2023-01-13 12002023-01-13 1230JiraJira offline for dashboard fixes.Jira will unavailable during this time.help@ncsa.illinois.edu

Status
subtletrue
colourGreen
titlecompleted

2023-01-12 08002023-01-13 1230JiraMinor issues noticed in Jira likely caused by the upgrade yesterday evening. Gadgets and dashboards are having issues.

Status
subtletrue
colourGreen
titleresolved

2023-01-11 07002023-01-12 1200NightingaleQuarterly Planned Maintenance

All Nightingale servers and services will be unavailable (other than the ngale-bastion* nodes)

Maintenance has been extended until noon Thu, Jan 12 due to complications with firmware update on the Lustre storage appliance.

help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETED

2023-01-12 07002023-01-12 0715NCSA VPN Router MigrationThe NCSA VPN was migrated to a different upstream router. Users were briefly disconnected. help+neteng@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETED

  0600

  0615

NCSA GitLabGitLab upgradeAll GitLab services were unavailable for a few minutes while it upgraded to the latest version.

help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETED

06:40 1/9/20232023-01-11 2100vSphere in 3003One of the storage appliances serving vsphere.ncsa.uiuc.edu started access issues. This has caused issues with 19vms.

crashplan has returned to service

help@ncsa.illinois.edu
Status
colourGreen
titleCOMPLETED


2023-01-11 17302023-01-11 1915Jira Jira software upgradeJira will be unavailable while software upgrades are applied.help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETED

1/9/2023 6:40am1/11/2023 variousvSphere in 3003One of the storage appliances serving vsphere.ncsa.uiuc.edu had access issues.  Data was moved to different storage for affected VMs.digitalag.ncsa.illinois.edu, gecat, reu.ncsa.illinois.edu, ACIpartnership.org, astro, edream, caiiwp, brainstormhpcd.org, internal-dev, cmdb-dev-kimber7, reu-international.ncsa.illinois.edu, avl-test, mharp - ergo, infews-er.net, ncsa30, bluewaters - 2018-03-05.help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETED

12/23/2022 6:30pm12/27/2022 1:30pmTaigaSingle OST is failing to re-mount following failoverFile system is unavailableset@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETED

 0530

 0600

Wireless at NCSA building.Router UpgradeTech Services will be upgrading their NCSA building router which will effect wireless at the NCSA building.  Downtime will be estimated at 15 mins. help+neteng@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETED

0800

1200

RadiantSystem maintenance

OpenStack:

  • "Minor system configuration changes will be made to increase system logging and optimize memory usage/allocation across nodes. No noticeable impact to end users is expected."

Networking:
  • Swap fiber links to correct issue with security taps: In order to minimize user impact, we will swap one link at a time. User should see no impact however there is a slight possibility of a temporary network outage potentially lasting a few minutes however we currently do not anticipate this happening.
  • Update Ethernet switch firmware: Switch reboots will be done in a rolling fashion and so are not expected to be disruptive to ongoing operations (due to switch/path redundancy).
help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETED

15 Dec 2022 090015 Dec 2022 0935NCSA KerberosNCSA's Read-Write KDC is being upgradedPassword changes and new accounts are being queued for completion after the upgrade.help@ncsa.illnois.edu

Status
colourGreen
titleCOMPLETED

  0600

  0615

NCSA GitLabGitLab was upgraded to latest versionAll GitLab services was unavailable for a few minutes.

help@ncsa.illinois.edu

Status
colourGreen
titlecompleted

12-01-2022 060012-01-2022 0700NCSA VPNSoftware UpgradesThe appliances hosting the NCSA VPN were patched. Users experienced a brief disconnect as load is failed over between the appliances. The anyconnect client was upgraded at this timeneteng@ncsa.illinois.edu

Status
colourGreen
titleResolved

1130

1400

NCSA identity password resetsThe password reset process is not completing.Users password resets were queued and then applied when the issue was fixed.  Users who tried to change their password should find there password is now set to the password of their last attempt.help@ncsa.illinois.edu

Status
colourGreen
titleresolved

 

 

capnjack (license server)Changes to IPTABLESUnknown servers. Licenses affected are IDL, PGI, Intel, MATLAB, Abaqus, Sention LM, Luda, Ansys, CDL, Adaptive, Converge, CFD, RLM Type, rr_ld

meberger@illinois.edu

re: SVCPLAN-1465

Status
colourGreen
titleCompleted

2022-11-16 10422022-11-16 1351CILogonDocker Swarm failureCILogon services were unavailable. See: https://cilogon.statuspage.io/incidents/2blf564965s0help@cilogon.org

Status
colourGreen
titleResolved

2022-11-15 07002022-11-15 1700HOLL-IQuarterly Planned Maintenanceall nodes and services will be unavailablehelp@ncsa.illinois.edu

Status
colourGreen
titlecompleted

2022-11-10 0700

2022-11-10 1200

vForge / license serversQuarterly Planned Maintenance

all nodes and services will be unavailable

help@ncsa.illinois.edu

Status
colourGreen
titleCompleted

2022-11-10 11:002022-11-10 11:50ASD Vsphere, specifically vm's using the tintri storage appliance.Network connections were upgraded to 25G speed.There was no disruption of service with this work.help@ncsa.illinois.edu

Status
colourGreen
titlecompleted

2022-11-07 09002022-11-07 0958set-analytics.ncsa.illinois.eduPhysical Machine Move from 3003 to NPCFThe SET managed Grafana/InfluxDB instance will be unavailableset@ncsa.illinois.edu

Status
colourGreen
titlecompleted

2022-11-04 19002022-11-04 1930SET TaigaSET  caused a failover of tgio02 and then failed back.  This fixed the mounting issue.Clients with taiga currently mounted may experience slow or stopped IO during the failover.  Failover completed properly and solved the mounting issue.set@ncsa.illinois.edu

Status
colourGreen
titlecompleted

2022-11-03 11322022-11-04 1930Delta

Taiga filesystem (/taiga/ and /projects/) problem on dt-login01 and dt-login02

The issue is limited to dt-login01 and dt-login02. Commands attempting to access /taiga/ or /projects/ on these nodes will hang.

Users are advised to use dt-login03 or the login.delta.ncsa.illinois.edu "round robin" address

UPDATE: dt-login01 and dt-login02 are fully functional again and back in the login.delta.ncsa.illinois.edu DNS "round robin".

help@ncsa.illinois.edu

Status
colourGreen
titlecompleted

2022-11-03 00482022-11-03 0106SET Taigatgio02 and tgio04 failed overOSTs on the two nodes were inaccessible until the reboots were complete. This is a known issue with a vendor patch in progress. set@ncsa.illinois.edu
Status
colourGreen
titlecompleted


2022-11-03 06002022-11-03 0615NCSA GitLabGitLab was upgraded to latest versionAll GitLab services was unavailable for a few minutes.

help@ncsa.illinois.edu

Status
colourGreen
titlecompleted

2022-11-02 17002022-11-02 2000DNS ServicesPatching for out of cycle security updates.DNS1 and DNS2 will be patched and rebooted (staggered) to applied needed updates.help+neteng@ncsa.illinois.edu

Status
colourGreen
titlecompleted

2022-11-01 18002022-11-02 0000NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were unavailable during maintenance.  Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable.help@ncsa.illinois.edu

Status
colourGreen
titlecompleted

0800

0830

idp.ncsa.illinois.eduEnable Duo Universal PromptNCSA Identity Provider will now use Duo Universal Prompthelp+idp@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETED

2022-10-25 08002022-10-25 0900NCSA building 1st Floor Wifi / Security CamerasTech Services is replacing a networking switch on the 1st for of the NCSA building that powers the Access Points on the first floor.This should be a short down time, but the access points will reboot while we migrate cables to the new switch.  help+neteng@ncsa.illinois.edu

Status
colourGreen
titlecompleted

2022-10-19 07:002022-10-20 07:15Some SSH Bastion HostsOut-of-Cycle reboot needed after failed patching.
Will reboot tomorrow at 07:00am
bwbh1.ncsa.illinois.edu
bwbh3.ncsa.illinois.edu
cerberus1.ncsa.illinois.edu
cerberus3.ncsa.illinois.edu
ache-bastion-1.ncsa.illinois.edu
ngale-bastion-1.ncsa.illinois.edu
help+security@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETED

2022-10-18 15:002022-10-18 15:30Radiant instance creation/managementsystem setting changesNo noticeable impactpl@illinois.edu

Status
colourGreen
titlecomplete

2022-10-18 12:002022-10-18 12:05identity, email to NCSA addressessystem updates 1 minute window to cause email delays and identity frontend unavailablecpl@illinois.edu

Status
colourGreen
titlecomplete

2022-10-14 22002022-10-15NCSA office firewall upgradeUpgrading code on the office firewall.Office networks will be offline during this upgrade.help+neteng@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2022-10-13 17002022-10-13 1800SSLVPN MaintenanceThe second member of the HA pair will be put back into service.The second member was added with no outage.help+neteng@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2022-10-12 11:002022-10-12 12:00ASD Vsphere, specifically vm's using the tintri storage appliance.Network connection on tintri storage box were switch to new hardware but their speed was unchanged. Additional work will need to be scheduled to complete the speed increase.This had no service impact.help@ncsa.illinois.edu

Status
colourYellow
titleINCOMPLETE

2022-10-10 00002022-10-10 1040NCSA VPNThe NCSA VPN had a member of the HA pair fail and licensing didn't fail over. Users were unable to connect to the VPN until the licensing issue was resolved.help+neteng@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2022-10-03 08002022-10-03 0845HOLL-Iinstall security updates and reboot
help@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2022-09-30 06002022-09-30 0615NCSA GitLabGitLab was upgraded to latest versionAll GitLab services were unavailable for a few minutes.

help@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2022-09-27 11002022-09-28 1700odd numbered bastion hosts (cerberus1, cerberus3, ache-bastion-1, ngale-bastion-1, etc.)puppet code refactoring for SSH configs

More changes were pushed out around 5p on 2022-09-28 and we believe the SSHD config issues are resolved.  You can use the even numbered (cerberus2, cerebrus4) bastions as a work-around if any issues persist.

help+security@ncsa.illinois.edu
Status
colourGreen
titleResolved


2022-09-28 09302022-09-28 1050Jira outgoing emailoutgoing email degradedJira failed to send some/most outgoing email during this time frame.help@ncsa.illinois.edu

Status
colourGreen
titleresolved

2022-09-24 14452022-09-25 1045GraniteBuilding power outage caused Disk Storage Unit to power cycleAny user operations on the cluster were interrupted and unavailable until resolution.bdickin2@illinois.edu

Status
colourGreen
titlecomplete

2022-09-21 08002022-09-21 0930HOLL-IChange CS-2 execution mode to PipelinedExecution mode of the CS-2 was changed from Weight Streaming to Piplined.help@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2022-09-08 08002022-09-10 1100GraniteGranite Bi-annual Maintenance (now back in service)Any ingest or retrieval to/from the Archivebdickin2@illinois.edu  slack-id: briandi
set@ncsa.illinois.edu 

Status
colourGreen
titlecomplete

2022-09-09 09432022-09-09 1457Jiraoutgoing email degradedJira failed to send some/most outgoing email during this time frame.help@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2022-09-08
0700

2022-09-08 1010: license servers

2022-09-09
0230: vForge

vForge / license serversQuarterly Planned Maintenance

all nodes and services will be unavailable

help@ncsa.illinois.edu

Status
colourGreen
titlecomplete

 

0500

0600

ASD VM services netRouting in the switch stacks is being swiched from NCSA 3003 to NPCFAll systems on the 141.142.192.x network will be unreachable for up to 5 minutes.help@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2022-08-31 18002022-09-01 0700NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were be unavailable during maintenance.  Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable.help@ncsa.illinois.edu

Status
colourGreen
titlecomplete

1730

1830

JiraJira service will be restartedJira will not be availablehelp@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

08-24-22 183008-26-22 0800Granite Tape ArchiveFS crash and lockupA few files that were transferred into the archive shortly before the crash needed to be re-transferred.

bdickin2@illinois.edu  slack-id: briandi
set@ncsa.illinois.edu 


Status
colourGreen
titleCOMPLETE

2022-08-17
1200
n/aAll LSST hosts at NCSAServers will be shutoff and retired.All LSST servers and services at NCSA.lsst-admin@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2022-08-17 07002022-08-17 1320NightingaleQuarterly Planned MaintenanceAll Nightingale servers and services will be unavailable (other than the ngale-bastion* nodes)help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2022-08-16 0700

2022-08-17 1305

HOLL-IQuarterly Planned Maintenance

All HOLL-I servers and services will be unavailable

2022-08-16 1505 - HOLL-I cluster return to service, but CS-2 remains offline for further work; CS-2 expected return to service by 2022-08-17 1000

2022-08-17 1305 - HOLL-I CS-2 is returned to service

help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2022-08-09 20002022-08-09 2300Office Networks on 2nd FloorCode updates on office network switches.Office ports will be offline as switches reboot. help+neteng@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2022-08-10 20002022-08-10 2300Office Networks on 3rd FloorCode updates on office network switches.Office ports will be offline as switches reboot. help+neteng@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2022-08-11 20002022-08-11 2300Office Networks on 4th FloorCode updates on office network switches.Office ports will be offline as switches reboot. help+neteng@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2022-08-03
0900
2022-08-03
1000
NPCF Center-wide management firewallsSecondary firewall will be upgradedNo impact to services is anticipated.  Traffic will flow normally through the primary firewall as the secondary is upgraded.help+security@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2022-07-27
0940
2022-07-28
15:36
ACHE, Nightingale Several accounts have had their Covered Entity status revokedAffected users/accounts will not be able to access resources that requires Covered Entity enrollment 

help+hippa@ncsa.illinois.edu

Status
colourGreen
titleresolved

2022-07-27
0900
2022-07-27
1000
NPCF Center-wide management firewallsPrimary firewall will be upgradedNo impact to services is anticipated.  Traffic will be failed over to the secondary firewall, the primary will be updated, and then traffic will be moved back to the primary.help+security@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

0900

0915

JiraAdditional LDAP group will be added for exclusion to sync with LDAP users.In theory, nothing.help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

0800

  2000

ICCICC Quarterly MaintenanceAll ICC services

help@campuscluster.illinois.edu

Status
colourGreen
titleCOMPLETE

2022-07-19 07002022-07-19 0900RadiantVictoria UpdateMinimally disruptive, brief interruptions to OpenStack services, such as the Horizon dashboardradiant-admin@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2022-07-14
2345
2022-07-14
2359
WikiThe service will be restarted in order to increase the login timeout.Wiki will be unavailable for about 5 mins while it restarts.

Status
colourGreen
titlecomplete

2022-07-08
1700
2022-07-11
0800
LSST hosts in NCSA 3003Due to a full building power outage at NCSA on Sunday, 10 July, some LSST servers will be unavailable over the weekend. Servers will be shutdown at COB on Friday and returned to service on Monday morning.lsst-dbb-fts1
lsst-dbb-rucio
lsst-demo
lsst-dm-monitor
lsst-int-monitor
lsst-mon-dev
lsst-pup
lsst-test5
lsst-xfer
l1-cl-arctl
l1-cl-fault
l1-cl-header
nts-ccamfwdr1
nts-acamfwdr2
nts-acamfwdr1
lsst-admin@ncsa.illinois.edu

Status
colourGreen
titlecomplete

2022-07-11 08:302-22-07-11 9:30All ITSM (CMDB) VMsAll ITSM VMs are currently down. Ticket has been created to get them brought back up.Production CMDB service (openDCIM) is not availablekimber7@illinois.edu

Status
colourGreen
titleresolved

2022-07-10 0700

2022-07-10 1430

NCSA building powerBuilding power feed work for multiple campus BuildingsAVL, LSST, ISL and Software standard services were down from Friday afternoon until Monday morning.Daniel Lapine 

Status
colourGreen
titleCOMPLETE

2022-07-8

1600

2022-07-11

0900

cerberus2 and cerberus4Campus is doing work on a common feed that affects multiple buildings, include the NCSA Building. Work is scheduled from 0700-1700, but may finish earlyVM hosts running these 2 bastions will be down for the weekend due to the scheduled power work at NCSAhelp+security@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2022-07-06 17302022-07-06 2030Wiki (wiki.ncsa.illinois.edu)Confluence and MySQL upgradeswiki will be down during the upgrade

Status
colourGreen
titleCOMPLETE

2022-07-05 18002022-07-05 2130NCSA File & Print ServersScheduled Windows Server MaintenanceFile & Print Shares were unavailable during maintenance.  Users were unable to access shares on Fileserver (e.g. home, busnoff, hr, etc.), and printing was unavailable.help@ncsa.illinois.edu

Status
colourGreen
titleCOMPLETE

2022-07-05 1800N/AiForgeend of serviceiForge was removed from service. Operations have moved to the new vForge virtual cluster.help+industry@ncsa.illinois.edu

Status
colourGreen
titleRESOLVED

Legend:

Status
colourRed
titleIN PROGRESS

Status
colourGreen
titleComplete

Status
colourGreen
titleresolved

Status
colourBlue
titlescheduled

Status
colourYellow
titleMonitoring