Minimally disruptive, brief interruptions to OpenStack services, such as the Horizon dashboard

Longer than expected outages of controller service. Instances that had floating IPs had no networking connectivity. Horizon dashboard and API was down (cannot launch new instances, etc).

NCSA LDAP1

replica is down

3 HSN switches were experiencing problems

switches were updated and reconfigured

• Slurm scheduler was paused to prevent new jobs from starting
• Taiga remained unmounted
• various nodes had no connectivity to the HSN
• most services were experiencing some amount of degradation

2022-05-10 0700

2022-05-10 1900

2022 04-14

0800

2022 04-09 0600

Ability to send to email lists.

Note: Bounced emails will need to be resent.

2022-03-17
09:08

Known resources impacted:

Granite: already planned to be offline for maintenance

iForge: cluster offline for the duration

Radiant: cluster online, without power redundancy

Emergency PM

We are seeing some network issues on the cluster. In order to resolve these issues, we need to upgrade code on our infiniband infrastructure

UPDATE: We are currently experiencing unforeseen technical issues with the cluster. We are investigating and expect resolution and restoration of all Campus Cluster services by March 3rd 12PM

UPDATE2: We are still experiencing issues where the compute clients will not properly mount storage. We are engaged with vendor support and continue to work on the situation. Thank you for your patience. We have moved expected return to service to March 4th, 12PM

UPDATE3: Campus cluster is experiencing SLURM job failures in certain pods(sections) of the cluster. Investigations continue and there is a partial return to service with login nodes, storage, and data transfer services still operational. New full return of service date: Monday, March 7th, 12PM.

2022-03-02 1237

2022-03-02 1715

Adding Ram
to improve performance

on-line maintenance

on-line maintenance

Had to restart rsyslog and Ldap after relocating /var/log

kimber7@illinois.edu

2022-02-09 1400

The NCSA VMWare cluster is experiencing storage performance issues.

-- Update: Adjustments have been made to storage used by the LDAP servers and other non-essential VM instances have been disabled. Testing is indicating that response times have improved and services are working normally again.

cerberus2.ncsa.illinois.edu, tg-kdc1.security.ncsa.illinois.edu, bwbh2.ncsa.illinois.edu

Info from Slack (https://status.slack.com/)

We've resolved the issue, and all impacted customers should now be able to access Slack. You may need to reload Slack (Cmd/Ctrl + Shift + R) to see the fix on your end. If that doesn't work, try clearing cache (Help > Troubleshooting > Clear Cache and Restart from the app menu). Thanks for bearing with us and we apologize for the disruption to your work day!

Feb 22, 1:24 PM CST

We're seeing signs of improvement. Please try reloading Slack, and if not a cache reset. We’re still monitoring the situation. We’ll confirm once this issue is fully resolved.

Feb 22, 11:07 AM CST

Slack is not loading for some users. We are continuing to investigate the cause and will provide more information as soon as it's available.

Feb 22, 9:23 AM CST

We're still working towards a full resolution. We'll be back with another update soon. Thank you for your patience.

Feb 22, 8:44 AM CST

We’re investigating the issue where Slack is not loading for some users. We’re looking into the cause and will provide more information as soon as it's available.

Feb 22, 8:25 AM CST

2022-02-18
2PM

Reboot to add ram/swap

This is to improve stability

The Lustre filesystem is not loading correctly. The support team has been contacted.

Still in progress. MDT0001 is partially recovered. Vendor is attempting to fully restore.

Near completion: Working with vendor on additional configuration changes. Hope to complete final validation and return to service by close of business 2022-02-18.

2022-02-18
2PM

Reboot to add ram/swap

This is to improve stability

2022-02-14

1PM

2022-02-14

4:15PM

2022-02-09

1000

2022-02-09

1200

LDAP (and dependent services, incl. Jira)

vSphere/ICI VMware

Authorization timeouts/failures in dependent services.

ICI staff are investigating.

LDAP (and dependent services, incl. Jira)

vSphere/ICI VMware

Cause of most severe issues was power fluctuations around 0555, but certain LDAP servers showed degraded slightly earlier.

MySQL database servers need to be synchronized to bring replicated database servers online.

NOTE: The MySQL database is back up, but users may experience issues due to an LDAP issue.

Wiki, JIRA, Savannah/Internal, Identity, and some web sites will stop working. More details are linked here.

22-02-08

3:15PM

Line card failure is affecting devices that are plugged into Neo-hpc-1 aggregation switch.  We've migrated links off the failed card, to other ports on the same switch.

No services are currently impacted.

help@campuscluster.illinois.edu

kimber7@illinois.edu

Kubernetes on NTS is not working properly after updates

LSST Quarterly Maintenance

2021-12-07

0930

2021-12-07

1030

2021-11-30

1100

The following VMs rebooted:

Idap-Isst-ncsa3
Isst-condordev-cm01
Isst-condordev-sub01
Isst-git
Isst-influxdb-0
Isst-kubh02
Isst-kubh05
Isst-kubh08
Isst-login03
Isst-logintest01
Isst-ora-dbm01
Isst-pup-npcf
Isst-ss-cfg02
Isst-telegraf-0

2021-11-03

0000

2021-11-03

1400

2021-11-03

1100

2021-11-03

1120

ICCP Quarterly Maintenance

• VLAN Change for IPMI network
• OS update

• ALL Nightingale hosts/services
• ALL firewalled traffic in/out of ACHE, which includes admin access & monitoring in/out of ALL of ACHE (this portion was complete by 1140)
  • network access to ALL of the ache-esxi-hosted VMs, including ache- and ngale-bastion hosts
  • ACHE FW IPMI interfaces

LSST Quarterly Maintenance

• OS updates
• K8S updates

2021-09-20 2130

help+service@ncsa.illinois.edu

LSST Emergency OS Patching

LSST services hosted at NCSA except:

• NTS will remain up (has already been patched)

help+serverice@ncsa.illinois.edu

ICCP Quarterly Maintenance

• TBD

help@campuscluster.illinois.edu

2021-07-01

2140

2021-07-01

1430

2021-07-01

0247

2021-07-01

1300

There was a power event in the Champaign-Urbana area at around 2:47AM today. Details about the cause are currently unknown.  This event caused disruptions to systems at the NCSA building, NPCF and ACB. Known issues have generally been resolved but there may be unidentified issues lingering. If you encounter any problems, please notify NCSA help desk staff (help@ncsa.illinois.edu).

2021-06-29 23:59

Matt Kollross

help+neteng@ncsa.illinois.edu

• Updates are being applied on Prod/Stable k8s, rebuild of some ingress nodes

LSST Quarterly Maintenance

• OS updates on all servers

All LSST services hosted at NCSA

EXCEPT Prod/Stable K8S

2021-06-22 0400

Matt Kollross

help+neteng@ncsa.illinois.edu

Matt Kollross

help+neteng@ncsa.illinois.edu

2021-05-26

1000

2021-05-26

1030

Matt Kollross

neteng+help@ncsa.illinois.edu

2021-05-21

1800

2021-05-21

1900

Matt Kollross

neteng+help@ncsa.illinois.edu

ESXi host outage causing degradation of select services.

Degradation of select services:

• data backbone gateway (lsst-dbb-gw01 down)
• HTCondor (Central Manager nodes down for Prod & DAC)
• login (lsst-login01 is down)

Also loss of redundancy for some underlying services, including auth/access & k8s head nodes.

2021-05-12 07:00

internal.ncsa.illinois.edu

help+service@ncsa.illinois.edu

2021-05-11

07:00

2021-05-11

19:00

iforge-admin@lists.ncsa.illinois.edu

Traffic was automatically re-routed to redundant paths during the link outage. Any connections relying on layer-2 connections over AL2S saw a brief blip as the connection is cut over. Affected parties were contacted in advance.

• HTCondor Prod
• HTcondor DAC

• HTCondor Prod
• HTcondor DAC

iccp-admins@campuscluster.illinois.edu

2021-04-15

12:25

2021-04-15

14:45

Several hosts on the vmware service were experiencing timeouts

• bluewaters
• bluewaters-test
• internal
• its-nagios
• ldap1
• vcenter

NCSA passwords can not access repositories with Git clients. Instead use ssh keys over SSH or personal access tokens over HTTPS.

We thought this went into effect during git changes on Nov 2, 2020 but discovered it was still working until we made changes to GitLab to fully remove LDAP functionality.

• Login to the Git web interface was timing out.
• Access from git clients continued to work during the outage.

2021-04-13 0830

2021-03-30

0800

2021-03-30

0845

2021-03-23

2000

2021-03-23

2025

An issue with SSL on the VPN service has caused an issue that has disconnected all users. Network engineering is looking into the issue.

Due to a hardware failure and the VPN not failing over properly to the standby users were unable to connect to the VPN. This was due to an issue with syncing certificates.

2021-03-11
0900

2021-03-11
0930

2021-03-04
0900

2021-03-04
0905

• HTCondor Prod
• HTcondor DAC

• HTCondor Prod
• HTcondor DAC

iccp-admins@campuscluster.illinois.edu

2021-04-15

12:25

2021-04-15

14:45

Several hosts on the vmware service were experiencing timeouts

• bluewaters
• bluewaters-test
• internal
• its-nagios
• ldap1
• vcenter

NCSA passwords can not access repositories with Git clients. Instead use ssh keys over SSH or personal access tokens over HTTPS.

We thought this went into effect during git changes on Nov 2, 2020 but discovered it was still working until we made changes to GitLab to fully remove LDAP functionality.

• Login to the Git web interface was timing out.
• Access from git clients continued to work during the outage.

2021-04-13 0830

2021-03-30

0800

2021-03-30

0845

2021-03-23

2000

2021-03-23

2025

An issue with SSL on the VPN service has caused an issue that has disconnected all users. Network engineering is looking into the issue.

Due to a hardware failure and the VPN not failing over properly to the standby users were unable to connect to the VPN. This was due to an issue with syncing certificates.

2021-03-11
0900

2021-03-11
0930

2021-03-04
0900

2021-03-04
0905

LSST Quarterly Maintenance

• GPFS appliance UPS battery replacements (requires GPFS downtime)
• OS updates
• Kubernetes update from 1.17 to 1.18

2021-02-
0900

2021-02-25
0915

“OmniPoP is doing maintenance for their hardware refresh on February 11 (600 W Chicago) between midnight and 4 a.m.CST.  This will mean that the CARNE 100G OmniPoP connection will go down for a time during the Feb 11 window. Most of the traffic that would take this link will reroute to using other links. The only ICCP user that may be impacted is MWT2 because their primary path to UChicago is over this circuit.  However, we do have a backup UChicago peering over the CARNE Internet2 100G circuit, so that path will be taken assuming that UChicago's backup path doesn't go through the 6WC OmniPoP switch.  The tertiary path to UChicago would be through the ESnet LHCONE peering which goes over the CARNE I2 AL2S 100G. MWT2 shouldn’t need to do anything on their side to prepare for this work.”

2021-02-09

07:00

2021-02-09

15:35

2021-02-09

07:00

2021-02-09

17:00

2021-01-27

8:00 am

2021-01-27

8:00 pm

ICCP Quarterly Maintenance

• Replacing IB cards on 134 nodes (EDR to HDR)
• Installing additional PDU in POD19 Rack5
• Redistributing power from WallPanel C3A3
• New image with GPFS 5.1.0.1
• Clean up IB cables from POD19 Rack[1,2 & 3]

JIRA Upgrade to 8.13.2

LSST Quarterly Maintenance

• Firmware/OS updates
• Kubernetes/Docker updates
• GPFS SSD firmware updates

2020-11-19

11:00 a.m.

2020-11-19

11:30 a.m.

2020-11-18

09:00

2020-11-17

17:00

Quarterly Maintenance

Switching to GPFS 5 formatted filesystem.

iforge-admin@ncsa.illinois.edu

2020-11-05

12:24

NCSA passwords can no longer access repositories. Use GitLab personal access tokens to authenticate against Git over HTTPS.

2020-11-02 09:05

2020-10-29

9:00

2020-10-29

11:15

iforge-admin@ncsa.illinois.edu

2020-10-27

22:10

Firmware updates were applied to the NCSA VPN. Any AnyConnect VPN sessions were reset during the maintenance and users may need to reconnect. Any IPSEC sessions failed over to the standby unit and were not affected.

idp.ncsa.illinois.edu

crl.ncsa.illinois.edu

1. Upgrade Shibboleth IdP software from v3.4.7 to v4.0.1.
2. Move IdP software from VM to Docker container.
3. Change DNS entry for idp.ncsa.illinois.edu and crl.ncsa.illinois.edu to point to new Docker server.

Monthly Maintenance:

• GPFS version upgrade from 4.x to 5.x
• Routine system OS and firmware updates

2020-09-02

1600

Contact Person

Campus Exchange could not deliver email addressed to "@ncsa.illinois.edu" addresses. A user requested a change to Exchange that broke delivery to NCSA.

Any email you sent to "@ncsa.illinois.edu" addresses from campus Exchange will need to be resent by you.

Campus Exchange could not deliver email addressed to "@ncsa.illinois.edu" addresses.

The change in Exchange was reverted around 1655, but it may take a bit of time for Office365 to update all of its servers.

2020-08-26 1200

2020-08-18

06:00

2020-08-08

06:20

Fileserver & Printing

NCSA Wiki

help+its@ncsa.illinois.edu

help+its@ncsa.illinois.edu

Fileserver & Printing

Kubernetes

updates completed 

at 1330

Monthly Maintenance:

• OS updates and reboots
• Other updates as needed
• Firmware on GPFS appliance (fix network issues)

• NCSA Open Source
• INCORE
• classtranscribe
• ...

Fileserver & Printing

help+its@ncsa.illinois.edu

2020-5-14 06:00

2020-04-13 0900

All Globus Services currently using RSA authentication

SSH logins on the hosts:

cerberus{1,2}..ncsa.illinois.edu

bwbh{1,2}.ncsa.illinois.edu

Kimber Blum (kimber7@illinois.edu) or help+idds@ncsa.illinois.edu,

Alina Banerjee(alinab@illinois.edu)

The NPCF data store failed. Optional NFS datastores are available to rebuild VMs.

VMs used by Industry, ICC and BW needed to be recovered and rebuilt.

2020-03-05

06:00

2020-03-05

06:55

Monthly Maintenance:

• OS updates and reboots
• Other updates as needed
• note: Slurm compute nodes will be updated through rolling reboots beginning on 2020-02-28

ALL LSST systems will be updated, including:

• TBD

All ITS managed LDAP servers, including:

• ldap, ldap1, ldap2 .ncsa.illinois.edu
• LSST LDAP replicas
• mg LDAP replicas

2020-02-04

10:00

help+idds@ncsa.illinois.edu

help+idds@ncsa.illinois.edu

Quarterly Maintenance

• Golub IB Core switch FW update
• Golub 10G Core switch FW update
• GPFS 5.0.4.1 update
• Moved golub Rack8 to accommodate expansion

21 servers in the NCSA Test Stand had their drive backplanes replaced by the vendor.

2020-01-06

10:00

2020-01-08

14:30

2020-01-03 

10:20

2019-12-18

08:00

2019-12-18

10:00

Facility infrastructure  Electrical Transformer

TX-5C

Complete

Monthly Maintenance:

• OS updates and reboots
• GPFS filesystem restructure

ALL LSST systems will be updated, including:

• lsst-dev01, lsst-xfer, etc.
• Slurm verification cluster
• PDAC/Kubernetes/LSP clusters
• tus-ats01
• L1 test stand

Connectivity has stabilized. Please report any issues should they arise.

2019-10-23

8 a.m.

2019-10-23

12:00 p.m.

2019-10-22 07:18

2019-10-16

8 a.m.

2019-10-16

12:00 p.m.

2019-10-10 07:30

2019-10-01

Some printers are having issues connecting to the NCSA Print Server.  

After updating drivers on the print server, public printers are working as expected.

2019-10-03

7:45AM

2019-10-01 07:30

Monthly Maintenance:

• 10G Network switch maintenance
• GPFS server updates
• OS updates and reboots
• Dell firmware updates
• Kubernetes update
• Pending Puppet changes

ALL LSST systems, including:

• lsst-dev01, lsst-xfer, etc.
• Slurm verification cluster
• PDAC/Kubernetes/LSP clusters
• tus-ats01

Quarterly Maintenance

• Ur1carne router code upgrade
• Centos 7.6 upgrade
• Deployment of HDR

2019-04-09 0900

2019-04-07

0645

2019-04-07

1650

New jobs not starting.
Current jobs may stall if access bad oss.

2019-04-04

0800

2019-04-04

0830

network testing

public DNS names were inadvertently removed for LSST's Oracle servers/service and the service became unavailable

• DNS was completed restored by 14:25
• slowness following return to service was initially reported by one user but this seems to have resolved itself

lsst-sysadm@ncsa.illinois.edu

help+security@ncsa.illinois.edu

IDDS will be applying several updates to the NCSA Allocations Processing Service (NAPS):

(1) Searches for logins will only find those logins for the current domain

(2) Logins will always be created for the same organization as the domain (instead of always creating an NCSA login)

(3) Valid login rules will check the rules for the organization of the current domain

(4) Bug fix to make sure int args to procedures are passed as ints, not strings

(5) Speed up project loading process

(6) Dynamically determine compute resources

(7) Correct information in confirmation message when terminating a user from a project

(8) When selecting allocation for new users, only show the most current allocations for each resource

help+neteng@ncsa.illinois.edu

Anyone who has a current session will not be impacted, it will only be for people trying to auth into a new session.   We expect Duo to be up most of this change window and actual downtime to be minutes.  All systems using Duo are affected including:

• NCSA VPN
• bastions (cerberus, ache)
• idp.ncsa.illinois.edu

help+neteng@ncsa.illinois.edu

help+security@ncsa.illinois.edu

This morning during a routine generator transfer test, one of the UPS units in a Tech Services networking node, node-1, failed resulting in a loss of power to portions of node-1.  Network Engineers were on-site during the test and were able to quickly resolve all issues stemming from that loss in power.  Not all equipment hosted in node-1 was impacted but one of the campus core routers, equipment hosting the science DMZ (CARNE and thus ICCP WAN as a whole) and other parts of the ICCN (Inter-Campus Communication Network) were impacted. 

Monthly maintenance

• OS/Yum updates
• Switch maintenance in NPCF N73 & P73
• pfSense update & port negotiation change
• GPFS server updates
• Firmware updates for Dell C6420s

ALL LSST systems, including:

• lsst-dev01, lsst-xfer, etc.
• PDAC, verification, and Kubernetes clusters
• tus-ats01

Maintenance was successfully completed with one pending issue:

• monitoring hosts (lsst-int-monitor; monitor-ncsa) are not showing status information due to problem reaching InfluxDB resolved

• NCSA VPN
• bastions (cerberus, ache)
• idp.ncsa.illinois.edu

ICCN router card crashed. Some commodity internet traffic was affected during the timeframe listed.

2019-02-12 11:50am

We will be doing a correction to a large number of Blue Waters job records in the IDDS database.
This process will begin at 11am and is expected to last around 6-7 hours.

NCSA OpenSource: JIRA, WIKI, BAMBOO, Confluence

BlueWaters HPSS

ncsa#Nearline globus service

HPSS core server encountered a bug and crashed

Vendor is installing a patch to the core hpss server. 

Anticipating the system will be returning to service by 17:20

Boot the system into single user mode and gzip old messages file and moved this to GPFS.

Having issue restarting moab after that. Restart moab with clear checkpoint option and it works.

help+its@ncsa.illinois.edu

Upgrade complete

2019-01-23

5PM

Maintenance complete

RSA OTP user portal online

JIRA, file-server, ad-a, jabber, vsphere, and email relay all rebooted

JIRA had index files corrupted and took a while to repair those

JIRA, file-server, ad-a, jabber, vsphere, and email relay rebooted and online

Monthly maintenance

• Power rebalancing in NPCF L73
• Switch maintenance in NPCF M73, N73, P73
• Critical security patching
• Dell firmware upgrades

Maintenance was completed successfully with the following caveats:

• lsp services in Kubernetes are not fully functional (this is carryover from before the PM; see discussion on Slack, dm-lsp-users and possibly other channels)
• lsst-l1-cl-dmcs will not boot after firmware updates

Please open tickets if you notice other issues.

2019-01-16

10:00

2019-01-16

10:15

01/12/2019 1PM

2019-01-10

5:35PM

help+industry@ncsa.illinois.edu

lsst-admin@ncsa.illinois.edu

01/08/2018 2:20PM

1/08/2018

2:40PM

12/26/18

7:31AM

21/31/18

12:50PM

Monthly maintenance

• cutting the cluster over to new Spectrum Scale v5 formatted file system

Total cluster outage.

Taking a bit longer to bring the system back because interface renaming script stop working.

Upgrade was completed successfully

Monthly maintenance

• Puppet code changes
• disable CPU hyperthreading
• OS/Yum updates
• code upgrades on select service & management switches NPCF
• pfSense updates

ALL LSST systems (incl. lsst-dev01, lsst-xfer, etc. as well as PDAC, verification, and Kubernetes clusters, and tus-ats01)

Monthly maintenance

• Split the filesystem
• Reformat with new v5 format

11/14/18
6:23PM

2018-11-14 11:00am

2018-11-14 10:45 am

2018-11-14 11:20

Blue Waters /Home filesystem

2018-11-06

2018-11-06 12:25 pm

Networking NetSure DC Distribution System

Tape Library QBERT and DIGDUG

iForge racks:

• Y121, Z121, AA121, CC121, DD121

Loss of power to the core network DC Distribution panel (B Side), the network is 2N power feed, no impact on the network due to redundancy.

Loss of power to two tape libraries, a temporary power feeds will be provided.

iForge system will be powered down for quarterly maintenance.

2018-11-13

7:30 PM

2018-11-13

8:30 PM

LSST

lspdev/Kubernetes

A power event caused some compute nodes to reboot

A power event caused some hosts to reboot:

• lspdev kubernetes cluster (3 nodes including master node did not come back on their own and were manually brought online around 09:30)
• some L1 nodes rebooted as well

lspdev/Kubernetes cluster was unavailable from ~04:40 until ~09:30

select L1 hosts rebooted

All systems were unavailable during the maintenance.

Maintenance was completed successfully:

• aForge returned to service at 21:15
• iForge returned to service at 21:45

NOTE: OFED was updated to v4 on the clusters during the PM. Some MPI software may need to be recompiled due to changes in libraries (e.g., libpsm_infinipath is no longer present in OFED v4). Frequently used openmpi installations have been updated to accommodate this change. Software compiled against affected MPI software may also need to be recompiled.

2018-11-02

6:10 AM

GPFS issue. "ls /usr/local" hangs.

direct access to some directories under /iusr/local was OK.

ie. "ls /usrlocal/modules-3.2.9.iforge" was OK.

iforge login node is currently down.

New ssh connections are hanging.

There is the potential for issues with running jobs.

Scheduler has been paused.

Jim Long

jlong1s@illinois.edu

Something odd going on with iforge020 was causing hangs.

Once iforge020 was rebooted, access to /usr/local was unlocked.

2018-10-30

11:00 p.m.

2018-10-22
12:00pm

Monthly maintenance

• firmware update and reboot on monitor01 (monitoring collector)
• OS & Kernel updates on tus-ats01.lsst.ncsa.edu
• Puppet code changes

• monitor01/InfluxDB (and likely the front-end Grafana monitoring, e.g., monitor-ncsa.lsst.org) will be unavailable for a short period of time
• tus-ats01 will be unavailable for OS & Kernel updates
• the Puppet changes are intended to be functional "no-ops" and should cause no outage, although we scheduled these changes during our monthly PM window in case something unexpected occurs

Monthly Maintenance

• Deploying new kernel with CVE-2018-14634 fix
• Switching to MTU9000 across
• GPFS 5.0.2 upgrade
• Firmware bug fixes applied to DDN SFA14KX 

2018-10-15

08:00 AM

2018-10-15

08:50 PM

2018-10-15 11:00 pm

affected: all physical LSST hosts (and VMs) at NCSA 3003:

• incl. lsst-dev*, lsst-xfer, lsst-l1*, lsst-daq, lsst-dev-db

• most physical hosts rebooted themselves after the event, although a few L1 systems had to be manually powered on
• most VMs had to be manually started after the event

help+neteng@ncsa.illinois.edu

WPAD has been applied to all user networks at NCSA and NPCF (including wireless).

ALL LSST systems (incl. lsst-dev01, lsst-xfer, etc. as well as PDAC, verification, and Kubernetes clusters)

The following systems will remain online and unaffected:

• tus-ats01

RESOLVED

LSPdev kubernetes is having a gateway error after upgrading

RESOLVED

Monthly maintenance (Sep):

1. Network switch firmware updates/reboots
2. Lenovo firmware updates/reboots
3. OS package updates/reboots
4. ESXi hypervisor updates/reboots

5. GPFS client changes and upgrade to 4.2.3-10

6. GPFS server upgrade to 4.2.3-10

RESOLVED

qserv-master01 and LSPdev are still having issues. These will be tracked as a separate incidents.

Monthly maintenance

1. Switching to CentOS 7.5 across cluster
2. Upgrading gpfs to 4.2.3.10 (client only)

All compute and login nodes were down.

The filesystems were also unavailable due to issues with the change to gpfs and RH7.5

DNS servers will be undergoing routine maintenance. During this maintenance window, system

and services will be restarted. One DNS server will always be responsive during the maintenance.

2018-10-10
17:00 

Preventative Maintenance

• FSCK on filesystem
• Reseat and reset management modules on IB core switch
• BIOS updates on some nodes
• Upgrade Carne uplink to 2x100G

Corrected bad inode on filesystem.

Rebooted IB core switch

2x100G links are working

feedback@slack.com

Slack reported this resolved at 12:41, though NCSA users reported it working around 11:38.

iforge-admin@ncsa.illinois.edu

In progress

• iForge was placed into production at 22:15
• aForge was brought back online by 19:45

• Renewed SSL certificate
• Upgraded GitLab software
• Increased CPU & RAM

horizon interface to nebula https://nebula.ncsa.illinois.edu/ and all open stack command line tools are non-functional. Keystone authentication services are also off-line.

Instances that were running should continue to run but restarting will probably fail until the controller is repaired. launching new instances will also fail.

LSST: lsst-dev-db and dependent services, including kubernetes lspdev

Following the July 19 planned maintenance, MariaDB services on lsst-dev-db are unavailable along with dependent services, including:

• kubernetes lspdev

DB services on lsst-dev-db along with dependent services, including:

• kubernetes lspdev

Monthly maintenance (July):

1. Dell firmware updates/reboots
2. OS package updates/reboots
   1. including upgrades to CentOS 7.5

3. GPFS client changes and upgrade to 4.2.3.9

4. GPFS server upgrade to 4.2.3.9

ALL lsst-dev systems (incl. lsst-dev01, lsst-xfer, etc. as well as PDAC, verification, and Kubernetes clusters)

The following systems will remain online and unaffected:

• lsst-daq
• lsst-l1-*
• tus-ats01

Maintenance was successfully completed, although the following resultant issue is being tracked in a separate status event:

DB services on lsst-dev-db are unavailable along with dependent services, including:

• lspdev

bwadmin@ncsa.illinois.edu
David King

Nodes/Services rebooted.

Kubernetes pods restarted.

2018-06-23
19:44

2018-06-23
19:59

Monthly maintenance (June):

• pfSense firewall update
• OS package updates/reboots for CentOS 6.9 servers (lsst-web, lsst-xfer, lsst-nagios)
• Slurm update (lsst-dev01, lsst-verify-worker*)
• Update host firewalls on GPFS servers
• iDRAC configuration updates on lsst-dev01 and ESXi hosts

CentOS 6.9 servers:

• lsst-web
• lsst-xfer
• lsst-nagios

Slurm/verification cluster

Other impact was not expected but unexpected issues could have lead to connectivity issues for other hosts or downtime for lsst-dev01 or hosted VMs

Scheduled Maintenance:

• BIOS firmware updates
• Puppet and firewall changes (including support of SAL unicast/multicast traffic)
• OS package updates (staying with CentOS 7.4)

Level One Test Stand, including:

• lsst-daq
• lsst-l1-*

cforge

PBSPro server was hung on cfsched

A storm caused a brief power event which impacted:

• big_mem queue
• skyake queue

Storm caused power event which impacted:

• Kubernetes Commons / lsst-lspdev
• 75% of verification cluster compute / Slurm

The following nodes rebooted because of the power event:

• all kub* nodes (causing outage of Kubernetes Commons / lsst-lspdev)
• 75% of verify-worker* nodes (partial outage of Slurm / verification cluster compute nodes

• verify-worker nodes were put back online in Slurm around 06:10
• Kubernetes Commons resumed service by around 09:00

2018-05-31 08:00

2018-06-24 11:08am

Monthly maintenance (May):

• GPFS server & client updates, plus nosuid mounting
• Physical firewall changes in NPCF for new vLANs
• BIOS firmware updates
• OS updates
• Update of puppet-stdlibs module

Maintenance was extended until 13:30 and then completed.

External Grafana monitoring (monitor-ncsa.lsst.org) was offline until 14:25 due to storage rebuild on lsst-monitor01.

Monthly maintenance (May)

• GPFS upgrade to 4.2.3.8
• FW upgrade on Juniper switches
• OS updates
• Add 4 more 40G cables for ccioe nodes for redundancy

relay-01 was updated and loadbalancer failed back.

sssd.conf updated, sssd restarted

verify-worker nodes were drained during the change

affected nodes may have slow LDAP response times for a short while (due to local cache needing rebuilt)

4/10

17:00

4/10

18:00

4/10

15:00

4/10

16:00

4/04/2018

16:00

4/04/2018

17:00

4/04/2018

16:00

4/04/2018

17:00

3/29/2018

17:00

New management server is up with Centos7. Installed Torque 6.1.2 and Maob 9.1.2. Bios update are done on most nodes. Configuration changes on GPFS done. Tech services CARNE code upgrade done.

Security patches of VM servers as well as backend filesystem

Updates of Bamboo, JIRA, Confluence, BitBucket and CROWD 

All systems will be unavailable for a brief period of time.

During updates of OpenSource services part of OpenSource will be offline for up to an hour.

Nebula Openstack cluster

Lingering issues on select nodes following March PM

• lsst-qserv-master01 - cannot mount local /qserv volume
• lsst-xfer - issue w/ sshd
• lsst-dts - issue w/ sshd
• lsst-l1-cl-dmcs - unknown issue
• lsst7 - issue w/ sshd

Following resolved by 13:23:

• lsst-qserv-master01
• lsst-xfer
• lsst-dts
• lsst-l1-cl-dmcs

Resolved by 16:20:

• lsst7