Page History

Monthly Maintenance:

• OS updates and reboots
• Dell firmware updates
• firmware update on bastion01 (06:00-08:00)
• pfSense firewall maintenance postponed

ALL LSST systems will be updated, including:

• lsst-dev01, lsst-xfer, etc.
• Slurm verification cluster
• PDAC/Kubernetes/LSP clusters
• tus-ats01
• L1 test stand

COMPLETE with the following exceptions:

• lsp-int is still inaccessible
• two one L1 test nodes are is still down (, lsst-l1-cL-frwd16 and lsst-l1-cl-ocs)
• production Oracle services (lsst-oradb.ncsa.illinois.edu) are still down

2019-07-08 06:38:00

remount failed.  system needed a reboot.

Resolved.

On iforge.ncsa.illinois.edu only:

• Access to GPFS shared file system was temporarily unavailable (1-2 minutes only), which could have affected user interactive processes.
• TORQUE commands (qstat, pbsnodes, etc.) were unavailable until the service was restarted at 11:25.

Facility power and cooling/ all production area will be impacted

Commissioning new protective relays: 30 minute rolling power outage

Sonexion re-power: 12 hours work duration 

Expect down time of 2 hours minimum for all production areas

Blue waters down time between 10-12 hours

• iForge was returned to service by 16:55
• aForge will be returned to service no sooner than June 24

• lsst-dev01, lsst-xfer, lsst-dbb-gw
• Slurm verification cluster
• PDAC/Kubernetes/LSP clusters

All VMs remained online during the maintenance, but management through vCenter was unavailable.

Note: Users of the HTML5 interface may need to clear cached browser data after the upgrade. Clear cookies and cached site data from your web browser.

NCSA Fileserver

NCSA-Print

NCSA AD-B

Fileserver shares were unavailable

Shared printers from NCSA-Print were unavailable

help+its@ncsa.illinois.edu

SMB file sharing is down, e.g.:

• H drive in Windows
• shared folders

Returned to Service

2019-05-28

1530

PostgreSQL will be upgraded to version 9.6

The outage will likely last about 10 minutes, but reserving the hour in case of issues.

All uses of IDDS production databases, except Tableau dashboards

• job usage
• accounting
• acctd, jobd, accounting.sh
• IDDS API
• NAPS
• identity services

LSST

Monthly Maintenance:

• Reference new LDAP & Kerberos servers

No interruption of service or downtime.

All LSST systems will be updated, including:

• lsst-dev01, lsst-xfer, etc.
• Slurm verification cluster
• PDAC/Kubernetes/LSP clusters
• tus-ats01

ad.ncsa.edu Domain Controllers (ad-a, ad-b)

Print Servers (ncsa-print, ncsa-printz)

Login to Windows Machines on the ad.ncsa.edu domain should not be affected, but may be interrupted for a short time.

Printing may be interrupted for a short time.

NCSA Identity
LSST Identity

syslog-sec.ncsa.illinois.edu will be changed from a DNS A record to a CNAME pointing to syslog.security.ncsa.illinois.edu

2019-05-07

11:00

2019-05-08

10:45

2019-05-02

14:00

2019-05-02

14:20

2019-04-30 0930

Monthly Maintenance:

• 10G Network switch maintenance
• GPFS server updates
• OS updates and reboots
• Dell firmware updates
• Kubernetes update
• Pending Puppet changes

ALL LSST systems, including:

• lsst-dev01, lsst-xfer, etc.
• Slurm verification cluster
• PDAC/Kubernetes/LSP clusters
• tus-ats01

Quarterly Maintenance

• Ur1carne router code upgrade
• Centos 7.6 upgrade
• Deployment of HDR

2019-04-09 0900

2019-04-07

0645

2019-04-07

1650

New jobs not starting.
Current jobs may stall if access bad oss.

2019-04-04

0800

2019-04-04

0830

network testing

public DNS names were inadvertently removed for LSST's Oracle servers/service and the service became unavailable

• DNS was completed restored by 14:25
• slowness following return to service was initially reported by one user but this seems to have resolved itself

lsst-sysadm@ncsa.illinois.edu

help+security@ncsa.illinois.edu

IDDS will be applying several updates to the NCSA Allocations Processing Service (NAPS):

(1) Searches for logins will only find those logins for the current domain

(2) Logins will always be created for the same organization as the domain (instead of always creating an NCSA login)

(3) Valid login rules will check the rules for the organization of the current domain

(4) Bug fix to make sure int args to procedures are passed as ints, not strings

(5) Speed up project loading process

(6) Dynamically determine compute resources

(7) Correct information in confirmation message when terminating a user from a project

(8) When selecting allocation for new users, only show the most current allocations for each resource

help+neteng@ncsa.illinois.edu

Anyone who has a current session will not be impacted, it will only be for people trying to auth into a new session.   We expect Duo to be up most of this change window and actual downtime to be minutes.  All systems using Duo are affected including:

• NCSA VPN
• bastions (cerberus, ache)
• idp.ncsa.illinois.edu

help+neteng@ncsa.illinois.edu

help+security@ncsa.illinois.edu

This morning during a routine generator transfer test, one of the UPS units in a Tech Services networking node, node-1, failed resulting in a loss of power to portions of node-1.  Network Engineers were on-site during the test and were able to quickly resolve all issues stemming from that loss in power.  Not all equipment hosted in node-1 was impacted but one of the campus core routers, equipment hosting the science DMZ (CARNE and thus ICCP WAN as a whole) and other parts of the ICCN (Inter-Campus Communication Network) were impacted. 

Monthly maintenance

• OS/Yum updates
• Switch maintenance in NPCF N73 & P73
• pfSense update & port negotiation change
• GPFS server updates
• Firmware updates for Dell C6420s

ALL LSST systems, including:

• lsst-dev01, lsst-xfer, etc.
• PDAC, verification, and Kubernetes clusters
• tus-ats01

Maintenance was successfully completed with one pending issue:

• monitoring hosts (lsst-int-monitor; monitor-ncsa) are not showing status information due to problem reaching InfluxDB resolved

• NCSA VPN
• bastions (cerberus, ache)
• idp.ncsa.illinois.edu

ICCN router card crashed. Some commodity internet traffic was affected during the timeframe listed.

2019-02-12 11:50am

We will be doing a correction to a large number of Blue Waters job records in the IDDS database.
This process will begin at 11am and is expected to last around 6-7 hours.

NCSA OpenSource: JIRA, WIKI, BAMBOO, Confluence

BlueWaters HPSS

ncsa#Nearline globus service

HPSS core server encountered a bug and crashed

Vendor is installing a patch to the core hpss server. 

Anticipating the system will be returning to service by 17:20

Boot the system into single user mode and gzip old messages file and moved this to GPFS.

Having issue restarting moab after that. Restart moab with clear checkpoint option and it works.

help+its@ncsa.illinois.edu

Upgrade complete

2019-01-23

5PM

Maintenance complete

RSA OTP user portal online

JIRA, file-server, ad-a, jabber, vsphere, and email relay all rebooted

JIRA had index files corrupted and took a while to repair those

JIRA, file-server, ad-a, jabber, vsphere, and email relay rebooted and online

Monthly maintenance

• Power rebalancing in NPCF L73
• Switch maintenance in NPCF M73, N73, P73
• Critical security patching
• Dell firmware upgrades

Maintenance was completed successfully with the following caveats:

• lsp services in Kubernetes are not fully functional (this is carryover from before the PM; see discussion on Slack, dm-lsp-users and possibly other channels)
• lsst-l1-cl-dmcs will not boot after firmware updates

Please open tickets if you notice other issues.

2019-01-16

10:00

2019-01-16

10:15

01/12/2019 1PM

2019-01-10

5:35PM

help+industry@ncsa.illinois.edu

lsst-admin@ncsa.illinois.edu

01/08/2018 2:20PM

1/08/2018

2:40PM