You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 1490 Next »

status.ncsa.illinois.edu


Watch this page in the wiki to subscribe to automatic updates to this status page.

Please do not refer to any NCSA Industry Partners on this page. Please use the iforge nomenclature for all of the *forge infrastructure.

To see older events, see Archive of NCSA Status Home

Report a problem 

Current Status  

START
ENDWhat System/Service is affectedWhat is happening?What will be affected?Contact PersonStatus


Upcoming Scheduled Maintenance

Listed below in chronological order.

StartEndWhat System/Service is affectedWhat is happening?What will be affected?Contact PersonStatus

2021-03-17

0600

2021-03-17

0700

NCSA VPNFirmware on the NCSA VPN appliance is being upgraded.The VPN will be unavailable for the duration of the window.neteng@ncsa.illinois.edu

SCHEDULED

2021-06-24
0800
2021-06-24
1200
LSST

LSST Quarterly Maintenance

  • TBD
All LSST services hosted at NCSAlsst-admin@ncsa.illinois.edu

SCHEDULED

2021-09-30
0800
2021-09-30
1200
LSST

LSST Quarterly Maintenance

  • TBD
All LSST services hosted at NCSAlsst-admin@ncsa.illinois.edu

SCHEDULED

2021-12-09
0800
2021-12-09
1200
LSST

LSST Quarterly Maintenance

  • TBD
All LSST services hosted at NCSAlsst-admin@ncsa.illinois.edu

SCHEDULED









Previous Outages or Maintenance

StartEndWhat System/Service was affected?What happened?What was affected?

Contact Person

Status
~16:4017:58AnyConnect VPN Service

An issue with SSL on the VPN service has caused an issue that has disconnected all users. Network engineering is looking into the issue.


Due to a hardware failure and the VPN not failing over properly to the standby users were unable to connect to the VPN. This was due to an issue with syncing certificates.

During the outage, expect that you won't be able to connect/maintain a connection to the VPNhelp+neteng@ncsa.illinois.edu

RESOLVED

2021-03-16 09502021-03-16 1000CMDBWill be applying updates per security vettingCMDB, including web interface, will be down briefly during the update.ncsagroup+org_itsm@ncsa.illinois.edu

RESOLVED

2021-03-11
0900

2021-03-11
0930

WAN Link MigrationNCSA Neteng migrated the link to ICCN to new hardware.Traffic was automatically re-routed to redundant paths during the link outage.help+neteng@ncsa.illinois.edu

RESOLVED

2021-03-04
0900

2021-03-04
0905

WAN Link MigrationNCSA Neteng migrated the 100G link to MREN to new hardware.Traffic was automatically re-routed to redundant paths during the link outage.help+neteng@ncsa.illinois.edu

RESOLVED

2021-03-01 22:112021-03-01 22:47NCSA vSphereAbout 40 VMs lost connection to their NFS storage.Several VM-based services were timing out during the issue, including: vSphere management, a kerberos replica, a ldap replica, httpproxy, license servers, NCSA fileserver, Identity message queuing, monitoring. That triggered some of those VMs to switch to use read-only disk, needing to be rebooted later.service@ncsa.illinois.edu

RESOLVED

2021-02-25
0800
2021-02-25
1200
LSST

LSST Quarterly Maintenance

  • GPFS appliance UPS battery replacements (requires GPFS downtime)
  • OS updates
  • Kubernetes update from 1.17 to 1.18
All LSST services hosted at NCSAlsst-admin@ncsa.illinois.edu

COMPLETE

2021-02-
0900

2021-02-25
0915

WAN Link MigrationNCSA Neteng migrated the 100G link to CARNE to new hardware.The link to campus through the CARNE router was migrated to new hardware. Traffic was automatically re-routed to redundant paths during each link outage.help+neteng@ncsa.illinois.edu

COMPLETE

2021-2-18 2:30 pm2021-2-18 6pmvsphere.ncsa.illinois.edulogins were broken due to a cert caching issue on vsphere.login to the administrative interface is availableservice@ncsa.illinois.edu

COMPLETE

2021-02-18 2pm2021-02-18 2:30pmldap The certs on several ldap servers were set to expire tomorrow and next week, they were refreshed.ldap server certs were refreshed prior to their expirations.help+service@ncsa.illinois.edu

COMPLETE

2021-02-17. 10:00 a.m.2021-02-17. 10:30 a.m.Netdot maintpatchingNetdot may be unavailable during this time.help+neteng@ncsa.illinois.edu

COMPLETE

2021-02-16 08002021-02-16 1000cilogon.orgUpdate to OA4MP v5.1 had problems.Several clients reported issues with OA4MP 5.1, so we reverted to OA4MP 4.4.5 at noon.help@cilogon.org

CANCELED

2021-02-15 18002021-02-17 00:40 File & Print ServersMonthly Windows File & Print Server MaintenanceWindows File Shares such as HR, Business Office, Home, etc. and printing in the NCSA & NPCF buildings will be unavailable.help+service@ncsa.illinois.edu 

COMPLETE

2021-02-11 00:002021-02-11 04:00ICCP - MWT2

“OmniPoP is doing maintenance for their hardware refresh on February 11 (600 W Chicago) between midnight and 4 a.m.CST.  This will mean that the CARNE 100G OmniPoP connection will go down for a time during the Feb 11 window. Most of the traffic that would take this link will reroute to using other links. The only ICCP user that may be impacted is MWT2 because their primary path to UChicago is over this circuit.  However, we do have a backup UChicago peering over the CARNE Internet2 100G circuit, so that path will be taken assuming that UChicago's backup path doesn't go through the 6WC OmniPoP switch.  The tertiary path to UChicago would be through the ESnet LHCONE peering which goes over the CARNE I2 AL2S 100G. MWT2 shouldn’t need to do anything on their side to prepare for this work.”

All traffic should reroute during this maintenance, but MWT2 may experience brief connectivity issues to UChicagoneteng@ncsa.illinois.edu

COMPLETE

2021-02-09

07:00

2021-02-09

15:35

iforgeftp1public interface is downS3 connectioniforge-admin@lists.ncsa.illinois.edu

COMPLETE

2021-02-09

07:00

2021-02-09

17:00

iForge clusterQuarterly MaintenanceAll systems unavailableiforge-admin@lists.ncsa.illinois.edu

COMPLETE

2021-01-28
10:00am
2021-01-28
12:00pm
RadiantSecurity updates and API endpoint hardeningThe web interface to Radiant and the API interfaces will be unavailable during the maintenance period.radiant-admin@lists.ncsa.illinois.edu

COMPLETE

2021-01-27

8:00 am

2021-01-27

8:00 pm

Open Storage Network PODUpdating Ceph to containerized implementationSee Previous Columnbdickin2@illinois.edu

COMPLETE

2021-01-252021-01-26NCSA LDAPUsers with /bin/csh had their shells changed to /bin/bashUsers logging into systems that don't override the /bin/csh data already will find they are using /bin/bash when they login.help+service@ncsa.illinois.edu

COMPLETE

2021-01-26 13:452021-01-26 14:05administrative interface to vsphere.ncsa.illinosi.eduadministrative interface to vsphere.ncsa.illinosi.edu was upgraded to current patch levelAdministrative interfaces to vm's were unavailable for about 20 minutes.help+service@ncsa.illinois.edu

COMPLETE

2021-01-20 08:002021-01-20 20:00ICCP

ICCP Quarterly Maintenance

  • Replacing IB cards on 134 nodes (EDR to HDR)
  • Installing additional PDU in POD19 Rack5
  • Redistributing power from WallPanel C3A3
  • New image with GPFS 5.1.0.1
  • Clean up IB cables from POD19 Rack[1,2 & 3]
Cluster-wide outagehelp@campuscluster.illinois.edu

COMPLETE

2021-01-12 07:002021-01-12 8:00JIRA

JIRA Upgrade to 8.13.2

All JIRA usershelp+its@ncsa.illinois.edu

COMPLETE

2021-01-04 09:142021-01-04 12:10SlackSlack service issuesAll Slack systemshttps://status.slack.com/OUTAGE
2020-12-21 18:002020-12-22 0700File & Print ServersMonthly Windows File & Print Server MaintenanceWindows File Shares such as HR, Business Office, Home, etc. and printing in the NCSA & NPCF buildings were unavailable.help+its@ncsa.illinois.edu 

COMPLETE

2020-12-182020-12-18NCSA LDAPNew user accounts will have their shell in ldap set to /bin/bashNew users will have /bin/bash as their default shellhelp+service@ncsa.illinois.edu

COMPLETE

2020-12-10
0800
2020-12-10
1400
LSST

LSST Quarterly Maintenance

  • Firmware/OS updates
  • Kubernetes/Docker updates
  • GPFS SSD firmware updates
All LSST services hosted at NCSAlsst-admin@ncsa.illinois.edu

COMPLETE

2020-12-09 08002020-12-10 1000HALHAL Quarterly MaintenanceHAL clusterhelp+isl@ncsa.illinois.edu

COMPLETED

2020-11-24 10:302020-11-24 17:35Blue Waters computeBoot RAID lost fiber channel connection for reasons not understoodFull system outagejenos@illinois.edu

COMPLETED

2020-11-19

11:00 a.m.

2020-11-19

11:30 a.m.

DNS1 MigrationNeteng will be migrating DNS1 to a new switch.  We need to physical move the cable to DNS1 which will cause momentary outage for dns queries to DNS1.  DNS2 will not be affected by the migration.  help+neteng@ncsa.illinois.edu

COMPLETED

2020-11-18

09:00

2020-11-17

17:00

UPS battery Monitor Program and configure the new BMS All UPS connected loadsrantissi@illinois.edu

COMPLETED

2020-11-16 18:002020-11-16 21:30NCSA File & Print ServersFile & Print servers were offline for scheduled maintenance.  Windows File Shares and printing were unavailable.Windows File Shares such as HR, Business Office, Home, etc. and printing in the NCSA & NPCF buildings were unavailable.help+its@ncsa.illinois.edu 

COMPLETED

2020-10-14 13:452020-10-14 14:04NCSA Wiki & JiraNCSA's Wiki & Jira servers were restarted.Wiki & Jira were offline while their servers reboot.help+its@ncsa.illinois.edu

COMPLETED

2020-11-13 12:002020-11-13 18:00Software Directorate VM FarmFailing power supply of switch will be replaced, will use this to upgrade OS as well.NCSA OpenSource, INCORE, etc (all machines running on 141.142.277.X).kooper@illinois.edu

COMPLETED

2020-11-12 9:552020-11-12 10:15cilogon.orgUpdate to OA4MP v5.0.2 was unsuccessful.CILogon Service has been reverted to OA4MP 4.4.5.help@cilogon.org

DELAYED

2020-11-10
07:00
2020-11-10 21:05iForge cluster

Quarterly Maintenance

Switching to GPFS 5 formatted filesystem.

All iForge nodes.

iforge-admin@ncsa.illinois.edu

COMPLETE

2020-11-05 10:00

2020-11-05

12:24

NPCF enterprise UPSUPS maintenance, replace defective communication cardsAny rack (system) that is UPS power fed rantissi@illinoise.edu

COMPLETE

2020-11-04 19:002020-110-4 21:00NCSA Building Router (2 of 2)Software UpdatesSoftware Updates will be applied to one of the NCSA building routers.  Traffic will fall back to the seconds router.  No network traffic should be affected.help+neteng@ncsa.illinois.edu

COMPLETED


2020-11-02 08:002020-11-02
10:20
NCSA GitLab
git.ncsa.illinois.edu
LDAP authentication was disabled for NCSA GitLab. Users of the GitLab web interface are required to authenticate to NCSA through CILogon.

NCSA passwords can no longer access repositories. Use GitLab personal access tokens to authenticate against Git over HTTPS.

help+its@ncsa.illinois.edu

COMPLETED

2020-11-02 09:00

2020-11-02 09:05

NCSA DuoThe icon shown in the Duo app for NCSA will be updated to match the icon used in NCSA Slack.NCSA Duo App pushes will show updated icon to match NCSA Slack. May need to restart phone/app to see updated icon.help+duo@ncsa.illinois.edu

COMPLETED

2020-10-30
08:00
2020-10-30
08:45
SVN at subversion.ncsa.illinois.eduRetired SVN Service at subversion.ncsa.illinois.edu SVN is no longer be available. NCSA users are recommended to use one of our various Git repository options.help+its@ncsa.illinois.edu

COMPLETED

2020-10-29

9:00

2020-10-29

11:15

iForgemaintenance to switch to GPFS version 5.All nodes.

iforge-admin@ncsa.illinois.edu


COMPLETED

2020-10-28 19:002020-10-28 21:00NCSA Building Router (1 of 2)Software UpdatesSoftware Updates will be applied to one of the NCSA building routers.  Traffic will fall back to the seconds router.  No network traffic should be affected.help+neteng@ncsa.illinois.edu

COMPLETED

2020-10-27 22:00

2020-10-27

22:10

NCSA VPNFirmware Updates

Firmware updates were applied to the NCSA VPN. Any AnyConnect VPN sessions were reset during the maintenance and users may need to reconnect. Any IPSEC sessions failed over to the standby unit and were not affected.

help+neteng@ncsa.illinois.edu

COMPLETED

2020-10-27 09:002020-10-27 09:15

idp.ncsa.illinois.edu

crl.ncsa.illinois.edu

  1. Upgrade Shibboleth IdP software from v3.4.7 to v4.0.1.
  2. Move IdP software from VM to Docker container.
  3. Change DNS entry for idp.ncsa.illinois.edu and crl.ncsa.illinois.edu to point to new Docker server.
The DNS CNAME entries for idp.ncsa.illinios.edu and crl.ncsa.illinois.edu will be changed from cilogon-web.ncsa.illinois.edu to shib-docker.security.ncsa.illinois.edu (141.142.149.33). NCSA Shib IdP v4.0.1 is currently up and running at 141.142.149.33.help+idp@ncsa.illinois.edu

COMPLETED

2020-10-22 11am2020-10-22 1pmsecurity.ncsa.illinois.edu and grid.ncsa.illinois.eduCert replacement issueSites were downcpitcel

RESOLVED

2020 10-21 09002020 10-21 1400WAN Link MigrationICCN Engineers will be migrating NCSA's 100G WAN links over to new optical cards.

Below is the timetable for the moves:

10:00am CARNE (Node 1) to I2 (710 N Lakeshore Dr)
10:30am NCSA (Node 2) to MREN (710 N Lakeshore Dr)

11:30am CARNE (Node 2) to OmniPop (600 West Chicago)
12:00pm NCSA (Node 1) to I2 (600 West Chicago)
12:15pm NCSA (Node 1) to ESNet (600 West Chicago)

Individual links will be migrated one at a time, each taking roughly 15-20 minutes to complete, leaving redundant paths operational. Traffic will automatically be re-routed to these redundant paths during each link outage.

There are exceptions where certain services won't failover in this way. In these cases, individual notifications have been sent out to affected parties.
help+neteng@ncsa.illinois.edu

COMPLETED

2020-10-13 08:002020-10-13 09:00CILogonUpdate to OA4MP v5.0 OAuth2/OIDC Libraries encountered issue with Syngenta IdP. Reverted to OA4MP v4.4.5. Will be addressed in future OA4MP update.https://cilogon.orghelp@cilogon.org

DELAYED

2020 10-13 06:002020 10-13 06:15CILogon COmanage Registry at https://registry.cilogon.orgService stack restart.COmanage Registry and LDAP directory for the multi-tenant services.help@cilogon.org

COMPLETED

2020-10-12 18:002020-10-12 21:30NCSA File & Print ServersMonthly Maintenance for Updates / Backup ChecksFile & Print Servers were unavailable.  Printing was offline, and fileserver shares were unavailable.help+its@ncsa.illinois.edu

COMPLETED

2020-10-01 06:002020-10-01 06:15NCSA VPNThe certificate for sslvpn.ncsa.illinois.edu was updated. The SSL certificate has been updated.neteng@ncsa.illinois.edu

COMPLETED

2020-09-28 21:002020-09-29 16:45Nebula Network card failed in network node, was replaced and network settings reconfigured All non-virtual networking services for Nebula instances (north/south traffic)nebula@ncsa.illinois.edu

COMPLETED

2020-09-23
19:00

2020-09-24
12:30

LSST

Monthly Maintenance:

  • GPFS version upgrade from 4.x to 5.x
  • Routine system OS and firmware updates
ALL LSST systemslsst-admin@ncsa.illinois.edu

COMPLETED

2020-09-222020-09-22CILogon multi-tenant COmanage RegistryUpgrade to version 3.3.0COmanage Registry service at https://registry.cilogon.orghelp@cilogon.org

COMPLETED

2020-09-14
18:00
2020-09-15
21:30
NCSA File & Print ServersWindows file and print servers were patched and unavailable during maintenance.Access to Fileserver (Business Office, HR, Home, and Swap shared drives) was unavailable, printing was unavailable.help+its@ncsa.illinois.edu 

COMPLETED

2020-09-08 08:002020-09-08 09:00CILogonUpdate CILogon OIDC Client Admin APIhttps://cilogon.orghelp@cilogon.org

COMPLETED

2020-09-02 0800

2020-09-02

1600

Core-EastSoftware UpgradesNo user impact expectedhelp+neteng@ncsa.illinois.edu

COMPLETED

2020-09-012020-09-01ldaps://ldap.cilogon.orgRestart of LDAP gateway service containers.All LDAP services operated by CILogon.help@cilogon.org

COMPLETED


  • No labels