Skip to end of metadata
Go to start of metadata

status.ncsa.illinois.edu


Watch this page in the wiki to subscribe to automatic updates to this status page.

Please do not refer to any NCSA Industry Partners on this page. Please use the iforge nomenclature for all of the *forge infrastructure.

To see older events, see Archive of NCSA Status Home

Report a problem 

Current Status  

START
ENDWhat System/Service is affectedWhat is happening?What will be affected?Contact PersonStatus








Upcoming Scheduled Maintenance

StartEndWhat System/Service is affectedWhat is happening?What will be affected?Contact PersonStatus

2020-11-19

11:00 a.m.

2020-11-19

11:30 a.m.

DNS1 MigrationNeteng will be migrating DNS1 to a new switch.  We need to physical move the cable to DNS1 which will cause momentary outage for dns queries to DNS1.  DNS2 will not be affected by the migration.  help+neteng@ncsa.illinois.edu

SCHEDULED


Previous Outages or Maintenance

StartEndWhat System/Service was affected?What happened?What was affected?

Contact Person

Status
2020-11-24 10:302020-11-24 17:35Blue Waters computeBoot RAID lost fiber channel connection for reasons not understoodFull system outagejenos@illinois.edu

COMPLETED

2020-11-18

09:00

2020-11-17

17:00

UPS battery Monitor Program and configure the new BMS All UPS connected loadsrantissi@illinois.edu

COMPLETED

2020-11-16 18:002020-11-16 21:30NCSA File & Print ServersFile & Print servers were offline for scheduled maintenance.  Windows File Shares and printing were unavailable.Windows File Shares such as HR, Business Office, Home, etc. and printing in the NCSA & NPCF buildings were unavailable.help+its@ncsa.illinois.edu 

COMPLETED

2020-10-14 13:452020-10-14 14:04NCSA Wiki & JiraNCSA's Wiki & Jira servers were restarted.Wiki & Jira were offline while their servers reboot.help+its@ncsa.illinois.edu

COMPLETED

2020-11-13 12:002020-11-13 18:00Software Directorate VM FarmFailing power supply of switch will be replaced, will use this to upgrade OS as well.NCSA OpenSource, INCORE, etc (all machines running on 141.142.277.X).kooper@illinois.edu

COMPLETED

2020-11-12 9:552020-11-12 10:15cilogon.orgUpdate to OA4MP v5.0.2 was unsuccessful.CILogon Service has been reverted to OA4MP 4.4.5.help@cilogon.org

DELAYED

2020-11-10
07:00
2020-11-10 21:05iForge cluster

Quarterly Maintenance

Switching to GPFS 5 formatted filesystem.

All iForge nodes.

iforge-admin@ncsa.illinois.edu

COMPLETE

2020-11-05 10:00

2020-11-05

12:24

NPCF enterprise UPSUPS maintenance, replace defective communication cardsAny rack (system) that is UPS power fed rantissi@illinoise.edu

COMPLETE

2020-11-04 19:002020-110-4 21:00NCSA Building Router (2 of 2)Software UpdatesSoftware Updates will be applied to one of the NCSA building routers.  Traffic will fall back to the seconds router.  No network traffic should be affected.help+neteng@ncsa.illinois.edu

COMPLETED


2020-11-02 08:002020-11-02
10:20
NCSA GitLab
git.ncsa.illinois.edu
LDAP authentication was disabled for NCSA GitLab. Users of the GitLab web interface are required to authenticate to NCSA through CILogon.

NCSA passwords can no longer access repositories. Use GitLab personal access tokens to authenticate against Git over HTTPS.

help+its@ncsa.illinois.edu

COMPLETED

2020-11-02 09:00

2020-11-02 09:05

NCSA DuoThe icon shown in the Duo app for NCSA will be updated to match the icon used in NCSA Slack.NCSA Duo App pushes will show updated icon to match NCSA Slack. May need to restart phone/app to see updated icon.help+duo@ncsa.illinois.edu

COMPLETED

2020-10-30
08:00
2020-10-30
08:45
SVN at subversion.ncsa.illinois.eduRetired SVN Service at subversion.ncsa.illinois.edu SVN is no longer be available. NCSA users are recommended to use one of our various Git repository options.help+its@ncsa.illinois.edu

COMPLETED

2020-10-29

9:00

2020-10-29

11:15

iForgemaintenance to switch to GPFS version 5.All nodes.

iforge-admin@ncsa.illinois.edu


COMPLETED

2020-10-28 19:002020-10-28 21:00NCSA Building Router (1 of 2)Software UpdatesSoftware Updates will be applied to one of the NCSA building routers.  Traffic will fall back to the seconds router.  No network traffic should be affected.help+neteng@ncsa.illinois.edu

COMPLETED

2020-10-27 22:00

2020-10-27

22:10

NCSA VPNFirmware Updates

Firmware updates were applied to the NCSA VPN. Any AnyConnect VPN sessions were reset during the maintenance and users may need to reconnect. Any IPSEC sessions failed over to the standby unit and were not affected.

help+neteng@ncsa.illinois.edu

COMPLETED

2020-10-27 09:002020-10-27 09:15

idp.ncsa.illinois.edu

crl.ncsa.illinois.edu

  1. Upgrade Shibboleth IdP software from v3.4.7 to v4.0.1.
  2. Move IdP software from VM to Docker container.
  3. Change DNS entry for idp.ncsa.illinois.edu and crl.ncsa.illinois.edu to point to new Docker server.
The DNS CNAME entries for idp.ncsa.illinios.edu and crl.ncsa.illinois.edu will be changed from cilogon-web.ncsa.illinois.edu to shib-docker.security.ncsa.illinois.edu (141.142.149.33). NCSA Shib IdP v4.0.1 is currently up and running at 141.142.149.33.help+idp@ncsa.illinois.edu

COMPLETED

2020-10-22 11am2020-10-22 1pmsecurity.ncsa.illinois.edu and grid.ncsa.illinois.eduCert replacement issueSites were downcpitcel

RESOLVED

2020 10-21 09002020 10-21 1400WAN Link MigrationICCN Engineers will be migrating NCSA's 100G WAN links over to new optical cards.

Below is the timetable for the moves:

10:00am CARNE (Node 1) to I2 (710 N Lakeshore Dr)
10:30am NCSA (Node 2) to MREN (710 N Lakeshore Dr)

11:30am CARNE (Node 2) to OmniPop (600 West Chicago)
12:00pm NCSA (Node 1) to I2 (600 West Chicago)
12:15pm NCSA (Node 1) to ESNet (600 West Chicago)

Individual links will be migrated one at a time, each taking roughly 15-20 minutes to complete, leaving redundant paths operational. Traffic will automatically be re-routed to these redundant paths during each link outage.

There are exceptions where certain services won't failover in this way. In these cases, individual notifications have been sent out to affected parties.
help+neteng@ncsa.illinois.edu

COMPLETED

2020-10-13 08:002020-10-13 09:00CILogonUpdate to OA4MP v5.0 OAuth2/OIDC Libraries encountered issue with Syngenta IdP. Reverted to OA4MP v4.4.5. Will be addressed in future OA4MP update.https://cilogon.orghelp@cilogon.org

DELAYED

2020 10-13 06:002020 10-13 06:15CILogon COmanage Registry at https://registry.cilogon.orgService stack restart.COmanage Registry and LDAP directory for the multi-tenant services.help@cilogon.org

COMPLETED

2020-10-12 18:002020-10-12 21:30NCSA File & Print ServersMonthly Maintenance for Updates / Backup ChecksFile & Print Servers were unavailable.  Printing was offline, and fileserver shares were unavailable.help+its@ncsa.illinois.edu

COMPLETED

2020-10-01 06:002020-10-01 06:15NCSA VPNThe certificate for sslvpn.ncsa.illinois.edu was updated. The SSL certificate has been updated.neteng@ncsa.illinois.edu

COMPLETED

2020-09-28 21:002020-09-29 16:45Nebula Network card failed in network node, was replaced and network settings reconfigured All non-virtual networking services for Nebula instances (north/south traffic)nebula@ncsa.illinois.edu

COMPLETED

2020-09-23
19:00

2020-09-24
12:30

LSST

Monthly Maintenance:

  • GPFS version upgrade from 4.x to 5.x
  • Routine system OS and firmware updates
ALL LSST systemslsst-admin@ncsa.illinois.edu

COMPLETED

2020-09-222020-09-22CILogon multi-tenant COmanage RegistryUpgrade to version 3.3.0COmanage Registry service at https://registry.cilogon.orghelp@cilogon.org

COMPLETED

2020-09-14
18:00
2020-09-15
21:30
NCSA File & Print ServersWindows file and print servers were patched and unavailable during maintenance.Access to Fileserver (Business Office, HR, Home, and Swap shared drives) was unavailable, printing was unavailable.help+its@ncsa.illinois.edu 

COMPLETED

2020-09-08 08:002020-09-08 09:00CILogonUpdate CILogon OIDC Client Admin APIhttps://cilogon.orghelp@cilogon.org

COMPLETED

2020-09-02 0800

2020-09-02

1600

Core-EastSoftware UpgradesNo user impact expectedhelp+neteng@ncsa.illinois.edu

COMPLETED

2020-09-012020-09-01ldaps://ldap.cilogon.orgRestart of LDAP gateway service containers.All LDAP services operated by CILogon.help@cilogon.org

COMPLETED

2020-08-28 1425

2020-08-28 1725NCSA Email from Campus Exchange

Campus Exchange could not deliver email addressed to "@ncsa.illinois.edu" addresses. A user requested a change to Exchange that broke delivery to NCSA.

Any email you sent to "@ncsa.illinois.edu" addresses from campus Exchange will need to be resent by you.

Campus Exchange could not deliver email addressed to "@ncsa.illinois.edu" addresses.

The change in Exchange was reverted around 1655, but it may take a bit of time for Office365 to update all of its servers.

help+its@ncsa.illinois.edu

RESOLVED

2020-08-28 08002020-08-28 1000NPCF Networking DC Power SystemTesting and maintenance of the DC power system and battery backup was performed.No outage.help+neteng@ncsa.illinois.edu

COMPLETED

2020-08-26 0800

2020-08-26 1200

Core-WestReplacing failed internal board.  Software upgradesNo user impact expected.  help+neteng@ncsa.illinois.edu

COMPLETED

2020-08-17 08002020-08-22 19:00Code42 Crashplan EndpointOne of the primary Code42 Crashplan Endpoints crashed and was not accepting backup requestsCode42 Crashplan clients that were assigned to backup to this server were unable to run backups.crashplan@ncsa.illinois.edu

RESOLVED

2020-08-18

06:00

2020-08-08

06:20

NCSA VPN ServiceSoftware upgrade was completed successfully.All IPSEC sessions failed over successfully to the standby. Any users connected to the AnyConnect VPN were briefly disconnected and need to reconnect. An upgrade of the AnyConnect client was included with this upgrade and users will receive the upgrade when they reconnect to the VPN.help+neteng@ncsa.illinois.edu

COMPLETED

2020-08-18
06:00
2020-08-18
06:15
registry.cilogon.orgCILogonPerson LDAP schema plugin was removed.The COmanage Registry service.help@cilogon.org

COMPLETED

2020-08-11 07:002020-08-11 18:30iForgeQuarterly MaintenanceAll systems will be unavailable during the maintenanceiforge-admin@ncsa.illinois.edu

COMPLETED

2020-08-04 08:002020-08-04 08:30CILogonhttps://go.ncsa.illinois.edu/CILogonServiceUpdate2020-08-04Remove <whitelisted> tag from idplist.xml file. Add "?initialidp=..." query parameter.help@cilogon.org

COMPLETED

2020-08-03 18:002020-08-03 23:00

Fileserver & Printing

Monthly Windows Server MaintenanceNCSA Fileserver(s) and NCSA-Print were unavailable during the maintenance.   Business Office, HR, home, swap, etc. shares were unavailable.  Printing was unavailable.help+its@ncsa.illinois.edu

COMPLETED

2020-07-28
06:00
2020-07-28
06:05
registry.cilogon.orgOA4MP plugin for creating/managing OIDC clients was updatedThe COmanage Registry service was unavailable during the outage.skoranda@illinois.edu

COMPLETED

2020-07-24 07002020-07-24 0707

NCSA Wiki

Applied some minor configuration adjustmentsPages were unavailable for 5 minutes as the service restarted.

help+its@ncsa.illinois.edu

COMPLETED

2020-07-22 08002020-07-22 0823git.ncsa.illinois.eduGitLab was upgraded to the latest versionThe git service was intermittently unavailable during this upgrade window.

help+its@ncsa.illinois.edu

COMPLETED

2020-07-18 05002020-07-18 0600NCSA WAN connectivity over Internet2 and ESnet links. Emergency network maintenance performed by ICCN (our WAN provider) Instability on WAN connectivity to Industry partners and research institutions using internet2 service provider network to connect to NCSA. General network connectivity will re-route accordingly. help+neteng@ncsa.illinois.edu

COMPLETED

2020-07-16 07002020-07-16 0800Core-WestA service alarm on Core-West switch requires a reset of an internal controller board.Traffic will be failed over to Core-East while performing the maint.  No impact expected.help+neteng@ncsa.illinois.edu

COMPLETED

2020-07-15 14002020-07-15 1730Storage CondoPerc Raid card failure in core serverCondo Services/GridFTP/NFSckerner@illinois.edu

COMPLETED

2020-07-07 18:002020-07-07 21:30

Fileserver & Printing

Monthly Windows Server MaintenanceNCSA Fileserver(s) and NCSA-Print were unavailable during the maintenance.   Business Office, HR, home, swap, etc. shares were unavailable.  Printing was unavailable.help+its@ncsa.illinois.edu

COMPLETED

2020-07-07 08002020-07-07 0900CILogonUpdate Admin Client APIhttps://cilogon.org . No downtime is expected.help@cilogon.org

COMPLETED

2020-06-24
1900

2020-06-25
1300

Kubernetes

updates completed 

at 1330

LSST

Monthly Maintenance:

  • OS updates and reboots
  • Other updates as needed
  • Firmware on GPFS appliance (fix network issues)
ALL LSST systemslsst-admin@ncsa.illinois.edu

COMPLETE

2020-06-17 06002020-06-17 1600Blue Waters scratch filesystemDisk failure during OST failover, both OSTs unavailable
14 drives offline, reassembly required
Some filesystem operations.

COMPLETED

2020-06-15 08:002020-06-15 11:50Software Directorate VM Farm
  • NCSA Open Source
  • INCORE
  • classtranscribe
  • ...
Upgrade of servers, including vm serversDuring this time all servers will be down, servers will be returned ASAP

COMPLETED

2020-06-10 09:002020-06-12 21:00hal.ncsa.illinois.eduQuarterly PMHAL System Servicedmu@illinois.edu

COMPLETE

2020-06-08 18:002020-06-09 07:00

Fileserver & Printing

Monthly Windows Server MaintenanceNCSA Fileserver(s) and NCSA-Print will be unavailable during the maintenance.   Business Office, HR, home, swap, etc. shares will be unavailable.  Printing will be unavailable.help+its@ncsa.illinois.edu

COMPLETE

2020-06-04 09:302020-06-04 14:46vsphere.ncsa.illinois.eduUpdated vCenter SSL certificate and trust chainManagement of VM's was unavailable while updating SSL certificateshelp+its@ncsa.illinois.edu

RESOLVED

2020-05-30 08:002020-06-01 12:00linux.ncsa.illinois.edu
public-linux.ncsa.illinois.edu
SSH password-based authentication were failing due to changes with intermediate certificates

SSH password-based authentication broke. Kerberos based auth continued to work.

help+its@ncsa.illinois.edu

RESOLVED

2020-5-28 07:002020-5-29 09:00NCSA Virtual ClassroomNodes were added, network reconfigured, and updates were applied.Student VM's were be unavailablehelp+its@ncsa.illinois.edu

COMPLETE

2020-5-22 07:572020-5-22 14:00Blue Waters ComputeMistaken cabinet removed from configuration causing unroutable configuration for HSNAll compute, all running jobs

COMPLETE

2020-05-20 10:002020-05-20 14:00DNS1/2UpgradesDNS servers will be rebooted during this time.help+neteng@ncsa.illinois.edu

COMPLETE

2020-5-14 06:00

2020-5-14 08:00NCSA WikiUpgradeWiki pages will be unavailableswrights@illinois.edu

COMPLETE

2020-05-19 09:302020-05-19 10:40netact.ncsa.illinois.eduFixing a problem with apachenetact is down.help+neteng@ncsa.illinois.edu

COMPLETE

2020-05-12 07:002020-05-12 17:00iForgeQuarterly MaintenanceAll systems will be unavailable during the maintenanceiforge-admin@ncsa.illinois.edu

COMPLETE

2020-05-04 18:002020-05-04 22:30NCSA Fileservers & Print ServersMonthly ITS Windows Server MaintenanceFileserver Shares (HR, Business Office, Home, Swap, etc.) and shared printers on NCSA-Printhelp+its@ncsa.illinois.edu 

COMPLETE

2020-04-30 9:002020-04-30 11:00Systems connection to idds-prodITS will be updating firewall settings for idds-prod.No impact is expected, but users should contact help+idds if issues occur.help+idds@ncsa.illinois.edu

COMPLETE

2020-04-22  8:002020-04-22 9:00CILogonhttps://go.ncsa.illinois.edu/CILogonServiceUpdate2020-04-22CILogon will relax name and email attribute requirement for IdPs.help@cilogon.org

COMPLETE

2020-04-21  6:002020-04-21  6:05CILogonAWS CILogon COmanage updateHTTP (80/443) and LDAP (389/636) ports will be unavailablehelp@cilogon.org

COMPLETE

2020-04-15 9:252020-04-15 10:01Campus Cluster user portalLogin access via UIUC Shibboleth was not working, while Shibboleth configurations were updated in support of new Shibboleth versionLogin access via UIUC Shibboleth was not working.help+its@ncsa.illinois.edu

COMPLETE

2020-04-14 11:082020-04-14 11:38vsphere.ncsa.illinois.eduVcenter was upgraded.Management of VM's was unavailable for 30 minutes.help+its@ncsa.illinois.edu

COMPLETE

2020-04-14 10:002020-04-14 12:00CILogonhttps://go.ncsa.illinois.edu/CILogonServiceUpdate2020-04-14DELAYEDAn incompatibility in the OIDC "getcert" endpoint was discovered. The update has been delayed.help@cilogon.org

COMPLETE

2020-04-13 13:55
RSA Authentication Manager and SecurIDRSA Authentication Manager service was turned off.Authentication using NCSA RSA tokens is no longer supported.  If you are using RSA with other organizations it should continue to work.otp@ncsa.illinois.edu

COMPLETE

2020-04-13 0900

2020-04-13 00930All Globus Services currently using RSA authenticationRSA authentication will be changed to DUO authentication

All Globus Services currently using RSA authentication


help+globus@ncsa.illinois.edu

COMPLETE

2020-04-07 10:002020-04-07 13:00Cerberus Bastions, BWBH BastionsThese systems will be migrated from using RSA to DUO for their second factor.

SSH logins on the hosts:

cerberus{1,2}..ncsa.illinois.edu

bwbh{1,2}.ncsa.illinois.edu

help+security@ncsa.illinois.edu

COMPLETE

2020-04-01 18:002020-04-01 22:00Windows Server MaintenanceNCSA Windows File & Print Servers were unavailable.  Users were not be able to access data on fileserver, or print to any printers in the building while maintenance was completed.NCSA File & Print Servershelp+its@ncsa.illinois.edu

COMPLETE

2020-04-01 9:02
RSA SecurID self-service portal, https://otp.ncsa.illinois.edu/ Portal was turned off.If you need to change your PIN on activate a new phone you won't be able to.otp@ncsa.illinois.edu

COMPLETE