...

The reports are discussed at regularly occurring meetings between the mForge administrators and the security teamSystems and Security teams. These meetings are also used to discuss other intelligence gathered by the NCSA security team; such as information gathered through threat hunting, other security intelligence gathering systems and any vendor or community provided notices and intelligence. Items that require action on the part of the Systems Team are communicated via the NCSA ticketing system. High priority items are also followed up directly with a system administrator and with management.

Major configuration changes or the addition of services require a vetting of the changed system and services by the NCSA Security team. The Security team reviews the configuration for adherence to best practices and runs vulnerability scanning tools against the changed service.

...

Vulnerability Response

Standard Updates

Standard patches are performed during regular quarterly outages and include basic OS updates (including security patches) and other updates from vendors. A full vulnerability scan is performed again after any of these planned maintenances (PM). Some software patches do not require downtime and may be done sooner than the next quarterly. 

...

These quarterly PMs are generally done during weekends and off hours with prep work to minimize customer downtime. However, they may can require a full service outage.

Urgent Updates

Urgent patches could be from a critical (See Understanding Severities in the SECURITY JIRA Queue) security vulnerability that cannot be mitigated or for something that destabilizes the system or a component. subcomponent. After the update a full vulnerability scan is run for confirmation.

When possible these are done in a rolling update to avoid complete system outages, but it can require and entire unplanned outage.

...

In these cases customers are promptly notified of the plan, and the outage will be posted on the NCSA service status page unless further discretion is required by the customer.

Special Requests

Customers may have special requests for updated packages or libraries. If this is any change beyond a simple update of a minor software version, it goes through the standard change control process. Otherwise a ticket with the request is sufficient, and it is at the Systems Team's discretion how and when to roll out the update.

Vulnerability Response

...

• deployment of necessary patches/fixes/mitigations
• communication of urgent actions to customer
• development of coordinate deployment/corrective actions with customer going forward

...

• these are done on weekends or off-hours
• preplanned schedule of actions and work list
• prep work as possible to minimize the necessary downtime
• work will be done on a rolling reboot method during 2 of these
• work requiring full service outages will be scheduled on a semi-annual basis
• attempts to preemptively address overall system issues

...

• these may follow some specific testing on that service/node for regression
• will be deployed in priority order, with appropriate service consideration

...