Scope and Purpose

This standard applies to all systems and networks within NCSA's control and ownership where there is not a more specific system-level standard. Exceptions can be requested for devices that cannot be scanned or updated.  Note that for end of life or support operating systems (OS) an exception must include a concrete plan for an OS upgrade or decommissioning.  NCSA policy requires all systems to have a plan to identify and remediate security vulnerabilities.

In addition to the use of Qualysguard automated tools for vulnerability management of production systemsexternally-exposed and internal assets, the NCSA Incident Response and Security Team Office performs active threat hunting on the entire NCSAnet vulnerability discovery to detect misconfigurations, systems, system applications and networks that are not compliant with University policies, and general system cyber weaknesses. The goal goals of this program are (1) to detect issues more broadly than just on all assets and (2) for all NCSA cyber-infrastructure and to investigate more deeply than simple checklists for NCSA's most critical infrastructure.

Threat Hunting Activities

Automated scans by the NCSA Incident Response and Security Team allow detection of devices on every routable network to check for common problems and include:

• port scanning
• authentication attempts using common username/password combinations
• input validation testing (eg: web pages, APIs, logins)
• vulnerability identification

Manual investigations of critical infrastructure involve similar activities but also additional process logic testing and validation.

Responsibilities

We all have a shared responsibility to protect the systems and data we are responsible for and to follow NCSA and University policies and standards. Recognizing that hunting against systems may sometimes be disruptive, NCSA IRST take extra precautions when hunting from privileged positions  from inside the NCSAnet (such as limiting scan rates and carefully monitoring for disruptions). Furthermore, NCSA IRST will inform system owners before any directed or manual penetration testing to help avoid tests at during a critical operational window, though such testing will not generally perform any actions that a malicious threat on the Internet could not do at any time. And if one of the automated scans is causing disruptions IRST will work with service operator to mitigate the effects and prevent future problems

It is also everyone's responsibility to report security incidents. If you believe your system is being attacked please follow NCSA Incident policies, regardless if you think it might be a NCSA IRST system scanning or attacking. https://wiki.ncsa.illinois.edu/x/l5BgAQ Even if it turns out not to be a real attacker, it demonstrates organizational responsiveness.

Resolution

The security team is not the owner of risks, and sometimes neither is the system operator. When problems are identified that are not simple policy compliance, IRST will make recommendations on remediation. For simple issues this may just be a single ticket, and for more complex issues it may require meetings and a remediation plan from the service operator. Significant risks will also be raised to the Security Office when the attention of senior management is appropriate.

Vulnerability Identification

The NCSA Security Office performs vulnerability scanning on NCSA cyberinfrastructure assets.  Discovered vulnerabilities are investigated, categorized and documented in NCSA’s ticketing system for remediation.  These vulnerabilities are discussed between the NCSA Security Office and system owners and/or system managers during regular meetings.

Vulnerability Management

Vulnerabilities must be acknowledged and a remediation plan made according to the severity of the vulnerability and criticality of the asset.  For all NCSA assets it is required by NCSA policy that detected vulnerabilities are remediated and that the NCSA Security Office is granted access for authenticated and non-authenticated vulnerability scans and to manually verify vulnerabilities. 

NCSA must be able to identify the assets on which the vulnerability was found and who is responsible for those assets. Such data is normally acquired through the NCSA system vetting  process. Additionally, the information should include the classification of that asset in terms of risk and criticality.

Vulnerability Remediation

The NCSA Security Office group will work to incorporate the context of the finding by noting the classification of the asset and the severity of the vulnerability.  Additionally, other factors may be considered such as the role of the vulnerable asset within NCSA.  These factors will be used to ultimately determine the actual severity of the vulnerability.

The severity of the vulnerability will determine the required time for remediation.  Vulnerabilities classified as “critical” need to be patched or remediated within 4 hours.  These are vulnerabilities with the potential to have a severe impact on the NCSA.  Vulnerabilities classified as “high” need to be patched or remediated within 24 hours.  There are vulnerabilities with the potential to have significant impact on the NCSA.  For lower vulnerability classifications: patching should occur within a planned maintenance cycle or a separate plan is made to patch or remediate the vulnerability within a reasonable timeframe.

In some rare cases, the vulnerability cannot be patched and remediation may not be adequate.  An exception must be filed with the NCSA Security Office.  The exception may generate significant risk and may need to be reported to NCSA Management for acceptance.

ACHE Vulnerability Remediation

The Advanced Computational Healthcare Enclave (ACHE) has a separate policy for vulnerability remediation stated below.

Standard Updates

Standard patches are performed during regular scheduled outages which occur at least twice a year and include basic OS updates (including security patches) and other updates from vendors. A full vulnerability scan is performed again after any planned maintenance (PM). 

These quarterly PMs are generally done during weekends and off hours with prep work to minimize customer downtime. However, they may require a full service outage.

Urgent Updates

Urgent patches could be from a critical (See Understanding Severities in the SECURITY JIRA Queue) security vulnerability that cannot be mitigated or for something that destabilizes the system or a subcomponent. After install the efficacy of the changes are tested.

When possible these are done in a rolling update to avoid complete system outages, but it can require an unplanned outage. In these cases customers are promptly notified of the plan, and the outage will be posted on the NCSA service status page unless further discretion is required by the customer.

References

Understanding Severities in the SECURITY JIRA Queue https://wiki.ncsa.illinois.edu/display/SecOps/Understanding+Severities+in+the+SECURITY+JIRA+Queue