In response to the Apache Log4Shell vulnerability, we have compiled a list of the most frequently asked questions we are receiving from clients and the strategies we are seeing pursued across the market. Protiviti is monitoring this event closely and will continue to update this blog post to reflect the most accurate information.
Last update: 1/7/2022
What is Log4j?
Log4j is a Java-based logging framework that is widely used by enterprises that develop applications or products. It is open-source and has been downloaded over 400,000 times.
What is the Log4j vulnerability?
A vulnerability (“Log4Shell” or CVE-2021-44228) was released on December 10, 2021, which impacts a popular logging library. This vulnerability allows attackers to exploit systems and applications which use Log4j by abusing the way a component (the Java Naming and Directory Interface or “JNDI”) operates. The vulnerability was given a CVSS (Common Vulnerability Scoring System) ranking of CRITICAL (10 of 10) due to the ease of exploitation and the possibility for attackers to obtain direct access to the underlying systems impacted by this vulnerability.
Additionally, Version 2.15.0, originally thought to be a potentially more secure version during the early stages of this event, has been identified to be susceptible to a Remote Code Execution vulnerability as detailed in (CVE 2021-45046) and reported by Apache (see additional detail here). (updated 12/17/21)
Version 2.16.0 was found to be vulnerable to a denial of service (DoS) issue (CVE-2021-45105 rated a 7.5 CVSS criticality) on December 17th. Apache has released a new version (2.17.0) that is thought to address the identified DoS issue. (update added 12/18/21)
Verison 2.17.0 was found to be vulnerable to a remote code execution (RCE) issue (CVE-2021-44832). The identified issue is currently rated a 6.6 CVSS criticality, despite the remote code execution impact, as it requires an attacker to have permissions to modify a configuration file to trigger the exploit. Apache has released an updated version (2.17.1) to address this identified issue. (Updated 12/30/21)
How long has Log4j been vulnerable?
Affected versions by the Log4Shell vulnerability include Apache Log4j 2.0 through 2.17.0. Version 2.0 was released in 2014. (updated 12/30/21)
How is this different than other vulnerabilities?
CVE-2021-44228 was released as a “zero-day” vulnerability, meaning that no immediate patch or fix was available before the vulnerability was released. This vulnerability is also not specific to a single application or operating system. Instead, it impacts a very popular logging framework that is used on many systems, applications and embedded devices. The ubiquitous nature of the Log4j framework makes it difficult for organizations to quickly identify and remediate all potentially vulnerable systems.
Additionally, the vulnerability identified is relatively easy for attackers to exploit. This made the process of “weaponizing” (developing a working tool or script to exploit a known software flaw) the vulnerability nearly instant. This rapid weaponization allowed attackers to begin scanning for, and exploiting, impacted systems within hours of the vulnerability release.
What steps should we take now?
It is recommended that organizations take immediate action to identify any system that has Log4j installed and update to the latest known version (2.17.1) as quickly as possible. In instances where vendor products are in use and direct version updates cannot be made, contact your vendor to seek guidance for when updates will be made available. An informal list of software that has been reviewed for potential impacts is being maintained by the US Cybersecurity and Infrastructure Security Agency here. (updated 12/30/21)
How do I detect which devices/applications in my environment have Log4j installed?
If your organization has a robust systems management solution in place that supports bulk querying of filesystem indexes, it is recommended to search for the vulnerable software by name. While there have been numerous versions, the software naming convention has remained consistent, beginning with “log4j-“.
In addition to the above, enterprise vulnerability management solutions (such as Qualys, Nexpose, and Tenable) have released updates to allow organizations to scan for the Log4j vulnerability. It is advised to leverage these scanning tools as an additional mechanism to identify potentially vulnerable systems. Due to the ease of exploitation and potential impacts associated with this vulnerability, it is advised that organizations prioritize any Internet-facing systems for scanning efforts as quickly as possible.
For organizations that may not have access to enterprise vulnerability management solutions or wish to enhance/provide additional validation against their own reporting, the Cybersecurity and Infrastructure Security Agency (CISA) recently released an open-source scanning solution found here that can be utilized to identify the log4j Remote Code Execution Vulnerabilities (CVE-2021-44228 & CVE-2021-45046). (updated 12/27/21)
Protiviti has identified that some applications and systems, despite being vulnerable to the Log4Shell issue, are not flagged by current vulnerability scanning checks. In those situations, manual testing may be required to effectively validate the susceptibility of individual systems, applications, or devices.
How do I know what products and services have been affected?
While not guaranteed to be a comprehensive and all-encompassing list, the US Cybersecurity and Infrastructure Security Agency has put one together here.
I heard a patch was released but was still vulnerable, what happened?
An initial patch was released on December 13, 2021 as Apache Log4j version 2.15.0, but was quickly found to be vulnerable in some non-default configurations. Additionally, new intelligence as of December 17, 2021 identified a Remote Code Execution (RCE) vulnerability in Version 2.15.0, effectively rendering it ineffective as even a short-term mitigation for the Log4Shell vulnerability. The vulnerability released specific to this version (CVE 2021-45046) is being updated from a CVSS criticality score of 3.7 to 9.0. Version 2.17.0 has since been released and should be considered the secure version for update at this point. See additional details as provided by Apache here.
The original vulnerability identified in the updated Log4j version (2.15.0) was thought to be less severe than the original vulnerability released, as it may have only allowed attackers to exploit systems impacted with a denial-of-service (DoS) attack. However, the identification of a remote code execution (RCE) vulnerability renders Version 2.15.0 as vulnerable as previous versions.
Version 2.16.0 was also found to be vulnerable to a denial of service (DoS) issue (CVE-2021-45105 rated a 7.5 CVSS criticality) on December 17th. and version 2.17.0 was found to be vulnerable to a remote code execution (RCE) issue (CVE-2021-44832 rated a 6.6 CVSS criticality) on December 28th. Apache has released an updated version (2.17.1) to address these identified issues. Apache has released a new version (2.17.0) that is thought to address the identified DoS issue. (updated 12/30/21)
If my IT teams updated Apache to version 2.15.0, do I have some level of protection until we can upgrade to version 2.16.0?
Although some level of protection may be provided by upgrading to 2.15.0, a complete upgrade to 2.16.0 should be performed immediately. Per CVE-2021-45046, version 2.15.0 was “incomplete in certain non-default configurations. This could allow attackers… to craft malicious input data using a JNDI Lookup pattern resulting in a denial of service (DOS) attack.”
***Critical Update as of December 17, 2021*** CVE-2021-45046, the vulnerability identified as a denial of service issue with the 2.15.0 version of Log4j, has been raised from a CVSS criticality score of 3.7 to 9.0. Version 2.15.0 is now known to be vulnerable to Remote Code Execution (a vulnerability type that allows attackers access to the underlying systems via exploitation), similar to the original Log4Shell issue identified. Teams should now assume that 2.15.0 provides no level of protection against exploitation. See additional details as provided by Apache here. (updated 12/17/21)
***Critical Update as of December 18, 2021*** Version 2.16.0, originally thought to address all known issues with the Log4j package, was found to be vulnerable to a denial of service (DoS) issue (CVE-2021-45105 rated a 7.5 CVSS criticality) on December 17. Apache has released a new version (2.17.0) that is thought to address the identified DoS issue. (updated 12/18/21)
***Critical Update as of December 30, 2021*** Version 2.17.0 was found to be vulnerable to a remote code execution (RCE) issue (CCVE-2021-44832 rated a 6.6 CVSS criticality) on December 28. This vulnerability requires an attacker to have permissions, or access, to the affected Log4J configuration files to craft a payload and trigger the exploit. It is currently thought to be less impactful than previous issues due to this required access, but should still be prioritized for remediation as quickly as possible. (updated 12/30/21)
Is this issue being actively exploited by attackers?
While the full global impact of this vulnerability will likely not be understood for some time, several entities are tracking active campaigns by attackers related to Log4j. Microsoft continues to publish updated activity (here) and has stated that multiple threat actors, including Nation-State and ransomware access brokers, are actively attempting to leverage the known vulnerability and have weaponized exploits to obtain access to systems.
What should our security operations team do to monitor for exploitation activities?
Security teams are working tirelessly to identify instances of Log4j in their environments and push the latest updates to impacted systems. Unfortunately, new vulnerabilities continue to be discovered in updated versions as the Apache team works to quickly address the various issues.
In order to help identify exploitation attempts or post-exploitation activities during this rapidly changing event, security technology vendors are developing detections for attempted and post-exploit detections of this vulnerability, and for indications that a system has been compromised. Be aware of the following:
IoC’s are being regularly updated and can be found here on the official CISA GitHub repository
- Suspicious shell command execution
- Potential cryptocurrency mining activity
- DDoS bot activity
- Untrusted file execution
- Remote activities (to include RCE)
- Privilege escalation or data exfiltration
It is highly recommended to continue to check with your vendor for the latest updates and information and ensure that in-place security controls are leveraged to their full extent, including:
- Configure perimeter controls (i.e WAF, IDS/IPS) to block Log4j related IoC signatures
- Ensure logging is available for visibility, detection of anomalies and response of areas within the infrastructure that may be affected
- Where possible, leverage endpoint visibility functionality with a prioritization on any systems that may be Internet-facing.
(updated 12/18/21)
How do I know if my organization is vulnerable?
This vulnerability is not limited to organizations that develop applications using java. The vulnerable logging component is often bundled with commercial platforms and software packages. Systems that are impacted range from desktop applications to network appliances and embedded systems. It is recommended that you reach out to individual vendors for more information, review the informal list of software maintained by the US Cybersecurity and Infrastructure Security Agency here, and scan your systems for the Log4j vulnerabilities.
How do I know if a vulnerable system has been compromised?
If a vulnerable system was successfully exploited, there will be evidence in the web request logs on the affected server. You should review your logs for any Indicators of Compromise (as this is a new event, IoC’s are being regularly updated and can be found here on the official CISA git repository). It may become necessary to review your edge router or firewall logs for further evidence of successful compromise. (updated 12/18/2021)
As you work through this event, a point-in-time threat hunt or compromise assessment may be beneficial to determine if a compromise occurred or is ongoing.
What should we do if we suspect a system has been exploited?
If you believe that systems in your environment may have been exploited, it may be time to initiate your incident response plan. While the scope and severity of response will depend on the business criticality of the systems, it is highly likely that an active response will be needed to contain, eradicate, and recover the impacted systems. If you do not have an incident response plan or need assistance in responding to a potential incident, Protiviti provides emergency cyber incident response services via: IR@protiviti.com.
Are OT systems impacted by the Log4j vulnerability?
Several OT systems and applications are impacted by Log4j; we recommend that IT security teams work with their contact at OT sites to review impacted product lists such as CISA’s GitHub page. Given the ubiquity of products impacted by Log4j, your OT environment may be vulnerable. Security, backup, and enterprise systems such as AD in the OT domain may also be vulnerable, so be careful to ensure that you validate that shared services like SIEM, NSM applications, Firewalls, Backup systems, are not vulnerable. (added 1/7/2022)
What should I do if my OT systems are vulnerable?
Work with network and engineering teams to determine which systems are externally accessible and prioritize patching those systems first. When scheduling a patch date, work with your engineers to ensure that all availability and safety concerns are addressed. Once external exposure is addressed, work internally to patch systems in a manner consistent with existing patch procedures.
If patching is not an option, some additional mitigations may include:
- Using a firewall to prevent remote call to other services. (Note: This does not eliminate the vulnerability but can reduce the size of the attack surface)
- Disable outbound connections from vulnerable systems to the Internet (i.e. ensure segmentations is effective)
- Configure WAF to detect known Log4j exploitation entries (added 1/7/2022)
How can I tell if my OT systems have been compromised and what do I do if they have?
If a vulnerable system was successfully exploited, there will be evidence in the web request logs on the affected server. Review logs for any Indicators of Compromise (as this is a new event, IoC’s are being regularly updated and can be found here on the official CISA git repository). If the OT system has been compromised, begin incident response procedures and failover to a backup system (if available). Compromised OT systems should have configurations reviewed to validate that unsafe set points and configurations have not been set. (added 1/7/2022)
How can I provide assurance to management that our OT Systems are safe?
Validate that there is no external exposure by scanning your external footprint for log4shell vulnerabilities and comparing your software list against CISA’s list of impacted products. Be upfront with management about any visibility or culture challenges that may prevent a full viewpoint. Provide clear timelines for any patching or mitigations that are being implemented.
You should be prepared to provide management answers to the following questions:
- Who is leading on our response?
- What is our plan in remediating log4j?
- How will we know if we’re being attacked and can we respond?
- What percentage visibility of our software/servers do we have?
- How are we addressing shadow IT/appliances?
- Do we know if key providers are covering themselves?
- Does anyone in our organization develop Java code?
- How will people report issues they find to us?
- When did we last check our business continuity plans (BCP) and crisis response?
- How are we preventing teams from burning out? (added 1/7/2022)
Mike Alessi, Caleb Davis, Derek Dunkel-JahanTigh, Shinoy George, Charles Godwin, Wesley Lee, Mike Lefebvre, Uriah Robins, Mike Weber, Chip Wolford also contributed to this post.
To learn more about our incident response, attack and penetration, and other cybersecurity consulting capabilities, contact us.
