In December 2021, a vulnerability with a severity score of 10 out of 10 in a widely used logging library affected the security of digital systems across the Internet. The vulnerability was in the Log4j logging tool. This article explores the importance of knowing and documenting the usage of different components in your software.
log4j is an open source Apache logging framework that developers use to keep a record of the activity within an application. The open source software creates a built-in log or record of the activity that software developers can use to troubleshoot problems or track data within their programs.
You can think of this software as a diary. Its job is to keep records of how online services and applications are being used.
The vulnerability within Log4j has been coined Log4Shell. It enables a remote attacker to take control of a device on the Internet if the device is running certain versions of Log4j 2.
A former cybersecurity commissioner for the Obama Administration says hackers have the capacity to, “Spy on the activities of the users of [affected] systems. They have the capacity to use that system to island-hop into other systems. They have the capacity to become disruptive. It really varies.”
Within four days of the vulnerability being discovered, 840,000 attacks were launched and some hackers had already developed tools that automatically attempted to exploit the bug. During the first 72 hours after this vulnerability went public, reports documented up to 100 attempted attacks per minute.
A high-profile attack deriving from Log4j was launched against one of the largest Vietnamese crypto trading platforms Onus. Hackers successfully planted backdoors to retrieve sensitive databases before Onus patched its systems. They demanded a US$ 5 million ransom, which Onus refused to pay, so they put the data of nearly 2 million Onus customers up for sale on different forums.
Reports from companies such as Google and Microsoft have indicated that government hacking groups around the world are leveraging the Log4j vulnerability for attacks. According to Microsoft, state-sponsored hackers from China, Turkey, Iran and North Korea have started testing, exploiting and using the Log4j bug to deploy a variety of malware, including ransomware.
Log4Shell: A vulnerability or a ‘cluster-bomb of zero-days’?
The Log4Shell vulnerability has been described as a ‘cluster-bomb of zero-days’ because it is such a foundational component in so many software frameworks that are available to developers for reuse.
Many pieces of software have logging libraries. It is highly unlikely that organisations, especially big corporations, are coding their own logging software. There is no benefit to this when an open source version is easily available. Log4j is widely used for this reason and, consequently, Log4Shell has impacted a very wide range of software and services from many major vendors.
Dependencies play a large role in the wide stretch of the Log4j vulnerability. According to one commentator: “It is mind-boggling how big dependency trees can get.” This is why so many users had no idea they were running affected Log4j versions between the thousands of dependencies in their stack.
As the number of Log4j exploits grew rapidly towards the end of last year, warnings were issued by several national cyber security agencies, including the US’s Cybersecurity and Infrastructure Security Agency (CISA) and the UK’s National Cyber Security Centre (NCSC).
The US’s Federal Trade Commission (FTC) has said that it “intends to use its full legal authority to pursue companies that fail to take reasonable steps to protect consumer data from exposure as a result of Log4j.”
The severity of the Log4j vulnerability has served as a wake-up call to organisations. It demonstrates you must understand what is in your software, where dependencies lie and not leverage vulnerable components, especially when safe ones are available.
Sonatype, a software company and core contributor of Apache Maven, reported that since December 10, 2021, when Log4Shell was disclosed, Log4j has been downloaded more than 10 million times (see Figure 1). It stated that nearly half of these were unsafe versions, despite the fact that fully patched and secure versions were available at the time.
Ilkka Turunen, field CTO at Sonatype, says, “The fact that we are still facing such high percentages of vulnerable downloads is indicative of a much bigger problem with supply chain security… If companies don’t understand what’s in their software, they’re unable to act with the requisite speed when threats arise – and in this instance, given the huge popularity of Log4j, this exposes them to significant risk.”
The reality is that if a vulnerable software project like Log4j has been sitting in the heart of the world’s Internet infrastructure, how many other ‘cluster-bombs’ are out there?
That cannot be determined until the risk materialises. However, you can take action to ensure that your organisation is prepared for such an event.
How can an SBOM help you?
Modern software solutions are generally made up of layers: the application code, open source and third-party libraries, the open source framework, databases and the underlying cloud infrastructure (see Figure 2).
The presence of security vulnerabilities in any one of these layers may put your solution, intellectual property, employee and customer data at risk of exploitation by hackers. The security features built into your hosted platform may get you one step closer to a secure solution.
However, if security management processes are not implemented across the remainder of your solution stack, your organisation and customers may still be at risk. So how can you protect the remaining layers?
The key question to dealing with vulnerabilities like Log4Shell is, on the day it went public, would your CTO walk into work and know whether Log4j is in your code?
The fundamental aspect to quickly remediating risk in a Log4j type situation is to know exactly what is in your software. The best practice for this is the consistent production of a software bill of materials (SBOM).
An SBOM is a formal record of components and their associated licences in a piece of software. It is analogous to a list of ingredients on food packaging. An SBOM identifies and lists software components and their licences, and the security vulnerabilities associated with those components (Figure 3).
If your organisation was consistently producing SBOMs, on the day Log4Shell was published, a reference could have quickly been made to the latest SBOM. This would have expressly told you whether Log4j is in your software and whether it is the compromised version.
A software composition analysis (SCA) tool will perform an automated source code scan to produce an SBOM. This will expressly identify all the third-party components within your software and tell you whether any security vulnerabilities are associated with them. An advanced tool will tell you whether your code calls the affected library and suggest a fix.
The National Telecommunications and Information Administration (NTIA), an agency of the United States Department of Commerce, advocates software component transparency. This agency was the main driver behind the SBOM concept back in 2019 as well as the White House Executive Order issued on May 12, 2021, on improving the nation’s cyber security.
The order states that when dealing with the various departments of the US government, buyers of software solutions must request an SBOM for each product and software solution providers should also supply an SBOM.
In a press conference in early January this year, Jen Easterly, director of the US Federal Cybersecurity and Infrastructure Security Agency, said that CISA is accelerating its efforts to create SBOMs.
Identifying the security vulnerabilities is the first step, but to reduce security risks remediations must be made quickly. Updates and patches for vulnerable software components must be implemented as soon as they are made available.
It is critical for organisations to take immediate actions to identify systems with the Apache Log4j vulnerability, implement mitigation measures, and continually monitor and remediate them. The Apache Foundation has released four patches since the first Log4Shell vulnerability was published, as previous patches left part of the vulnerability unfixed.
Continuous management is the next step to securing your solution stack. A source code scan and SBOM represent one moment in time — the moment the scan was performed. Due to the nature of modern software development, the reality is that software composition may change on an operational day-to-day basis. Therefore, SBOMs should be produced consistently, and when vulnerabilities are identified remediations should also be made consistently if a patch is available.
The ‘eternal’ debate and Log4j
The disclosure of the critical vulnerability in Log4j has sparked discussions online about the ‘eternal’ debate of open source sustainability. This discussion covers a broad collection of issues including licensing, business models, ethics, diversity and inclusion.
With the disclosure of Log4j, debates have re-surfaced on the ethics of how open source is developed, paid for (or lack thereof) and maintained. It is the maintainers, like those at the Apache Foundation, who must find the time to fix critical bugs like Log4Shell so that those upstream can secure their solutions.
Often, the voluntary contributions of maintainers to open source projects stem from a hobby. Whilst working day jobs, they can struggle to find the time to fix critical bugs, all the while facing demands from users requesting free support. Consequently, maintainer burnout is a challenge.
The fallout of Log4Shell made us all realise that we are relying on open source projects. The Internet and computing giants who arguably are “the heaviest users of open source in the world – are collectively worth trillions of dollars.” However, this wealth is not shared with the open source projects at the start of the supply chain.
Donations are commonly used as a source of funds for open source projects. For example, GitHub has set up its GitHub Sponsors platform which allows users to donate to the projects they depend on. However, this source of funding may only be sustainable for extremely popular projects.
Filippo Valsorda, a Google cryptographer and security lead of the Go programming language, notes that contributors only get small amounts of money through GitHub or Patreon, while their projects may be responsible for millions of dollars. He adds that a bug in these important systems can break the Internet, and it’s unfair to depend on people who work on these projects in their spare time without any assured rewards.
However, others believe that funding is not the issue; the real issue behind the Log4j crisis was that organisations did not know what was in their software. This lack of knowledge caused delays in patches and fixes. Without the delays, hacker exploits and security breaches may have been prevented.
Curl creator and WolfSSL developer Daniel Stenberg says, “The Log4j case is not a showcase for bad open source software funding. It is a showcase for naive and cheap users not doing their due diligence, code review, and testing before using components. Remember ‘goto fail’ (Figure 4). Silly bugs are shipped even with the greatest funding.”
This debate brings us back to the need for SBOMs. If organisations were producing SBOMs in December 2021, they would have known whether they needed to implement fixes for Log4Shell immediately.
As consumers, there is an expectation for our cars, planes and even mundane items such as food processors to have a detailed bill of materials, because if any parts fail a product recall is expected. Why are companies not doing this for their software?
In a perfect world, all maintainers and projects would be adequately compensated. However, would this ensure that bugs like Log4j do not exist? We cannot be sure; but we do know companies are responsible for ensuring the code they ship to production is safe, secure and reliable. Therefore, the best we can do is be prepared and secure each link in the software supply chain.
