Government agencies have encountered many obstacles as they work to better serve their communities through new software initiatives. Open source code can be an easy and cost-effective solution when developing applications that evolve with agencies and the digital transformation they are experiencing. But software developers using open source packages in the applications they develop brings inherent security risks that can make it difficult for agencies to deliver secure application software that meets federal security standards. We talked with Steve Boone, Checkmarx Head of Product and an expert in open source code, application security, and digital transformation, to learn how agencies can transform the services they deliver with modern software applications.
Government Technology Insider (GTI): What are the benefits for federal agencies of using open source when creating software applications?
Steve Boone (SB): Open source in general provides considerable value to developers, the most obvious one being reusability and speeding the time it takes to develop functioning software. If we look at recent statistics from Forbes, 96 percent of the software projects being built today are leveraging open source packages in some fashion or another.
We’re constantly putting pressure on developers to take on more responsibility. Not only are they expected to develop great, secure software, but we’re also asking them to test that software, deploy, troubleshoot, and provide support for that software.
The benefit of open source when developing code is that we don’t have to reinvent the wheel. We can just go grab the pieces, parts, and the tools that we need from the open source community in hopes of speeding our time to market as well as speeding our time to value for whomever our end users might be. That’s the real value of having open source software, and it’s quite transformational.
GTI: What are some of the risks associated with open source software?
SB: Open source is extremely prevalent, but obviously with all of that goodness come challenges and risks. When it comes to the most commonly used open source packages, there’s this sense of security that if everybody is using it, then it must be safe. What we’re learning, though, especially as we’re entering this new phase of open source risk, is how to adequately address that risk while still supporting the use of open source in today’s codebases.
Previously, when we talked about open source risk, we would talk about known vulnerabilities or CVEs. There are many tools out on the market that can help you identify what known risks are associated with an open source package. Agencies can use these tools to look up risks, compare them to their risk tolerance, and then prioritize which risks they need to address and which they can allow.
However, we have this new type of risk – malicious code in open source packages. The difference is that known vulnerabilities might affect you, but malicious code vulnerabilities will absolutely affect you. And no agency has any tolerance for anything malicious.
What we’re seeing is that threat actors are purposefully targeting the most commonly used open source packages, because if they can infect those, then they majorly expand their attack surface. Since many organizations may be using a popular open source package, if it becomes infected with malicious code, then they may be unknowingly consuming that malicious code. Then, whatever type of attack threat actors want to pull off, whether it’s malware, ransomware, or even bitcoin miners, they have a better opportunity of exploiting the most organizations possible.
Log4j is a great example, but it isn’t really malicious code. This was open source code that was being used by an extremely large number of organizations as part of their codebases. Some attacker realized that they could exploit an unknown vulnerability in Log4j, and once that vulnerability was exploited, everyone that was using Log4j was now vulnerable to exploitation as well. It caused massive disruption in the industry because organization had to first determine if they were using the vulnerable Log4j versions in any of their applications, then they had to stop everything, and patch all applications with the latest updates to Log4j. To date, organizations are still dealing with the aftermath – nearly seven months later.
Today’s attackers understand that if they can infect popular open source packages it will definitely make you regret pulling it into your codebase. Many of these now-malicious packages are designed to spread, cause chaos, and cause damage, not only to the applications running the malicious code, but can even infect developers’ computers.
There can be several different ways these malicious code packages can affect your organization. Threat actors can basically hold organizations hostage by ransoming their data and requesting millions of dollars to regain access. Or they can affect your organization slowly overtime by creeping through your network and seeing what other servers they can communicate with and what information might be valuable for them to exploit. These attacks are less identifiable, but equally damaging.
GTI: What are the biggest challenges when trying to design application software that meets the CISA standards and federal requirements for source code analysis?
SB: The executive order that came out last year really focused on creating transparency around open source usage. Nowadays, if you’re selling software to the federal government, you must provide them with what we call a Software Bill of Materials (SBOM). That SBOM is a full list of the open source packages that are used in your applications. This requirement in and of itself, is new and changing over time.
But I think there are some important things that people should understand when using the SBOM. One is the importance of understanding what your open source usage is so that, in the event of another Log4j, you can determine if your applications are affected. Yes, it’s a requirement to have an SBOM do business with the federal government. But even if your job isn’t to directly do business with the federal government, having this bill of materials is extremely important so that if something like Log4j happens again, you can easily identify the open source packages in your applications. This speeds the time to identify where the issues may lie and consequently speeds up the time it takes to fix those issues.
We’re seeing lot of other valuable uses of SBOMs, even internally. Different teams internally share what’s known as second party packages, which are source code packages that another team built within the same organization. This differs from third-party packages, which are traditionally considered as open source. By sharing an SBOM that includes a second-party package, the other team has a better understanding of any open source packages that are in their project, and what potential issues they should be concerned about. This way you don’t simply assume that the team who developed the code did all the correct security checks.
I think one of the big things that the Executive Order on Cybersecurity also mentions is how we can create a response playbook to be better prepared next time something like Log4j happens. This is the new normal; we have to be able to respond and react appropriately to attacks on the open source supply chain.
GTI: What are some of the best practices for reducing and eliminating the risk of using open source code in federal application software?
SB: A really popular project that Google is sponsoring right now is a framework called the Supply Chain Levels for Software Artifacts, or “Salsa” (SLSA). What they did was pull together like-minded companies to address supply chain security problems by documenting what they believed to be the real challenges and then motivated the industry to solve those problems. Under the SLSA framework, they came up with a couple of different attack vectors that they think are high risk for supply chain attacks and dependencies was high on the list.
In modern application development today, especially with agile and DevOps best practices, teams are using Continuous Integration and Continuous Deployment tools to automate a lot of the CI/CD processes. The CI process allows organizations to continuously integrate new software builds that may have added new features or fixed previous code issues. Once the software is built, CD processes allow you to easily deploy those builds as a continuous process.
The challenge with integrating CI/CD tools into your other software development processes is that it’s a lot of work. One of the ways you make it easy is to leverage plugins. And these plugins help tie your build/deployment servers to other parts of your business, such as your issue ticketing and tracking tools, your security processes, and your application security testing tools as part of an overall AppSec program.
Another best practice for developers is protecting your internal package repositories. Companies like Nexus, Artifactory, and JFrog produce tools to store and safeguard valuable artifacts. These tools are designed to help protect the software that’s being made and deployed to the end users. If malicious actors can breach these internal systems, they can possibly swap out their malicious code into those repositories by replacing your packages with their packages. Without these types of protection measures in place, hijacking internal repositories ends up becoming a considerable risk as well.
A third best practice is to secure application dependencies, or those open source packages that are part of your codebase. When we look across the many attack vectors, the source code, the CI/CD servers, and the artifact repository; those all have several layers of protection in place. Most of them have two factor authentication and they’re behind an organization’s firewalls. That’s a much more sophisticated, much more difficult attack scenario for anybody to pull off. Not to say that it’s not possible, because it is. But the biggest risk is those third-party dependencies.
As developers, we have basically been inviting a stranger’s code into our organization and, in light of that, our mantra at Checkmarx has been: “We don’t take code from strangers.” What that really means is that we don’t trust code created by strangers and we need to make sure that we can trust it and that it’s safe before making it part of our applications.
I really would encourage companies to think about: what is your process for vetting open source code before it makes it into your organization’s applications? A lot of developers just pull in whatever open source they want while developing applications and then they go through the process of getting it approved. But at that point, especially if the code is malicious, it’s often too late. We need to arm our developers with solutions that allow them to become immediately aware of malicious packages, that track open source contributor reputations, that provide behavior analysis of suspect packages, and delivers continuous updates so they are increasingly aware of packages they don’t want being part of the applications they are delivering to the public.
GTI: Is there anything else you’d like to add?
SB: We want to make the best and most secure application software. We’re trying to give our developers the best tools to do that, but at the same time, open source is so prevalent today. Nobody alone can monitor all the different packages and all the updates happening at once, and that’s one of the biggest challenges developers are facing.
We give our customers that one-click ability to generate an SBOM for their applications so that, at any given time, we have a full list of the open source packages that applications use. And as we find new vulnerabilities, as they inevitably make themselves known, when we get the next Log4j or the next Spring4Shell, we can proactively notify users of those vulnerabilities. We’ve got one of the best research teams in the market. We deliver remediation advice and how to fix those issues within hours of finding the actual vulnerability itself. In the case of Log4j, we were able to help organizations safely and quickly overcome those issues and come out better protected and better educated as a result.
Learn more about how to what SBOM is and why you should care here.