Milk spoils. Iron rusts. And software goes bad. Yet the difference is, with the first two, you know the change has occurred. With software, those changes are not always obvious.
Your 5,100 Binaries Went Bad
There is no way to prevent software from ‘going bad’. As with all products, bugs, defects, or other issues are bound to happen at some point. No one and no code is immune from these issues. But when 5,100 software components in your organization went bad last year (meaning new security vulnerabilities were discovered in them), it is all too likely that no one noticed.
Earlier this year, I took a deep dive into the analysis of software supply chains that fuel high velocity development practices and IT operations. The analysis revealed that some of the largest development organizations were consuming an average of 240,000 open source components to expedite development, accelerate innovation, and meet time to release deadlines. While reaping huge benefits from this consumption, these same organizations were unintentionally growing their technical and security debt.
Here is a summary of the their average debt:
The analysis showed that over 15,000 (7.5%) of the open source components being consumed by these organizations in 2014 had known security vulnerabilities. Of those 15,000 components, an average of 66% (9,900) had known vulnerabilities dated 2013 or older. That means, they were known vulnerable components (‘bad‘) before they were downloaded.
The remaining 34% (approx. 5,100) might have actually been ‘good‘ components at the time they were downloaded by development teams from public open source repositories, but at some time during 2014 a new security vulnerability was discovered and a CVE identifier was assigned.
Even with the best controls in place to protect developers from downloading open source components with known vulnerabilities, the fact that new vulnerabilities are continuously discovered – about 50 new CVEs are announced daily – means that continuous integration and delivery practices also need to be complemented by continuous governance and visibility practices.
We’re Not Alone
Software development is not the only industry facing such issues of good things going bad. I bet all of us have received notification from our car’s manufacturer that a certain part is being recalled. While the part was known to be safe when the manufacturer sourced it from suppliers, placed it on the car during assembly and shipped it, discoveries of defects are often found years after the vehicle has been on the road. The same scenario has been played over and over in the food, pharmaceutical, defense, toy, electronics, and other industries.
The difference is those other industries are known to have more mature supply chain practices where they can track and trace every part or ingredient ever used in every product they have shipped. And when a part goes bad, they can recall and repair that defective component.
In software, track and trace practices are still in the earliest stages of maturity. In a 2014 survey of 3,300 development professionals, 63% stated that their organizations have incomplete or no records of the components used in their applications. This lack of traceability means that even when good components go bad, development and IT operations teams have no way to quickly find the defective component and replace it with a better, safer version – dramatically extending mean times to repair, or in the case of no action, growing their technical and security debt. And when applications with those defective components are in the hands of customers, the ability to ‘recall’ the impacted software or distribute newer versions of the application are hindered.
Traceability via a Software Bill of Materials
Unlike ‘bill of materials software’, which is used in traditional manufacturing supply chains to list the suppliers and parts used in a product, a ‘software bill of materials’ (BOM) is an inventory of the third party and open source components used to build an application.
As noted on Wikipedia, “The concept of a BOM is well-established in traditional manufacturing as part of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. If defects are later found in a specific part, the BOM makes it easy to locate affected products.
“A software BOM is useful both to the development organization (manufacturer) and the buyer (customer) of a software product. Builders often leverage available open source and third-party software components to create a product; a software BOM allows the builder to make sure those components are up to date and to respond quickly to new vulnerabilities. Buyers can use a software BOM to perform vulnerability or license analysis, both of which can be used to evaluate risk in a product. Understanding the supply chain of software, obtaining a software BOM, and using it to analyze known vulnerabilities are crucial in managing risk.”
A software bill of materials not only inventories what is used, but in some cases it also syncs with real-time component defect data to indicate which components have known vulnerabilities or license risks. Various software supply chain automation tools expand the bill of software much further, alerting stakeholders automatically when a defect alert occurs. In advanced tools, the entire software lifecycle is automated to ensure that defective components are avoided and continuous monitoring instantly identifies newly announced vulnerabilities as soon as they are discovered.
Breaking Bad
As mentioned earlier, there is no way to prevent software from ‘going bad’. No one is immune to these issues. Therefore, we need to focus on our responses to these issues.
To paraphrase Vince Lombardi, “It’s not how you get knocked down, it’s how you get up”. Following the OpenSSL vulnerability announcement in 2014 (or pick any other new security vulnerability with or without an assigned logo), every development and IT operations team on the planet was asking two questions:
-
Did we ever use that open source component?
-
if yes, where is it?
For those organizations tracking their use of components with a software bill of materials, the answer was easy to find. For other organizations, weeks and often months of research were required. To get a sense of the lack of maturity around traceability practices, take a look at this image from Sonatype research showing announcements of OpenSSL vulnerabilities in products following the initial vulnerability disclosure in April 2014.
Automating Traceability Across the Software Supply Chain
Software supply chain complexities, coupled with the volume and velocity of open source and third party components flowing through them, are forcing us to think differently about our current practices across development and operations. We are not the first to face these challenges and we can learn a great deal from the mature practices of other high-velocity manufacturing organizations and industries.
One thing is certainly clear. Manual practices of monitoring or occasional checkpoints cannot keep pace with volume and velocity of components being consumed. And if you extend these considerations beyond just open source components and into containers, microservices, or other images used in development and operations, the problem is further out of reach for manual processes to keep pace. The volume and velocity within our software supply chains will not diminish – and without a new approach, the volume of unchecked quality and integrity of parts being consumed will continue to build up as technical debt.
But the answer is here today: automation. Automation can unleash the potential of an organization’s development capacity. Rarely is there such an opportunity to simultaneously increase speed, efficiency, and quality. Solutions that facilitate comprehensive software supply chain automation are poised to usher in the next wave in development productivity – with gains on par or even greater than possible with agile, Lean, and DevOps.
More information on the current state of software supply chain practices as well as examples of best practices being employed by software development teams, can be found in the 2015 State of the Software Supply Chain Report.
