The notorious SolarWinds cyber-attack propelled the concept of supply chain security into the consciousness of the wider industry and governments.
The idea that a single compromise could impact hundreds, even thousands, of organizations, including government agencies, became a huge source of alarm. This concern has been magnified following further software supply chain incidents since then, such as Log4j, Kaseya and, more recently, MOVEit.
Since the 2020 incident, believed to be perpetrated by Russian state actors, SolarWinds has been public about its plans to boost security by design across the wider software supply chain. This is also a major component of the US federal government’s National Cybersecurity Strategy.
Infosecurity recently spoke to SolarWinds’ CISO Tim Brown to discuss how to develop a more secure software ecosystem, incident response learnings from the supply chain incident in 2020 and advice for other CISOs.
Infosecurity Magazine: You have been working on the SolarWinds Next-Generation Build System, which aligns with NIST’s Secure Software Development Framework (SSDF). What are the key components of this effort?
Tim Brown: This program developed out of the SolarWinds incident in 2020. One of the unique aspects of that incident was that our source code had been compromised, not the product. We had to work out how to explain that – we said it was an attack on our build supply chain because it started somewhere along that chain of building a product, and the code was modified or inserted.
This was on day one – we were notified about the incident on Saturday [December 12], and on the Sunday night [December 13] we put out our filings. What we didn’t realize at the time was just how big the supply chain is overall. We’d previously focused on the little supply chain, which is how we build the product – all the individual components that go into the solution.
The big supply chain is the customers, manufacturing, and infrastructure of our business. You need to split the two because we build products with our little supply chain but we’re also part of other supply chains.
One of the areas where we really help with that supply chain model is using advanced techniques to do comparison: undertaking multiple builds and comparing them at the end, making sure that they match. We’ve been public about that and tried to provide examples for individuals and companies on how you can build a model that is more resilient to attack.
"There are millions of volumes of code and it’s hard to ensure every line is perfect"
IM: How important is collaborations between software companies to support secure software development?
TB: It’s very important from different perspectives. What you produce and put into your software from a software bill of materials (SBOM) perspective and how you protect from your line of code to the product or service.
We took the requirements for the SSDF and pulled them into all the components. This wasn’t just SBOM, we also looked at our processes relating to how to build products as it’s important that people go beyond the basics of ‘I need to provide an SBOM to you.’
It’s important to understand that we’ve had lots of legacy products for a long time, so we’re not looking for perfection but instead for control – we’re looking for information about how things are built, and how we protect risk. It’s very hard to expect perfection – we have [Microsoft’s] patch Tuesday for a reason. There are millions of volumes of code and it’s hard to ensure every line is perfect, so our regulations have to assume that we’re not going to be perfect.
IM: What are main lessons you have learned about effective incident response management from the supply-chain attack in 2020?
TB: Number one is to put your customers first. You need to make sure that your customers are not affected, or that they know what to do to not become affected.
Communicate what you know quickly but don’t overestimate what you know at a certain point in time – don’t guess. Make sure you present the facts well.
Make sure you are careful on what you are putting out – every word matters and every word that you write has a lot of impact, so ensure that you’re reviewing the words before you go public.
Also, own the incident and if it’s as big as our incident, make it front and center. When you came to our website [during the SolarWinds attack] the first thing you were put through was the incident page. We knew that’s what people needed information on.
We took an approach of being transparent, and that was a unique approach at the time. And it's worked as we’ve gained respect from our community, our customers.
Another lesson is to make sure you bring the right help in. We brought CrowdStrike in first. They weren’t development experts, but they’re fantastic at threat hunting and threat analysis. Then we brought in KPMG’s forensics team – they were fantastic from a development front. Then we needed to be able to reach all the global defenders, so we brought in Chris Krebs [former Director of CISA] and Chris gave us access to the world.
"Make sure you are careful on what you are putting out – every word matters"
IM: We have seen other major supply chain attacks since, what is your advice to CISOs to deal with the issue of software supply chain vulnerabilities?
TB: Make sure you understand your business-critical services, and the components inside of those that make up your large supply chain. Your business-critical areas should have a map that shows your third parties. Then, when you discover a third party is being widely exploited, you know where it affects you and if it affects your business-critical areas.
From the other side, what the software companies need to be concerned with is understanding where all of their libraries exist, what open-source software and third-party libraries they are utilizing, and make sure they have a catalog of that.
Then, you need to be able to react and test the reaction if one of your major libraries has a vulnerability and is exploitable in your system; for example, how do you get an update out or how update your servers as quickly as possible.
IM: What are your biggest concerns within cybersecurity today?
Nation-state actors are a big concern. Because of the Ukraine war, Russia has stopped participating in shutdowns and they seem to be linking up more tightly with their organized crime components in the country. China is very active in utilizing vulnerabilities and on stealing IP to move their agendas forward.
Also, the supply chain that we have talked about, because I think some of the low-hanging fruit of security will be those product libraries that are being utilized by many that don’t have the maturity needed to build them securely. So they’ll be a ripe target moving forward.
IM: What are the biggest successes that you think the cybersecurity industry is experiencing today?
TB: We’ve actually done very well in a lot of places that we don’t always give ourselves credit for. We’ve got to a really good point in the identity world of a common identity being used for many places.
With our network controls and sharing of threat intel and tactics, techniques and procedures (TTPs), we’ve done a really good job of not making that proprietary and sharing across the industry.
I see a lot of advances. We’re still going and we’ve got a great community of small and large security vendors that are helping to push the envelope in new technology and new ways to do things. We’re not being passive in implementation. If you just look at the last few weeks – we’re getting more quantum ready, so we’re becoming more proactive in how we look at security.
IM: If you could give one piece of advice to fellow CISOs, what would it be?
One of the things CISOs need to do is be ready for their job because sometimes things change instantly. Take our incident – on December 11, 2020, my job was very different to December 12, 2020. After December 12 I had to be external, I had to be able to speak to people, be out there in the world to promote our response and how we were managing the incident.
If you’re a CISO, start getting speaker training, understand incident response from a global perspective, and prepare for more board conversations. We still have a number of CISOs that sit in backrooms, and work with data and operations. This is extremely important but it’s also important as a CISO to grow your skills and be able to take on some of the other challenges that you have from the business perspective as well as from the incident potential perspective.