What some call the worst bug in history is only a few months old. Danny Bradbury asks: Do you really think Heartbleed is over?
April 1, 2014: There couldn’t have been a more appropriate date for members of the OpenSSL team to learn that their code was giving away passwords and digital certificates all over the internet. That morning, an email arrived from Google, outlining details of what would become one of the most devastating computer bugs in history.
A flaw in the open-source code enabled attackers to use the service’s ‘heartbeat’ feature. This allows one computer to request data from an SSL record held in the other computer’s memory, to confirm that it’s still active during a session. The computer receiving the request doesn’t check the length of the requested payload, enabling the requester to ask for far more data than it really needs. This data – up to 64Kb of it – comes from memory close to the SSL record, which contains lots of sensitive information, including certificates and passwords.
This attack can be performed repeatedly, with no trace, enabling those in the know to devastate server security. What’s more, it was in existence for two years before a research team from Google discovered it.
Mending a Bleeding Heart
The remediation process for Heartbleed was troublesome for the organizations it affected. Not only did they have to upgrade from the vulnerable versions of OpenSSL, but they also had to re-obtain digital certificates from their certificate authorities. Then, they had to ask (or make) their users log out, log on, and change their passwords again. For companies that rely on making their service as easy to use as possible, that’s a big deal.
Still, at least it’s all taken care of now, right?
Not so fast, warns Tom Brennan. Brennan left Trustwave to start his security firm, proactiveRISK, on April 30. It was timely – he received lots of calls in his first week from friends, colleagues, and family, asking about Heartbleed, so he ended up writing a Firefox plugin that would check every site that a user visited.
“200,000 of the most popular websites were still vulnerable as of May 2”, he observes. Other estimates suggest that the number is even higher.
The Heartbleed vulnerability is a two-headed beast, Brennan warns. Even if a company fixes the bug, that won’t be enough, he says, unless they renew the certificates that have been compromised. Those that haven’t will still be at risk.
Many people have focused on the public facing services without taking a proper look at the internal aspects of their networks that may also be using OpenSSL, and could be similarly compromised.
“If I was to target an individual user, hooking his browser, at that point I’m able to pivot through that machine and go through to the internal network”, Brennan says, suggesting that even VoIP phones could be vulnerable.
“Call managers do login with service IDs. It’s easy to get internal organization access by leveraging a vulnerability that was believed to be external and public facing.”Tom Brennan, vice chair, OWASP Foundation and founder, proactiveRISK
Taking Responsibility
One of the biggest worries about Heartbleed is that it’s up to organizations to fix it. Technology journalists have an unwritten rule when a security flaw emerges: detail how it happened, and then make recommendations to ensure that it doesn’t happen again.
When end-users are involved, this often means reiterating basic security best practices. Use strong passwords, change them often, don’t give out credit card numbers, don’t click on suspicious links, and so on. If nothing else, it makes end-users feel a little empowered.
Unfortunately, there are no such measures with Heartbleed. It attacked organizations, rather than individual users. Users could have demanded proof that organizations were not affected, but that’s hardly helpful two years after the fact.
“There’s a class of infrastructure software where, as an end user, you are essentially powerless”, says Simon Phipps. He is the founder of open-source management consulting firm Meshed Insights, and vice chair at the Open Source Initiative, a California non-profit that focuses on building open-source communities.
Inevitably, when a security flaw of this magnitude occurs, people will ask who is to blame. The OpenSSL core development team consisted of four people, only one of whom is full time. It has a budget that ranged up to $1m per annum. Is it culpable for having not caught the bug?
Not a chance, says Phipps, who points the finger squarely at the companies using the software. “For me, rather than raising questions about the open-source process, Heartbleed raises questions about the proprietary processes of the companies that are using OpenSSL”, he says.
“If any of them spent a fraction of a second checking up on OpenSSL they would have realized that they needed to deploy staff into the community and maybe apply backup process to ensure the integrity of the software themselves.”Simon Phipps, founder of open source management consulting firm Meshed Insights
Companies are getting involved in open source, argues Phil Granof, chief marketing officer at Black Duck Software, which sells open-source management software and consulting services. The firm conducts a regular survey of open-source software users.
“Thirty percent of companies are making it easier for their employees to get involved in open source, and certainly, the percentage is higher if the products are relevant to the company”, he argues.
That didn’t stop Steve Marquess from complaining about the ones that didn’t, though. Marquess is the co-founder and president of the OpenSSL Software Foundation, the commercial entity
that supports the OpenSSL project with support contracts.
In a blog post called ‘Of Money, Responsibility, and Pride’, he called out Fortune 500 companies for not supporting open source more. “The ones who don’t have to fund an in-house team of programmers to wrangle crypto code, and who then nag us for free consulting services when you can’t figure out how to use it”, he wrote. “The ones who have never lifted a finger to contribute to the open source community that gave you this gift. You know who you are.”
It’s all very well to criticize large companies for not catching the bugs, but this belies the fact that there are simply too many projects, says Art Gilliland, senior VP and general manager for enterprise security products at HP.
“There are hundreds and thousands of different open-source projects and so it’s not realistic for any company to invest in any one of them”, says Gilliland, pointing out that HP invests hundreds of engineers’ time in protecting open source.
HP was one notable omission from the Core Infrastructure Initiative, a project organized by the Linux Foundation to support security efforts on large open-source software projects. The initiative includes Microsoft, despite the fact that the company’s recently retired CEO, Steve Ballmer, once called open source a “cancer”. How things have changed.
One of the questions in the initiative’s FAQ asks why they hadn’t done this before. “We’re doing what we can now”, begins the reply.
Better late than never, and never too soon, because this won’t be the last time. “It's the last vulnerability of its type. There will never be another vulnerability capable of affecting more than a single percent of the internet”, quips Gunter Ollman, CTO of security consulting firm IOActive. "Oh, and a unicorn gave birth to a flying pig yesterday.”
He anticipates “close facsimiles” of this bug in other software.
Preparing for the Unknown
Given the difficulties of spotting even show-stopping bugs like Heartbleed, it’s fair to say that there are still plenty of ‘known unknowns’ on the internet. We know that the vulnerabilities are out there, but we don’t know where. Companies have to prepare themselves against an unknown enemy that could render any system vulnerable in unexpected ways. So, how do they accomplish that?
Stare too hard at any particular bug and you’ll lose the bigger picture. HP’s Gilliland points to the broader attack cycle, and says that organizations need to understand that if they are to protect their systems.
He breaks that attack cycle down into five main areas. The attackers first research the target, and then infiltrate it. They map out its environment, and then they capture the data that they want. Finally, they exfiltrate the data. The dark market economy means that each of these activities gets a specialist, who is very good at it.
“So how does a company respond to the fact that they’re competing against the best in the world at those steps?”, he asks. “You don’t rely on any one of those controls to protect your infrastructure. You build a capability in every one of those steps.”
Gilliland is talking about defense in depth. The idea is that the next time there’s a ‘Heartbleed’, and a company hasn’t caught the flaw in the software it’s using, it’ll represent only one stage in an attack. The well-prepared company will have good protections built in to prevent the rest.
This is why in 2010, Debora Plunkett, then head of the NSA's Information Assurance Directorate, revealed the agency’s policy of already assuming that its networks have been compromised.
Government spooks have the right idea, says the director of security at one well-known IT company affected by Heartbleed, who asked not to be named.
“We design systems so that we assume that they will fail”, says the source, who confirms that he runs an entirely open-source stack. “We assume that bad guys will land in our environment. We are running file integrity monitoring, which is an integrated part of our Puppet process, so if a file is changed and it’s not through puppet and the SHAs don’t reconcile to the RPM database, then we have a problem.”
It isn’t about being rigid and ring-fencing your perimeter, says the source. It’s about accepting that there is another Heartbleed. It’s already out there, and waiting.
"We tell everyone, ‘you’re going to get hacked’,” he concluded. “It’s going to happen. Just assume that you have a sophisticated bad guy.”Director of security, large web service provider.