When the Heartbleed Bug compromised OpenSSL in April 2014, it not only triggered a “rekey-the-internet event,” but it also transformed forever how OpenSSL would be developed, patched and tested.
At the 2016 RSA conference in San Francisco, OpenSSL development team member Rich Salz, and security architecture expert Tim Hudson, described how much has changed – and how much needed to change – in the post-Heartbleed era.
Despite being the world’s most widely deployed TLS library, the open-source encryption software was largely overseen by two semi-volunteers working with almost no money, resources or infrastructure. Salz and Hudson acknowledge that the disparity between the large number of companies that used the product, and tiny the number that supported its development, was crazy.
“We should have put a lot of WTFs and dingbats into this presentation,” Salz said. “People still ask how we could let this happen. Why didn’t all those companies making money off OpenSSL contribute?”
In reality, OpenSSL had little organizational structure that would have allowed companies to offer financial support. Also, Salz said, because there hadn’t been a major disaster, few people considered that OpenSSL posed any danger.
Since the Heartbleed incident, not only has the security community woken up to the fact that OpenSSL needs support, but the organization itself has been professionalized. The team has grown to about fifteen people – a combination of volunteers and paid staff. Even more important, the team now have processes in place to ensure that updates happen in a timely fashion, that downstream users are part of the process, that new code has multiple eyes on it before it is released, and that old code that is no longer helpful is removed.
“We have a formal decision-making process,” said Hudson. “You have to have a process that gets you results in a finite period of time. Any time we get to a point where there are multiple ways to approach a project, we get a decision quickly so we can move forward.”
In addition, the OpenSSL team held their first ever in-person meeting – people who had known each other exclusively online for years met face-to-face for the first time. It was both a bonding experience, and a productive one.
“We got consensus on a coding style – not easy when you have ten C programmers in the room” Salz said. “We developed a release strategy, posted on the internet, and we’re keeping to it. There’s nothing like working in a room together to make things smoother.”
Documentation and feedback mechanisms have been systematized, and the team members now solicit donations more actively, to ensure the funding is in place to maintain these higher standards.
“We’ve learned that you can’t depend on any individual or two individuals to do superhuman feats,” Salz said. “You need funding, support, project management, email, websites, and you need to build a community. It’s not what programmers are necessarily best at, but it needs to be done.”
Picture credit - @GluuFederation