In an interactive panel titled ‘Cyber Risk: This is not your father’s playbook’ at EWF USA 2014, Leigh Honeywell, security engineer at Heroku, and Carolyn Munoz, senior business security officer at ADP, took to the stage to discuss best practice for incident response, from both a technical and business perspective.
Honeywell, formerly Microsoft, used the example of Heartbleed to demonstrate how Heroku coped in an incident response scenario. “The embargo was lifted on the bug a day early, and we were completely vulnerable to this bug and needed to do something,” began Honeywell, who admitted it was “tricky to know how much to tell customers. It was hard to know what to tell the users that wouldn’t also have helped the hackers.”
Heartbleed, she quoted researcher Matt Blaze, was a rare incident where the bug was worse than not having any cryptography at all. “The bug did not just disclose the encrypted data that was being transmitted, but also potentially other data in the server's memory.”
During the incident response process – which Honeywell confirmed took a total of 32 hours from the time the bug went public to the time of patch – staff were working around the clock. “That’s the thing we failed on – everyone tried to be a hero, to keep working through, but you need to be disciplined with shifts and breaks to be efficient,” she admitted.
“If you want to have a long career in incident response and not burn out, you need to figure out ways of making it sustainable. From long-term issues like making sure your IR team isn't short-staffed to in-the-moment health issues like staying hydrated and getting enough sleep, watch for the things you miss when you're ‘in the zone’ focused on a critical issue,” she advised.
An incident response team, she continued, should always be over-staffed. “You need that slack on-call for when things do fall apart. My boss says incident response isn’t efficient if no-one’s watching video games at some point during on-call hours.”
If you want to have a long career in incident response and not burn out, you need to figure out ways of making it sustainableLeigh Honeywell, Heroku
Honeywell shared a series of incident response learnings which she attributed to her experience in her role at Heroku, and formerly at Microsoft. “Centralise authority,” she began. “Have an incident commander that you can override if necessary. And limit the spread of mis-information. Set up a private space – [whether a secure phone bridge, virtual or physical space] – where those managing the incident can communicate.”
Remaining calm throughout was another piece of advice shared by Honeywell. “It’s stressful, emotional, and there can be tempers. But keeping calm is crucial.”
“Follow your playbook when it makes sense, and throw it out when it doesn’t,” continued Honeywell. “One important insight on checklists (and playbooks) that I got from Atul Gawande's book The Checklist Manifesto is that checklists aren't really training materials and they aren't instructions - they are guides for practitioners who already know what they are doing, so that they can do it right every time.”
ADP’s Munoz emphasised the importance of a playbook for incident response. “You need a go-to guide when these things happen. Gather your facts, know who your first-level and second-level stakeholders are. People move, people change, stakeholders change,” she said. Knowing who they are by name, and understanding who actually needs to know what, is integral to successful incident response.
The next essential, Munoz clarified, is a checklist. “You’ll need to know the basics: what happened, how, who is impacted, what data is impacted, how many people have been affected and in what jurisdictions, and how long has it been going on. That’s what should be on your checklist, and what you need to know.
In addition to a checklist and fact sheets, a playbook should contain information on stakeholders, documentation, instructions, talking points, FAQs, client letters and a press statement, advised Munoz. “During Black Hat, the Russian hacker password story broke, and we were ready, We put a security alert on our website with a statement. We put it right out there,” she recalled.
Finally, Munoz added, a playbook should be ready to hand on to your predecessor to continue the best practice for incident response.