According to security trainer Tanya Janca, not all metrics actually matter for cybersecurity and there are some that can have significantly more impact than others.
Janca, the founder of training firm We Hack Purple, detailed her views on metrics during a session at the virtual SecTor security conference.
She began by stating that most people simply define metrics as a method of measuring something. The reality though is that there is more to metrics than just measurement. When done properly, metrics provide a way to spot patterns and trends that can help improve cybersecurity outcomes.
“We measure things and gather metrics specifically so that we can report and so that we can improve,” she said. “We report up to management and other teams on what we’re up to and then we use metrics so that we can improve ourselves.”
Why Reports Matter
As cybersecurity professionals, Janca said that generating reports for management is critical for a number of reasons. Reports are used to help get budget for tools and are typically also necessary for regulatory compliance. She added that reports also make management happy.
“If you don’t write reports, your boss doesn’t know what you’re doing,” Janca added. “You can’t have a security program that costs hundreds of thousands or millions of dollars and then not tell them [management] how you’re doing, that’s not going to go on for very long.”
However, while it’s important to keep management informed with reports, it’s equally important to have useful metrics that are tracked, Janca said. For example, some companies will count the number of vulnerabilities they have as a metric. She doesn’t see counting vulnerabilities as anything more than a “vanity metric” as it’s not particularly helpful. Having more software vulnerabilities could just mean that the organization has done a better job of testing and not that the organization is any more, or less, secure.
Metrics that Matter
Among the metrics that Janca does see as having meaning for cybersecurity professionals and the organizations that employ them is time to detection for a given security issue or vulnerability. Equally important is time to remediation of the issue as it’s critical to understand what the capabilities of the organizations are for fixing or patching a given issue.
Looking at vulnerabilities, understanding if the organization is detecting the same vulnerabilities time and again, or if it is finding different new vulnerabilities, is also important to measure. It’s also important to identify if there is a decline, or a rise, in a particular type of vulnerability. By identifying trends in vulnerabilities as opposed to just generically counting them, it’s possible to target categories of issues for training to help reduce them over time.
When looking at measuring the impact of an incident Janca said that it’s important to identify if established best practices were followed or not and if the various teams within the company worked together.
“If we aren’t measuring, we don’t know where to start,” she concluded.