We’re all familiar with the story – sophisticated cyber-attacks are increasing in number and complexity, adding to the workload of already beleaguered security professionals, and helping fuel a surge in cybersecurity job postings. But these positions are incredibly tough to fill because of the insufficient supply of experienced professionals and lack of new talent. In fact, a Stanford University study shows the cybersecurity skilled personnel gap stands at 200,000 unfilled jobs.
Between the challenge of choosing the right tool amongst a myriad and the challenge of finding skilled security personnel, you have to wonder: how can companies defend themselves in this increasingly complex threat landscape?
Machine learning is an excellent place from which to start. I’m not suggesting that security analysts should be replaced by automated systems, rather, they should complement one another. For example, machine learning-based analytics tools can provide less senior security personnel with comprehensive attack information, enabling them to go further than they otherwise would be able to on their own, significantly improving the quality of the incidents that they pass upward to more experienced analysts.
This in turn frees up experienced analysts to focus on advanced activities, using previously unavailable insights that have been unearthed by machine learning to test complex hypotheses when hunting for threats.
Machine learning is well suited to take advantage of the large volumes of data produced by cybersecurity systems, and is especially useful considering that cybercriminals are constantly changing their tactics to stay undetected. With machine learning, security analysts have the ability to:
- Accurately detect attacks despite weak signals – Machine learning can analyze data from multiple perspectives and string together seemingly unrelated actions to accurately surface an attack. It’s done without contributing to the “alert white noise problem,” an issue for traditional detection technologies that have a reputation for generating alerts for anything suspicious.
- Shorten the time to detect attacks – On average, it takes over six months to identify attacks inside the corporate network. Many factors contribute to this, a key one being that companies aren’t always looking at the right data. Machine learning can continuously analyze an almost endless variety of data to quickly pinpoint an attack.
- Identify attackers still lurking in the network – Machine learning can continually sift through the vast amounts of data within an organization, annotating and enriching it, even when suspicious activity is not flagged as an alert. This enriched data effectively provides a rich taxonomy for hypothesis testing that helps lead to the discovery of hidden attacks.
So how can organizations effectively introduce machine learning into their cybersecurity initiatives? I would recommend the following steps:
- First, identify the problem. Is it to detect compromised users and malicious insiders? Do incident investigations need to be more efficient? Do analysts need to hunt for threats?
- Then determine what data is required to solve the identified problem. To support the comprehensive visibility needed to identify advanced attacks, use relevant security information from the security and network infrastructure. Different data sources enable specific and unique views into a problem, combining to create clarity where there was none. Thus, access to the “right” data sources (e.g., packets, network flows, logs, alerts, threat feeds, endpoint, etc.), rather than simply “more” data sources, is crucial.
- Next, ensure that the machine learning analytics are multi-dimensional in order to realize the promise of comprehensive visibility. Spotting abnormalities in authentication, privilege escalation, lateral movement, high-value resource access and exfiltration activities – these are just some examples of what’s needed to build up 360° views of users and hosts to catch attacks early.
- Make sure that a variety of techniques are being used in the machine-learning models. A combination of supervised, unsupervised and semi-supervised techniques allows machine learning to deliver immediate value by automatically detecting attacks without upfront configuration or rules, while also adapting to the uniqueness of your company’s environment.
- Finally, ensure that supporting evidence is easily accessible. Despite automation, analysts will need to validate the accuracy of identified anomalies, determine whether the abnormality is in fact harmful and unearth needed contextual evidence to paint a complete picture of the incident. This evidence not only helps explain why something was flagged, but is a key component in determining necessary remediation activities, and is vital to support threat hunting efforts, providing the proof needed to support hypothesis testing.
Attacks are constantly evolving and true artificial intelligence is a long way off – primarily because there is no substitute for human knowledge, skill and intuition. However, machine learning is not a cybersecurity pipe dream. When effectively leveraged, it can provide deep insights to support human-driven workflows to quickly analyze and respond to attacks – conducting investigations that would’ve previously taken weeks if not months.
Given the cybersecurity skill shortage that continues to be an issue, and with the average cost of a data breach rising to $4 million (up from $3.8 million), machine learning will rise as the solution of choice to help support security teams significantly improve their organizations’ security postures.