Most enterprises have come to terms with the fact that a security incident is not an “if,” but a “when.” Many are still struggling to translate this into a security architecture and mindset that focuses beyond prevention.
Teams need the right tools to detect and investigate critical security threats, tools for hunting, and performing diagnostics. Some organizations try to “capture everything” at enormous infrastructure and workforce cost, only to find they can’t analyze or operationalize it effectively come crunch time due to the same reasons.
Hunting for signs of an attacker on your network is like searching for needles in a haystack. Rich metadata is a part of the solution that allows you to decode the haystack and drive insights to find the attacker, halt their intrusion or exfiltration of data and stop the next attack.
Why metadata?
Rich metadata gathered from your network can capture more than 90% of the useful data that a full packet capture system would. More importantly, you can actually store and analyze it in real-time so you can actually impact attacks that you might never have been able to discover otherwise.
Incorporating metadata in this way can lower the overall cost of storing the actual packet captures (PCAP) while providing nearly the same level of visibility into the communication. Metadata can be stored as flat text which has the benefit of optimal compression rates for long term storage. Metadata has the ability to be stored in many common formats like JSON or XML rendering it searchable and reference-able by standard libraries.
Think about a phone conversation. If you had a recording of a conversation you could listen to every word that was said. However, that takes a lot of time. If you had an easily searchable description of everything that was said you could get almost the same value in a format much easier to consume. The richer the metadata you have, the richer the set of questions you can ask and answer quickly and without specific expertise.
For example, extracting the following attributes web application tracking details, protocol level and document metadata to the level of IP address and location, email communication metadata, internal filesharing, files accessed/transferred with document author, filename/file hash, header - footer information, and creation date, you can start to answer the following questions:
- Have we seen the document transmitted before?
- Who authored the document and when?
- Does the document have tags that describe sensitive data?
- Who else in the enterprise has a copy of the document?
- Was there any personally identifiable information (PII) or protected health information (PHI) in the document?
- Who was logged onto the machine that sent the document?
Extracting data like this from metadata in as close to near real-time as possible should be the new standard for cybersecurity within your enterprise team.
Secrets your metadata can tell you
Every incident responder or security analyst can tell you what happens when you get a “serious” alert. You swing into investigation mode – pulling logs, triaging endpoints, and piecing together disparate data. In many cases the data you want just isn’t available or can often take days to retrieve and digest, or even require expertise to understand.
By providing content-enriched metadata in near-real time, security teams can investigate suspected incidents in seconds and get answers to questions that were previously impossible to know.
For example, organizations can leverage metadata to routinely detect multi-vector attacks such as the Angler Exploit Kit by correlating related activity across multiple sessions. Instead of just alerting on a visit to the landing page or the delivery of the exploit, metadata can tie together each stage of the exploitation chain, so teams can understand how they were initially compromised, what malware was downloaded, which enables the team for rapid and comprehensive remediation.
The rich metadata captures every session that the network sensor can see in the network so that teams can investigate immediately. Placement of these sensors enables both server operations teams to hunt, and incident response (IR) teams to gather needed information for all packets that move across such a sensor.
Enriching this metadata is the next important step in gaining context on both tactics, techniques, and procedures (TTP) that adversary’s use along with their intent. Adversary intent is critical to adversary intelligence plans of battle which constitute a critical part in cyber battle planning by adversaries.
You can apply new threat intelligence and indicators of compromise (IOC) to all metadata from the network sensor that gathered the traffic. Storing this data over time facilitates retrospective analyses through data mining tools that can be tuned and instrumented around criteria and associated probability of ‘bad’. This simply means that you can look back in time and determine if you were affected by the threat.
Retrospective analyses also provides deeper understanding around adversary intent producing deeper details to support a comprehensive analyses of the adversary IPB. In this respect, leveraging well-understood military doctrine principles along with enriching captured metadata with new threat intelligence to historical data is an incredibly unique capability. Not only will it enable you to confidently answer the question: “are we safe?” but it will equip you to detect attacks, intuit adversary intent, and piece together their intent through their approach toward IPB.
Enterprises that store their historical metadata for several months can use quantitative techniques along with customized security analytics to support forensic analysis, including retrospective analysis of host activity to attempt to understand the initial event ‘blast radius.’
Without this ability, security teams often spend days or weeks piecing together information to determine exactly how their cybersecurity defence was penetrated, what the threat did and what needs to be done to prevent future breaches.
Rich metadata is crucial to find a needle in the haystack and drive insights to find the attacker, halt their intrusion or exfiltration of data and stop the next attack.