Threat hunting is the process of an experienced cybersecurity analyst proactively using manual or machine-based techniques to identify security incidents or threats that currently deployed automated detection methods didn’t catch. To be successful with threat hunting, analysts need to know how to coax their toolsets into finding the most dangerous threats. They also require ample knowledge of different types of malware, exploits and network protocols to navigate the large volume of data consisting of logs, metadata and packet capture (PCAP) data.
Deciphering an Open-Ended Term
Threat hunting can mean slightly different things to different organizations and analysts. For example, some believe threat hunting is based entirely on difficulty. If the activity is simple, such as querying for known indicators of compromise (IOCs) or searching for POSTs to IP hosts without referrers, it may not be considered threat hunting. On the other hand, searching for things that could be indicative of malicious activity and require analysts to sift through benign traffic may be viewed as threat hunting.
No matter the interpretation, it’s important to note that threat hunting requires a significant time investment, as successfully identifying items of interest is far more difficult when there aren’t signatures available. Furthermore, what matters most is not the semantics of the term, but that organizations and their analysts continually conduct threat hunting by ensuring they have the capabilities for discovering and remediating any cyber risks.
Key Threat Hunting Characteristics
Threat hunting isn’t reserved only for large enterprises with extensive resources. Rather, any organization can employ the best practice by prioritizing the following key characteristics:
- Being Proactive: Rather than waiting for an alert from an existing security tool, threat hunting requires proactively sniffing out potential intruders before any alerts are generated.
- Trusting Gut Feelings: The best threat hunters avoid relying too heavily on conclusive alerts from tools and rule-based detections. Instead, they look for clues and listen to their gut, and eventually apply those findings to create automated threat detection rules.
- Following Traces: The concept of threat hunting assumes there was a compromise and that attackers have left traces in an organization’s environment. Following all traces and leads fully is therefore crucial, no matter how meandering or lengthy the hunt.
- Embracing Creativity: Threat hunting isn’t about following established rules. To stay ahead of the most skilled and inventive attackers, threat hunting requires embracing creativity and any relevant methodologies (established or not).
However, it is also clear based on these characteristics that many organizations can struggle with establishing a threat hunting regimen. Instead, it becomes a work of art that only one or two individuals are capable of and even for those requires tremendous investment of time. This lack of repeatability stems from a lack of support for this process within most existing security tools and even the most proficient threat hunters struggle to consistently producing valuable results.
Common Threat Hunting Techniques
There are four common threat hunting techniques used to pinpoint threats in an organization’s environment, including:
- Searching: This involves querying evidential data, such as full packet data, flow records, logs, alerts, system events, digital images and memory dumps, for specific artifacts using clearly defined search criteria. Since it’s rare to know exactly what to look for when starting to search for threats, it’s important to find a balance between not making search criteria too broad (i.e. becoming overwhelmed by receiving too many results) and not making the criteria too narrow (i.e. missing out on threats by receiving too few results).
- Clustering: Using machine learning and AI technology, clustering involves separating clusters of similar data points based on particular characteristics from a larger data set. The practice allows analysts to gain a wider view of data that’s of the most interest, find similarities and/or unrelated correlations, and weave those insights together to get a clearer picture of what’s happening within their organization’s network and determine how to proceed next.
- Grouping: This technique involves taking multiple unique artifacts and identifying when multiples of them appear together based on the predetermined search criteria. While similar to the Clustering step, Grouping only includes searching an explicit set of items that have already been established as suspicious (whereas Clustering includes searching large volumes of data to identify data sets that need to be investigated further).
- Stack Counting: Often referred to as Stacking, this practice involves counting the number of occurrences for values of a particular type of data and analyzing the outliers of those results. Stacking is most effective with data sets that produce a finite number of results and when inputs are carefully designed. Being able to organize, filter and manipulate the data in question is key to finding any anomalies in large data sets, so leveraging technology — even something as basic as Excel — is critical when Stacking.
Democratizing Threat Hunting
Organizations of all sizes and industries want to try to find every possible threat as soon as it manifests itself. That’s why spending on automated cybersecurity solutions continues to rise so rapidly. However automated tools can only do so much, especially since new attacks may not have signatures for what’s most important and the fact that not all threats can be found using traditional detection methods. In fact, research shows that 44 percent of all threats go undetected by automated security tools.
To keep up with ever-resourceful and persistent attackers, organizations must prioritize threat hunting and view it as a continuous improvement process. These teams would also be well served by investing in technologies that enable hunting and follow-on workflows. For example, if threat hunting methods are discovered that produce results, make them repeatable and incorporate them into existing, automated detection methods. If the same threat hunting workflow keeps getting repeated and produces results without a lot of false positives, try automating those workflows.
The effectiveness of threat hunting greatly depends on an organization’s level of analyst expertise as well as the breadth and quality of tools available. An organization’s acceptable risk level, IT staff makeup and security stack can also impact the type of threat hunting that’s feasible, so it behooves organizations to leverage technology such as the Awake Security Platform to mitigate the complexity and tribal knowledge required for threat hunting. In doing so, organizations can ensure all analysts are able to hunt and better protect critical business assets, regardless of their skill level.