Network Threat Hunting to Defend the Broad Attack Surface
Network Threat Hunting to Defend the Broad Attack Surface
Now that we have discussed the basics of hunting for a threat that is already widely known, let us discuss some more ambitious techniques. As we mentioned in our prior posts the challenge with hunting for just known indicators is that they are very low in the Pyramid of Pain and the attacker is able to change these easily to evade detection. In part 2 therefore we will attempt to get more generic. The goal here is to identify the attack surface (including third party systems like SolarWinds). Often organizations will depend on tools such as Bro / Zeek to perform this function. The goal here is to understand what devices, applications etc. exist within the environment and then using threat intelligence and perhaps some basic unsupervised machine learning to spot the outliers.
This is an effective approach to find “badness”. But it will often overwhelm the security analyst with the volume of outliers, most of which are false positives. Awake has refined this approach by using an ensemble of machine learning approaches that are not just focused on identifying the “bad” but also on building an understanding of the known good e.g., how and where do the devices, applications and services on the network communicate, both internally and to the Internet. Our network threat hunting methodology uses this knowledge of “known good” to narrow down the list of outliers to a manageable scope for human triage and investigation.
The first step to defending the infrastructure is to understand what it is that is being protected. Interestingly, it is also where the adversary begins. As the MITRE ATT&CK framework points out threats often first manifest through Reconnaissance (pre-attack) and Discovery (post-attack) e.g. Gather Victim Host Information, Process Discovery and Software Discovery. In Awake’s experience, most often the organization’s configuration management database (CMDB) is outdated and not maintained. We have seen this be exacerbated with the COVID-19 pandemic and the changes that has forced on the digital infrastructure. Shadow IT, version changes, IP address changes, visibility gaps, DevOps and CloudOps processes, M&A, etc. all contribute to the challenges of maintaining an up to date CMDB.
Awake alleviates many of these challenges by automatically identifying and tagging devices in real time and labeling applications based on protocols, ports and machine learned intelligence. As a result, a threat hunter may start looking for third-party systems by using a simple passive DNS based model. Figure 2 shows how you can uncover all the SolarWinds domains accessed from the environment. With this list, the analyst can pivot to the domains that seem to be used for common services like the API, installer and / or updates.
Figure 1: Awake Model to Search for All SolarWinds Related Domains in an Environment
The astute reader is likely to point out at least two challenges with this approach. Firstly, there is no guarantee that this traffic originated from a SolarWinds application. It could have easily been a user browsing their website. Secondly, the domains in Figure 1 are not specific to the Orion platform, which is currently the only known compromised SolarWinds application. How can we narrow this list down further?
Awake incorporates detection for various MITRE ATT&CK techniques related to SSL/TLS traffic inspection without relying on decryption. Customers can decrypt data if they are inclined to and feed it into the Awake Security Platform. On the other hand, if they are looking to avoid the legal, compliance and privacy challenges involved with decryption, they can still analyze SSL/TLS traffic using encrypted traffic analysis and avoid going entirely blind to encrypted threats. The platform instead relies on encrypted traffic analysis. In its most basic form, Awake allows hunters to query across all TLS certificates seen in the environment (over a 6+ month period if needed). This information can be used to filter the list of domains further, based on information within the certificate or even the JA3 hash, which the Awake Security Platform automatically computes for every TLS handshake in the environment. For instance, our own threat researchers and managed NDR analysts will often use the hash to eliminate known browsers and zero in on activity that may be originating from applications such as PowerShell, a tool commonly used to download additional malware as a follow-on to phishing. Figure 2 illustrates how an analyst can build a model to look for a specific subject for the TLS certificate. Within the matching traffic, they can then observe details about the device, the certificate and the JA3 hash. All of these can be used for additional investigative pivots.
Figure 2: Identifying the TLS Certificate for SolarWinds Orion.
As the Awake team first began investigating the Sunburst threat, we discovered some environments where the SolarWinds devices were air gapped. These therefore did not generate any external traffic and therefore hunting for domain names or TLS fingerprints would not be effective. In this case, we stole a page from the attacker playbook. Just like MITRE ATT&CK mentions adversaries using ICMP and other Non-Application layer protocols for communication, we were able to use these protocols for discovering applications or devices with limited TCP/UDP communication. As MITRE mentions, “It is not as commonly monitored as other Internet Protocols such as TCP or UDP and may be used by adversaries to hide communications.” The Awake platform provides unfiltered access to over 3000 protocols, so threat hunters have broad datasets to analyze.
For this specific threat, our analysis (Figure 3) helped identify the payload hash of ICMP pings generated by SolarWinds appliances. We discovered these to be unique and therefore a good way to surface systems that may be on an isolated management or segmented networks.
Figure 3: Detection of Air Gapped SolarWinds Systems Based on ICMP Ping Payload Hash.
After discovering the third-party systems in an environment, a hunter would move onto the next step of identifying and comparing against the expected pattern of communication for these systems.
The Awake platform allows hunters to easily create a list of devices or IP addresses they are interested in (e.g., critical infrastructure devices such as Active Directory, database servers, industrial control systems, executive workstations etc.) and look for all external traffic over a given time window. This helps in identifying any IP or Domains that were used in the Command-and-Control or Exfiltration stages of the MITRE ATT&CK Framework. The statistical analysis presented will also indicate if any domain or IP address stands out as anomalous. Figure 4 illustrates how we do this for the SolarWinds systems on a network.
Figure 4: Map out External Communication Patterns for SolarWinds Devices
Often multiple services run on the same device and that may result in a large pool of external communications. Awake therefore lets you intelligently filter and triage the list. As discussed earlier, the list of communications can be filtered to only include specific traffic e.g., only activity not originating from a browser. Furthermore, Awake allows hunters to reduce this dataset by limiting results to domain-related information such as: first-seen, registered-date, updated-date, WHOIS data and other information. For instance, Figure 5 illustrates how we can hunt for any external traffic from a SolarWinds device but limit the results only to domains first seen in the last 90 days.
Figure 5: Identify Communications from SolarWinds Devices to Domains First Seen in the Last 90 Days
Awake also tracks ASNs for external sources / destinations. Threat hunters can use this information as well as data from the vendor to label ASNs as trusted by the organization or “known good”. Consequently, this allows the platform to easily identify traffic from management devices (such as SolarWinds) going to untrusted ASNs (Figure 6). Similarly, using machine learning models, Awake is also able to detect “impossible rate of travel” type situations. This was one of the Sunburst campaign TTPs documented by FireEye, where a compromised account is used by a legitimate user and then within minutes is used by the attackers from a location across the globe.
Figure 6: Identify Communications from SolarWinds Devices to Untrusted / Suspect ASNs
Once a hunter detects unusual activity from an ASN or CIDR block, Awake allows them to look at all malicious activity from those ranges. For example, it is possible to detect when any type of authentication originates from an external location, specific to geolocation or ASN. Similarly, the Awake Security Platform learns / can be configured to recognize legitimate external remote access and then detect any traffic that is outside the norm e.g., untrusted IP ranges (Figure 7), geo-locations etc. It is important to point out that geolocation is not a reliable attribute since many sophisticated attackers will use a device in the same location / time zone / country as the victim. In other words, while the absence of geo-infeasible logins is not an indicator of no-compromise, the presence of these logins is certainly a strong indicator of a potential breach.
Figure 7: Identify Authentication from Untrusted / Suspect IP Ranges
Awake customers use this technique to gain an understanding of authentication requests coming directly from the Internet and potentially bypassing corporate VPN or zero trust network access (ZTNA) solutions.
As a bit of a sidebar, it is worth spending a couple of minutes thinking about the activities that are most interesting to an experienced threat hunter. It is not hard to find network anomalies. Many systems based on Bro / Zeek and other such network analysis solutions can do this. The goal however is to efficiently identify only those anomalous behaviors that are actually interesting.
Let us focus on one use case–a device communicating out to the Internet. Many devices communicate with external sites all day long e.g., checking email, syncing calendars etc. There are also fewer common activities that might be more specific to a department or workgroup (e.g., accessing a source code control system within the developer team or a customer database within sales and marketing). Typically, these behaviors are not as interesting from a threat hunting perspective. It is the relatively uncommon activities performed at high frequency from a small subset of systems that are most interesting. Figure 8 illustrates the conceptual model.
Figure 8: The Uncommonly Common Activities Where Malicious Intent Often Sits
The Awake Security Platform automatically surfaces these uncommonly common characteristic artifacts (protocols, commands, destinations, TLS negotiations, etc.) and allows the analyst to pivot based on these and track them over time. More importantly, analysts can look for cases where multiple artifacts from the same underlying activities are flagged as interesting. This is a strong indicator that the underlying behavior is worthy of deeper attention by the threat hunter. For instance, Figure 9 illustrates an example where we hunt for malicious behavior that is previously unknown. In this case we look for workstations and servers communicating externally where the destination ASN and one or more characteristics of web traffic (whether TLS or HTTP) are unique to this device but recurring over time. As the results in Figure 9 show, the device in question is connecting to an ASN which is rare to this environment and to a domain which is rare as well. This combination makes the device an interesting target for deeper investigations.
Figure 9: Uncovering Devices and Behaviors with Unique Communication Patterns
Digging deeper into the device shown in Figure 9 and its activities, helps us identify a user / device tunneling communications out to a TOR server (Figure 10).
Figure 10: Zeroing-in on Potentially Malicious Behavior
While these capabilities are heavily used within the platform to automatically detect behaviors like that illustrated in Figure 10, they can also be used as part of a threat hunt using the building blocks called recipes. Furthermore, as we will discuss in part 3 of this series, Awake’s virtual security analyst Ava also automates further OSINT / threat intelligence-based analysis of these uncommonly common behaviors. It is this mechanism that has enabled Awake to uncover malicious activity during the initial stages of the attack as opposed to waiting for the more easily detectable command and control (C2) or data exfiltration. While detecting a threat at the C2 stage is better than not detecting it at all, C2 is a post-compromise stage. This means the organization will need incident response, remediation plans and potentially public relations and reputation management. Clearly, detecting the threat earlier saves time, expense and resources.
VP, Threat Research
Dig Deeper with These Resources
Awake Security 2 Minute Explainer Video
What if security could think? What if it could sense danger, calculate risk, and react quickly based…
The Internet’s New Arms Dealers: Malicious Domain Registrars
This report dives into the results of a multi-month investigation that uncovered a massive global surveillance campaign…