Detecting Supply Chain Threats like SolarWinds / Sunburst
Untrusting the Trust
By now if you work in security (and frankly even if you do not), you surely have heard of the recent compromises of FireEye and then this week, SolarWinds. As it turns out the two were related—FireEye discovered they were breached and on investigating the attack zeroed in on the SolarWinds Orion Platform they had deployed internally. The same SolarWinds (also being referred to as Sunburst) threat affected multiple other high-profile organizations including parts of the US government. As many have pointed out this is what is described as a “supply chain” attack. Given the gravity of the threat, much has and will be written about the SolarWinds hack, how to detect the threat, what the indicators are etc. In this blog post we wanted to focus more broadly on how security teams should look at the general threat model around supply chain threats and how they can detect them before they become headline news.
Why the Supply Chain Threat?
Frankly, attackers have had to up their game as they realize that security teams have become more effective at detecting “conventional” malware-based threats. Put differently, if you are looking to compromise an organization such as FireEye, the age-old techniques such as phishing and exploits are perhaps less likely to be effective. Instead, if you can make your way into the network by compromising a trusted software application they use every day, that is far more likely to fly under the radar. That is essentially what happened here, the build infrastructure for the SolarWinds Orion platform was compromised, which allowed the attacker to then inject malicious code into a DLL file that is part of the platform. The build went through SolarWinds’ code-signing process with no one realizing anything was amiss and next thing you know, customers who downloaded and updated their instance of Orion, just brought the proverbial trojan horse in!
It is important to point out that such threats are not new. Just over two years ago, news broke about a semiconductor provider that was breached as a conduit into several large technology companies. While the details have been disputed, it is an example of how sophisticated attackers are approaching their targets. In fact, more than 10 years ago, some of the Awake team responded to a breach at a think tank where the command and control / exfiltration channel was established through a malicious print driver that once installed allowed the nation-state adversary to monitor any document that was printed.
Can We Really Detect These Threats Before the News Breaks?
Over the last couple of days as we have discussed this issue with our customers, the concern has been could this type of threat be the new norm? And if so, how should a security team factor this into their threat model and their defensive strategies. While we do not know exactly how FireEye first discovered this threat in their environment, our educated guess would suggest this was likely uncovered through manual threat hunting: looking through historical data to spot communications that “just don’t make sense”. While that might sound simple enough, the reality is most organizations do not have the access to skills and threat intelligence FireEye has, given the nature of their business. There is also the matter of the time and effort it would take to do this at scale across all the supply chain components in the enterprise.
A Generic Supply Chain Threat Protection Methodology
Using the example of SolarWinds / Sunburst attack campaign, we lay out a methodology security teams can use to uncover supply chain threats in general. We propose a five-step process (Figure 1) to address the needs of a customer who does not have the resources FireEye has its disposal.
Figure 1: Supply Chain Threat Protection Methodology
- Discover: You first need to know what it is that you are protecting. In this era of shadow IT and DevOps, unfortunately the supply chain is not just what your procurement team or your CMDB tells you. We often find those sources of information typically cover only between 40% and 50% of the systems and applications on your network. You therefore must be in a continuous vigilance mode to discover new additions to the enterprise. Awake’s Network Detection and Response (NDR) platform and our EntityIQ capability tracks and monitors devices, users and applications. Figure 2 illustrates one of our techniques: using TLS certificate attributes to track the systems communicating with the SolarWinds web infrastructure. This is an effective way to zero in on the actual Orion systems on your network.
Figure 2: Discovering SolarWinds Orion in the Environment
Note: For this specific SolarWinds / Sunburst threat, there will be many more details to unfold over the days and weeks ahead. But, at this point, it would be prudent to assume that potentially every SolarWinds product could be compromised. After all, if the attacker had privileged access to the build server for the Orion platform, who is to say what else they had access to. So, while you look for the Orion platform on your network, make sure you are also looking for everything else that is SolarWinds related (Figure 3) and put defensive controls and monitoring around these systems right away.
Figure 3: Discovering all SolarWinds Services in the Environment
- Understand: Of course, seeing more is just the beginning, the next step is actually knowing more. What are these systems doing? What is their typical network footprint? Who needs to communicate to these systems internally? Who do they communicate with externally? Do they typically upload or download data etc? It is a good practice to ask your supply chain partners these questions and then behaviorally profiling these systems can be used to enforce a zero-trust / default-deny policy for all but the expected behaviors. For instance, if the Orion system is not supposed to be connecting laterally to multiple devices on your network with a slew of different credentials, then do not allow it to. Figure 4 illustrates one of the techniques to establish an understanding of the external footprint for the SolarWinds deployment and can compare it to information we have from the vendor. Are all those destinations needed? Are all those expected?
Figure 4: Understand the External Destination Footprint for SolarWinds Orion
- Monitor: As mentioned above, with the visibility and understanding of the behaviors, threat hunting can be automated using data science techniques to spot outliers and deviations from expected behavior. If the same Kerberos credentials are being used within a small period of time from widely different geographical locations, is that even legitimate? Awake’s AI-driven approach automatically hunts for these threat behaviors on your behalf, allowing you to scale the monitoring without relying on a battalion of threat hunting experts. For example, the Awake Security Platform automatically monitors for new / unexpected connections from the systems in question. Figure 5 shows how the Awake Security Platform identifies connections from the SolarWinds devices to new destinations first seen within the organization recently (within the last 90 days in our example).
Figure 5: Identifying SolarWinds Devices Communicating with Unexpected / New Destinations
- Investigate: Of course, not every outlier or anomaly is going to turn out to be a supply chain threat. You therefore need a triage and investigative process that examines the forensic data to determine if this is simply a software defect, a new behavior or indeed a threat. In fact, much like an experienced threat hunter, Awake automatically accounts for additional context about the observed communication to help with this decision making. For instance, our NDR platform uses TLS fingerprinting to identify and prioritize non-browser connections to unusual external destinations, since experience shows these are far more likely to be malicious. Figure 6 shows a connection from one such non-browser source to one of the domains identified as an indicator of compromise in the Solar Winds / Sunburst campaign.
Figure 6: Non-browser Connection to a Domain Associated with the Sunburst Campaign
- Respond: Once you have detected the threat, the goal is to shut it down as soon as possible. Awake customers rely on our integrations with endpoint, network and infrastructure such as Active Directory to contain suspect devices, block command and control destinations and prevent lateral movement. Using known IOCs is certainly one way to do this, and you should. But, also consider the dynamic nature of the threat: changing command and control domains etc. Therefore, the process we lay out here is a continuous loop.
Perhaps the sad reality is that we should just assume we exist in a compromised environment and then focus our efforts on protecting the crown jewels. If you live in this zero-trust world it is critical to understand what exists in your environment and only explicitly allow the behaviors / communications that are expected. This is also a good time to change your privileged account passwords and review your incident response processes to ensure you are set up to defend and respond to threats like this. Talk to us about how Awake’s AI-driven network detection and response platform and our Awake Labs experts can help deliver 360o visibility and continuous diagnostics and monitoring of the environment as well as compromise assessment, incident response and forensic services.
If you liked what you just read, subscribe to hear about our threat research and security analysis.
VP, Security Strategy, Business Development and Marketing
Dig Deeper with These Resources
Awake Security 2 Minute Explainer Video
What if security could think? What if it could sense danger, calculate risk, and react quickly based…
The Internet’s New Arms Dealers: Malicious Domain Registrars
This report dives into the results of a multi-month investigation that uncovered a massive global surveillance campaign…