Blog Post

Lessons for Security Teams from the Season’s Mega Breaches


Everyone is talking about how bad the situation is with Equifax, how its security was “allegedly abysmal” and “no surprise they got hacked,” etc. Unfortunately though, in our opinion, we are seeing a reality where organizations (who btw are spending significant amounts on security) are increasingly being compromised NOT by the most sophisticated 0 days, but instead by the missing patches. It is deja vu – no? When was the last time you heard of a company being in the news for being compromised by a web server vulnerability? Are we back to the early 2000s? Where are my Britney Spears CDs?

The reason that these non-sexy mechanisms are successful in breaching organizations is because we have trained a generation of security analysts that the threats that are most important are those originating from [pick your favorite foreign power here]. While that is quite likely true, those are not high probability threats. So what happens to the issues that are more mundane? Well, they get dropped to the floor – there just isn’t enough time to deal with those threats, given the average security team barely gets through five or so percent of the daily alerts. So hope is in fact a strategy – at least when it comes to cyber security.

The unfortunate reality is these attacks are not the typical smash and grab – attackers are often in the environment for weeks, if not months or years. They may not even be the most sophisticated attackers either, (all though these days just blaming a nation state seems like the easy way out) and don’t need to be! All that leads to media frenzy and commentators that will then talk about how the organization brought it on themselves. Questions will be asked about who in this day and age gets hacked due to missed patches or weak admin passwords? Well if recent events are anything to go by, seems like a lot of folks.

What’s a security team to do?

Well the obvious guidance is you have to push out your patches – but with vulnerabilities in foundational components like Apache Struts, that may be easier said than done – it can often require recompiling every web app that uses this component. And in an organization with hundreds of such apps, that can take months if not longer. So what can security teams do in the meantime to make sure you have eyes on the problem?

Threats like this don’t just happen overnight and even once the attackers are on the inside, they don’t typically steal hundreds of millions of records in a matter of hours or even days. So how do you find them quickly and mitigate the impact of the breach, perhaps even before data is stolen? It starts by understanding the attack cycle and looking for attackers early in that cycle.

Here are three critical junctures in the attack cycle where you attackers may be the most visible:

  • Reconnaissance: Unless this is a malicious insider already intimately familiar with the web application architecture, the threat actors will often spend time “casing the joint”. Even after they are inside, they need to do this to figure out where the crown jewels are and how to get to them. All of this generates activity on the network and while attackers can delete their activity from logs, it’s a lot harder for them to “delete” traffic from the network. Observing network connection patterns from the web server to the database that don’t fit the typical use or that occur during atypical hours is a technique to hunt for these kinds of reconnaissance. Similarly, are there attempts to connect to other servers in the environment or access other internal resources?
  • Command and Control: More often than not, once an attacker has a foothold in the environment, they will ensure they have relatively unfettered access to the compromised systems. This is often in the form of webshells, remote access tools, etc. Again, these tools can be stealthy to evade detection, but they need to communicate over the network to be effective. An analyst who has a platform that allows her to query network traffic and knows what to look for can find evidence of these even if she doesn’t know the specific remote access tool being used. For instance, one common technique experienced hunters use is to look for screen resolution values that show up in protocol metadata. Devices that are remotely controlled often have “oddball” resolutions that could never possibly be used by real world display devices. That in itself is not an indicator of compromise, especially in server environments which are often accessed remotely. But it gives the analyst a starting point rather than just a mountain of data.
  • Data Exfiltration: Even once the attacker has compromised the environment, they still need to get the data out of there. For instance, if your web server is intended to bring data into the enterprise network or data center, connections or data volumes that point in the reverse direction are something worth looking into. Perhaps this is a file sharing service and maybe nothing abnormal about these traffic patterns. But, on the other hand, this can help uncover odd and possibly malicious usage in a lot of environments. As mentioned earlier, this is especially important when the behavior stands out from other servers that are functionally similar or serve similar business functions. One thing worth mentioning is that even when connections are encrypted, an experienced analyst can look at attributes like payload lengths and make educated guesses on the type of transactions occurring i.e. is this someone or something transferring files out or just a streaming music service.

The other orthogonal consideration is threat prioritization. Most prioritization these days tends to occur based on the severity and impact of the threat itself. The threat alerting tools have no real context of the “thing being attacked,” i.e. human analysts may prioritize a “simple” exploit of a unpatched vulnerability on a server that houses your crown jewels, higher than a sophisticated 0-day (nation state, …) attack currently targeting the machine that shows the cafeteria menu.

Shameless Plug and Conclusion

So how does one do this? Clearly you cannot and do not have the time to look through logs for every web app and network data from every server. It is what takes a tremendous amount of time and expertise, neither of which is in abundant supply. That is where good security analytics come in. While humans are not great at processing large amounts of data, machines are. On the other hand, machines are not so great at drawing inferences on maliciousness by “eyeballing” the results of the analysis – luckily humans empowered with information can “sniff” things out.

For instance, if the technology highlighted that one of your servers has a bunch of persistent connections from external locations unlike any other web servers in your data center, an analyst can dive further into it and see if the behavior is justified. Or, if the analytics could tell you that a web server atypically has a number of outbound connections, an analyst can quickly make a judgement call on whether that is troubling or not.

Just as importantly, technology should be able to instantly tell you everything about the device and the users associated with the device so you are not having to consult other data sources. This would allow you to focus on the real risks to the business rather than just a theoretical list of priorities.

Awake’s Advanced Security Analytics identify the “uncommonly common” behaviors that are interesting to expert investigators. They automate the painstaking analysis these experts do today, instead delivering the information at the analysts fingertips. Analysts of all levels can therefore dive straight into the investigation or threat hunting, rather than trying to gather the information and context needed – something that unfortunately will most likely not happen since again where is the time for that?

As an industry, we need to focus on improving the productivity coming out of our security teams. Nation states hacking an organization are an obvious priority – but security teams also need to be able to get to and then investigate the threats that are not just slam dunk exploit kit infections. We need to arm the analysts with technology and workflows that allows them to quickly assess and determine if something is malicious and then give them all the information they need to investigate quickly and effectively. If we don’t bridge these gaps, enterprises will continue to be breached by things that look simple in hindsight.

Contact us if you would like to see how Awake can help with situations like this.


If you liked what you just read, subscribe to hear about our threat research and security analysis.

Rudolph Araujo
Rudolph Araujo

VP, Security Strategy, Business Development and Marketing