Blog Post

Lessons New SOC Analysts Need to Learn … Quickly

The Sky is Not Falling: Not This Time Anyways

the sky is not falling for SOC Analysts

Starting any new job may feel like a daunting task, but this is especially true for analysts starting out in the security field, whose job depends on them protecting the organization. In my experience, both personally and by watching junior coworkers, your first SOC Analyst role is very much a trial by fire experience. Training typically consists of reading some outdated documents and shadowing someone senior until you feel comfortable going at it alone.

Maybe that’s not too different from other occupations, but what makes matters worse in infosec is there’s a good chance you’ve never seen a majority of what you’re about to encounter, ever. And being a successful analyst comes from a combination of habits and knowledge that you probably didn’t pick up from a traditional 4-year degree. Furthermore, security is constantly evolving. Even the most experienced analysts encounter new things on a regular basis. So, you learn quickly, among other things that:

  • An alert does not mean that something bad is actually occurring, in fact, it’s most commonly a false positive.
  • Everyone is scanning everyone, all of the time.
  • Just because a packet was sent from one IP to another, does not mean that its cause for alarm or even abnormal (Even if it’s a Chinese IP).
  • Sometimes the tools you use and think are telling the truth are not.
  • Even in a junior role, you typically make decisions that have a substantial business impact, frequently with little ability to “fact-check”
  • Content delivery networks are like a bunch of people wearing ski masks, spewing information at you.

There are plenty of others I’m sure—dead ends that direct time and attention away from more productive investigations. This is true for senior analysts as well, but more junior analysts are often the ones left spinning wheels and wasting cycles over what turns out to be nothing.

Digging Into Scanning Activity

Let’s explore and demystify one of these today—“everyone is scanning everyone, all the time”.. (And if you want the backstory as to why I chose to dive into this, it’s worth checking out my recent Help Net Security article, but for those that can’t be bothered to click through, it all begins with a late night phone call when someone thought they were being hacked!

What better way to help demonstrate the ubiquity of various types of scanning than spinning up a honeypot. This isn’t supposed to be an exact representation of Internet traffic as a whole, but more so a snapshot of just how much scanning there is. I can almost guarantee that if you set up your own honeypot (which I completely encourage), no matter where it is hosted you’ll see results. In my experience, what you see is going to differ based on whether you’re hosting on a VPS, corporate network, or residential IP.

On my honeypot I listened for / responded to traffic on the following ports:

7 (Echo)7 (Echo)
8 (MOTD)22 (SSH)
53 (DNS)23 (UNIX Telnet)
69 (TFTP)24 (Windows Telnet)
123 (NTP)25 (SMTP)
5060 (SIP)80 (HTTP)
2525 (SMTP)
9200 (Elastic Search)

HoneyPy emulates most of the above protocols, however, since there wasn’t a plugin written for SSH, I just had it respond with random bytes so I could see if anyone was looking for anything on TCP 22. Just to demonstrate what kind of emulation we’re talking about here (remember that HoneyPy is low interaction), you can direct your attention to the clip below (me interacting with the honeypot over port 23 with telnet).

So, what did I see after it was setup? The first thing worth noting is how quickly I saw activity. After I started the honeypot, I left the logs running (tail –f) on a secondary screen so I would notice if there was any activity… also because anyone who looked over my shoulder would think that I’m a super cool hacker ?

As though this little experiment was trying to prove my point, within 13 seconds of starting the honeypot, I had a bite.

log file entries honeypy

The first entry in the log snippet above is the last bit of startup (indicating that HoneyPy has just started), and the very next log 13 seconds later is someone trying to connect over port 22. It appears once they figured out that the connection wasn’t a real SSH connection, they stopped trying to connect.

While most of the attempted connections were over SSH for the day and a half I left the honeypot running, I also saw HTTP, NTP, SIP, and Telnet. There were 771 connection attempts over that period targeting the ports I was listening on.

honepy scanning protocol mix

The telnet traffic consisted entirely of traffic like that below:

honeypy honeypot telnet scans

The traffic in this case was likely coming from a IoT botnet; the password-guessing and commands look similar to Mirai. The SIP traffic consisted of the not-so-friendly “friendly-scanner” activity. The NTP activity seemed to just be scanning to see if the port was open and the service was running. Finally, the tiny amount of HTTP traffic I saw was trying to fingerprint the webserver; looking for phpMyAdmin and Apache related URLs. There was also some weird traffic that was likely misconfigured (at least I’m not aware of why you would scan like this intentionally).

honeypy honeypot IoT botnet mirai scans

It looks like their script was mixing up pieces of the user agent with the URIs.

So What Does All of this Mean?

First, I want to point out that this was not some elaborate, targeted attack on me. Instead, on the interwebs, everything is being constantly scanned. All of that scanning above would be considered noise in any SOC I’ve worked in. The point is that I spun up a honeypot on a VPS and gave no one any reason to scan me and within seconds, my server was already being scanned. 771 connection attempts are actually lower than I thought it would be for a day and a half (perhaps I should have listened on SMB and RDP? ? ). However, it’s still a surprising amount I’m sure, to those that are not familiar with Internet noise. If you give someone a reason to scan you by being a well-known corporate network or an established website, then your level of noise is going to far exceed that of a random VPS in the cloud.

In my experience monitoring multiple networks belonging to different companies located in different parts of the world, you will see the same IPs scanning for the same things across many of those networks. The scanners don’t care if they didn’t get any positive results yesterday, they’re coming back today. They don’t take lunch breaks or go home for the weekend. It’s scanning like this that creates giant botnets like Mirai. The scans are entirely spray-and-pray, and they don’t care who knows it because at the end of the day they’re going to collect enough bots to take down

The moral of the story is that online everything is being constantly scanned- its just not one of those things they always talk about in school. And fortunately or unfortunately we have a lot of chicken little technology in security that expects you to filter the noise. One of the best skills I learnt when I was starting out was to dig around and learn some of these “truths” quickly so I didn’t spend cycles spinning when it really mattered. So, spin up your own honey pot, or start to talk to those you know about how you can learn more. We need as many curious minds as we can get.

Troy Kent
Troy Kent

Threat Researcher