It’s not you, it’s your threat hunting tools

, , ,

Paul, John and myself had a very cool conversation regarding why threat hunting is so much harder than it has to be (spoiler: Our tools are not doing their job). I decided to write a blog entry about the points we were making so that I could incorporate some visual aids.

Consider the following threat hunting scale which describes your possible reaction to a session or set of log entries you are analyzing:

Think of this as a threat level rating where a “0” means “I am 100% convinced that this is perfectly safe” and “100” which means “I’m 100% convinced that this is absolutely evil”. Both of these extremes are great from a threat hunting standpoint because we know how to process the event (ignore it or initiate our incident handling runbook). It takes very little time to disposition and move on to the next event we need to analyze.

The problem is events that score somewhere in the middle. This is that nasty gray area where I am unable to perform a quick disposition. If I see an event and sore it as a 50, I then need to collect further data to move the score up or down the threat scale. This is where threat hunting becomes hard and time consuming. I need to move that needle closer to 0 or 100 so that I can move on to the next event that needs to be analyzed.

Where Most Tools Fall Short

Where most threat hunting tools fall short is that they are focused on collecting as much information as possible, not with helping you identify the relative threat scale of each piece of information being provided. If you think about it, this is actually counterproductive. If I increase from 10 to 1,000 data points but still don’t have context or threat rating, I’m actually in worse shape than when I started because there are a lot more events I need to sort through. I’ve increased the amount of noise around my signal which is never a good thing.

This is why threat hunting is such a tribal activity. It is heavily reliant on the skill set of the person performing the threat hunt. For example someone like John Strand, who has been doing threat hunting for many years, can probably quickly disposition threats between 0 – 40 and 60 – 100. That just leaves threats that fall between 40 – 60 that chew up his time. A junior analyst however, may have a gray area that runs closer to 10 – 90. This means they will need to deep dive on a lot more events.

Sorting Through The Noise

When we created AI-Hunter, we had one goal in mind, to bring threat hunting to the masses. By definition this means that we cannot simply throw lots of data at you and hope that some of it sticks. The tool needs to do the first pass at data analysis and add or subtract threat points to each of the systems being analyzed. This is the only way to reduce the time it takes to perform a successful threat hunt.

For example, AI-Hunter will sort through all of your communication sessions and assign a high score to the ones that most closely match a compromised system beaconing home. So rather than needing to review every communication session for clues of beaconing behavior, that work has already been done for you.  

All that’s left is to identify if this pattern is expected behavior. Is this an NTP session to a known NTP server? If so we can whitelist the target and move on to the next session for analysis. Is the target IP listed on one or more blacklists? If so we need to have someone investigate the internal system because we’ve probably been whacked.

What to Look For in a Threat Hunting Tool

Regardless of whether you are looking at AI-Hunter or some other threat hunting tool, there are three basic questions you need to ask yourself:

  1. Will the tool reduce the time we spend performing threat hunts, or simply give us more data that needs to be reviewed?
  2. Can a junior analyst be successful with the tool or will I be forced to surrender even more time from my senior staff?
  3. Can I write a runbook around using the tool so that different analysts will produce consistent results?

If you cannot respond positively to each of the above points, it’s time to look at other tools.

New Version in the Wild

, , ,

Hey folks!

We released another new version of AI-Hunter last week. It’s been so crazy busy, I have not had a chance to announce it until now. There are some features in this release that have me super excited!

We’ve had some cosmetic changes to clean up the interface. Those will be apparent when you login. What I want to dive into here are the features that will make threat hunting your network that much easier. Let’s start by talking about ASN whitelisting.

I’ve discussed AI-Hunter’s whitelisting feature in the past. Along with all the other ways you can identify systems as trustworthy, you can now whitelist based on the Autonomous System Number (ASN) associated with the source or destination address of a session. As a use case example, consider Microsoft’s network. You are probably willing to trust any beacon like activity that takes place between internal systems and Microsoft’s patching servers. That’s expected behavior. However, you may have an entirely different opinion of Microsoft Azure, as anyone can spin up servers in that space.

Microsoft has 23 different ASNs to route traffic to different portions of their network. By whitelisting ASN 8075, you’ll trust all of Microsoft’s patching systems, even when they change IP address, without trusting system running in Azure. This is much easier than trying to whitelist based on IP or subnet!

There is a new reporting mode that lets you easily copy/paste information out of the user interface without all of the HTML code. This is handy when you need to Slack the info to a teammate or open a Jira ticket to get a forensic analysis performed.

Finally, we’ve added a feature to display the hostname a system was trying to reach when it generated the session. This is not a simple lookup of the IP address to see what hostname is associated with it (PTR query). We actually look at the DNS A or AAAA record query generated by the host and pull the name from that session. Here’s an example:

In this example, the IP address is part of a cloud hosting provider. So looking up the IP address is simply going to show us whatever name the hosting provider assigned. However because AI-Hunter is capturing the name server queries, we get to see the actual FQDN our internal system was attempting to reach. This can go a long way towards helping us identify if we are looking at malicious activity.

We have another release in the works so please stay tuned!

Whitelisting Import/Export Feature

,

Yet again we have received more awesome feedback from our customers!

AI-Hunter gives you the ability to “whitelist” IP addresses that you wish to exclude from your threat hunting analysis. It’s a popular feature, as you can whitelist based on individual IP address, subnets, or even full autonomous system numbers (ASNs). It’s a great way to move the stuff you understand out-of-the-way so you can focus in on the actual threats.

One of the requested features was the ability to share whitelists. For example, let’s say you have multiple analysts using AI-Hunter. You want to ensure they are whitelisting a consistent list of targets so that analysis becomes more consistent across multiple team members. We’ve even even had customers request a “best practice” list of systems to whitelist so they can quickly ignore targets that show beaconing behavior but are known to be safe (patching servers are a great example).

We are happy to announce that we have a new version going through testing that permits you to import and export whitelists. In fact, we took this feature one step further. With AI-Hunter, you will be able to load multiple whitelists and combine them together. For example, let’s say you are an international organizations with sites across the globe. There are some targets you wish to whitelist globally, but you have others that should only be whitelisted regionally. This new feature will permit you to create multiple whitelists and combine them as your needs require. Further, the save file is in JSON format. So you are free to edit and manage the lists outside of the AI-Hunter interface. Pro tip: Check these into your software repository (Github or similar) so you can maintain revision control.

We plan on including a default whitelist you can choose to load prior to analysis. This will include all of the IPs we’ve identified that exhibit beacon like behaviour but are actually legitimate (false positives).  

As always we love hearing from our customers so if you have comments or questions on any of the features within AI-Hunter please drop me a line.

Threat Hunting as a Process

, , ,

We keep hearing this question, “How do I get started with threat hunting and what process should be used?” From a high level perspective, here’s what your threat hunting process should look like:

If you’ve worked in incident handling or forensics, the process and many of its steps should look familiar. Here’s a bit of color around each of the steps.

Education

Threat hunting is arguably the most difficult security discipline to master. It requires a working knowledge of multiple security subjects including packet decoding, intrusion detection, forensics, malware analysis and incident handling. Further, attackers are updating their techniques all the time so a good threat hunting program needs to include constant educational updates.

Preparation

Once you understand how attackers maintain connectivity with their compromised hosts, you need to identify how you will go about detecting this activity. Will you monitor the network, system logs, or both? What tools will you use and how? Who will be responsible for each step in the process? Will it be a single team or multiple? Will network hardware and Internet of Things (IoT) devices be monitored as well? It may be helpful to run a simulated event from start to finish in order to ensure you have all of the correct pieces in place.

Detection

While “Preparation” pulls all of the needed tools and processes together, “Detection” is their day-to-day execution. Your run books should identify how your tools and processes will be used to detect suspect activity. They should also identify how potential detections should be escalated including what other resources can be drawn on when needed.

Forensic Analysis

If you have a documented forensic analysis processes, you can simply leverage it when needed. This will include everything from threat isolation to a detailed analysis of the compromise. A good forensic analysis should identify both when and how the attacker gained a foothold on your network. It should also identify if any other additional systems have been impacted.

Recovery

Recovery identifies how we return to a normal operating state once the threat has been identified. Typically, recovery will include both short-term and long-term responses. For example a short-term response may be to swap out the impacted system with another capable of providing the same services. The long-term solution may be to identify how the attacker gained access, and implement steps to ensure it does not happen again (missing patches, loose firewall rules, etc.). It’s not uncommon for the long-term recovery plan to come out of the blameless postmortem process.

Blameless Postmortem

While some may refer to this as “lessons learned”, I greatly prefer the more descriptive “blameless postmortem” as it helps to point the analysis in the right direction. It is far too easy, and common, to try to blame a specific individual for an event. “Bob clicked an email he shouldn’t have” or “Betty missed installing a patch”. By blaming a person instead of a process, we can go back to business as usual and act surprised when it happens again. Humans are fallible. Our processes need to take this into account.

So the goal of a blameless postmortem is to identify how our processes can be improved. This may include keeping the attackers off of our network or expediting the process of detection once they have gained access. What we learn should also be fed back into the educational portion of our process so that we see continuous improvement.

We will dig deeper into each of the steps in this process in future blog entries.

New Beacon Graph in the Works

, , ,

One of the things I love about this gig is talking to customers, especially those that are willing to share their input. I have been receiving feedback that the beaconing detail graph was a bit confusing for first time users. If you are not familiar with it, this is what it looks like:

While the design is slick and aesthetically pleasing, there was a bit of confusion around the concentric circles. Do the color codes mean anything? What about the thickness of the lines? Does it convey information if the circle is towards the outside versus the inside of the circle? Good UI should always be intuitive. You should not have to wrestle with questions like these in order to get your job done.

So based on customer feedback, we reworked the graph:

It conveys all of the same information, but in a much cleaner format. Now an analyst can just focus on getting their job done. This change is currently moving through engineering and testing. We expect it to go live in the very near future.

Are you a customer with a suggested improvement? We listen and absolutely value your feedback!

tim-gouw-68319-unsplash

What Is Threat Hunting and Why Do I Need It?

“Do I need to threat hunt my network?”

I heard this question a lot. You’ve deployed one or more firewalls, intrusion detection systems, locked down your servers and desktops, deployed anti-virus and you review logs on a regular basis. So why add threat hunting to the list?

Each of the above technologies is a preventive measure against known attack patterns. For example firewall’s keep the outside world from connecting to services you do not wish to expose while intrusion detection looks for signatures of a known threat pattern. None of these technologies answer the question “Did one or more of my protective measures fail?”. While it can be argued that log review may be able to answer this question, again it typically needs to be a known vector. For example you may detect an external IP address connecting via SSH because you know what to look for. You may completely miss a buffer overflow that provides high level access to a server because you have not seen this attack pattern in the past.

Threat hunting is the active exploration of your network in order to determine if one or more of your preventive measures has failed. Imagine you have a room full of secrets. You have locks on the door, motion sensors in the hallways, infrared around the building, and a host of other technologies straight out of your favorite spy movie. While all of these measures would be designed to keep people out or detect attempted entry, threat hunting would be actually checking the room to see if a perpetrator has made it in side. So think of threat hunting as being the ultimate integrity verification. Without it you are simply hoping that your preventative measures are working as expected.

So how do you get started with a threat hunting program? We’ll jump into that in future blog entries.