It’s not you, it’s your threat hunting tools

Paul, John and myself had a very cool conversation regarding why threat hunting is so much harder than it has to be (spoiler: Our tools are not doing their job). I decided to write a blog entry about the points we were making so that I could incorporate some visual aids.

Consider the following threat hunting scale which describes your possible reaction to a session or set of log entries you are analyzing:

Think of this as a threat level rating where a “0” means “I am 100% convinced that this is perfectly safe” and “100” which means “I’m 100% convinced that this is absolutely evil”. Both of these extremes are great from a threat hunting standpoint because we know how to process the event (ignore it or initiate our incident handling runbook). It takes very little time to disposition and move on to the next event we need to analyze.

The problem is events that score somewhere in the middle. This is that nasty gray area where I am unable to perform a quick disposition. If I see an event and sore it as a 50, I then need to collect further data to move the score up or down the threat scale. This is where threat hunting becomes hard and time consuming. I need to move that needle closer to 0 or 100 so that I can move on to the next event that needs to be analyzed.

Where Most Tools Fall Short

Where most threat hunting tools fall short is that they are focused on collecting as much information as possible, not with helping you identify the relative threat scale of each piece of information being provided. If you think about it, this is actually counterproductive. If I increase from 10 to 1,000 data points but still don’t have context or threat rating, I’m actually in worse shape than when I started because there are a lot more events I need to sort through. I’ve increased the amount of noise around my signal which is never a good thing.

This is why threat hunting is such a tribal activity. It is heavily reliant on the skill set of the person performing the threat hunt. For example someone like John Strand, who has been doing threat hunting for many years, can probably quickly disposition threats between 0 – 40 and 60 – 100. That just leaves threats that fall between 40 – 60 that chew up his time. A junior analyst however, may have a gray area that runs closer to 10 – 90. This means they will need to deep dive on a lot more events.

Sorting Through The Noise

When we created AI-Hunter, we had one goal in mind, to bring threat hunting to the masses. By definition this means that we cannot simply throw lots of data at you and hope that some of it sticks. The tool needs to do the first pass at data analysis and add or subtract threat points to each of the systems being analyzed. This is the only way to reduce the time it takes to perform a successful threat hunt.

For example, AI-Hunter will sort through all of your communication sessions and assign a high score to the ones that most closely match a compromised system beaconing home. So rather than needing to review every communication session for clues of beaconing behavior, that work has already been done for you.  

All that’s left is to identify if this pattern is expected behavior. Is this an NTP session to a known NTP server? If so we can whitelist the target and move on to the next session for analysis. Is the target IP listed on one or more blacklists? If so we need to have someone investigate the internal system because we’ve probably been whacked.

What to Look For in a Threat Hunting Tool

Regardless of whether you are looking at AI-Hunter or some other threat hunting tool, there are three basic questions you need to ask yourself:

1. Will the tool reduce the time we spend performing threat hunts, or simply give us more data that needs to be reviewed?
2. Can a junior analyst be successful with the tool or will I be forced to surrender even more time from my senior staff?
3. Can I write a runbook around using the tool so that different analysts will produce consistent results?

If you cannot respond positively to each of the above points, it’s time to look at other tools.