Tightly Defining Cyber Threat Hunting

I briefly considered titling this blog entry “Why Threat Hunting Scope Creep Will Swallow Your Soul”, because I’ve spoken with quite a few folks that are completely overwhelmed with tasks the get labeled as “threat hunting”. Many feel like there are not enough hours in the day to provide full coverage of the typical corporate network. When you have one or more 500Mb links, and thousands of systems to keep track of, the raw quantity of data that needs to be reviewed can seem like an Impossible undertaking. However, I think much of this is actually a scope creep problem, and its only going to get worse unless we lock down what “threat hunting” actually means.

Threat Hunting Scope Creep Example

Imagine you are going through packet logs and you find evidence of a possible beacon. You see there is consistency in the timing and packet size. You investigate the external system, and it is not apparent why one of your internal systems would be communicating with this host. The service port being used is not a well-known port. You review the packet decode and the data appears to be encrypted or obfuscated with no obvious application headers. This makes you highly suspicious, so you jump to reviewing logs in your SIEM or go hands-on-keyboard with the system in question to run tools like Windows Resource Monitor, or the ‘netstat -on’ command in order to identify the process creating the connections.

Notice what we did in that last statement. We very quickly moved from executing a threat hunt to executing a forensic analysis. By conflating the two we have dramatically increased the time it will take to complete a successful threat hunt, potentially tained the forensic evidence trail, and possibly circumvented the incidence response process. We have created more work for ourselves while damaging the integrity of our response.

What is Cyber Threat Hunting?

Note that in the previous example we were following a logical process. We were collecting evidence and simply followed the path that the evidence trail created. Unfortunately, our overzealousness caused us to charge boldly into other disciplines. While it is clear that a deeper analysis of the system is needed, we can do more harm than good by not properly changing gears mentally or involving the correct teams.

Cyber threat hunting has one specific goal, to validate the integrity of every system connected to the network. This is a huge undertaking, which can be extremely challenging given the plethora of hardware and operating systems one has to evaluate. As a process, threat hunting looks something like this:

Note that the central decision is identified as “Possible threat detected”. The operative word here is “possible”, as it is when we feel like we need definitive proof that we end up jumping down the rabbit hole into forensics. Forensics is an extremely time-consuming process. Think of threat hunting as the gatekeeper which identifies where those limited resources should be allocated. In this context, threat hunting becomes an extremely powerful tool with the capability to laser focus our forensic capability exactly when and where it is needed.

Best Practice Next Steps

So imagine we’ve performed a threat hunt and we’ve identified five highly suspicious systems. What should we do next? At this point, our incident handling process should get triggered. The business threat of each of the potential targets should be evaluated and prioritized accordingly. For example, a system that appears to currently be exfiltrating data may rise to the top of the list. A system that is known to host company private data may take precedence over end-user systems.

Once the threats are prioritized, we can come up with the best plan of attack to address business risk as quickly as possible. For example, assume we have a very small forensic team and can only evaluate one system at a time. In that last example, we may decide to have them immediately focus on the system that hosts company private data, but simply retrieve the end user systems for later evaluation and have the help desk issue temporary loaner systems. This permits us to quickly mitigate all threats simultaneously while still working in the confines of our limited resources. This also helps to ensure that upper management is involved early in the process so that threats to the business are addressed holistically rather than just the security threat of one specific system.

Summary

Threat hunting is a new security discipline that is still evolving. During this growth stage, it is normal that its definition and boundaries are initially perceived as fluid. However, by tightly defining the discipline so that it does not overlap or conflict with other established processes, we can ensure that threat hunting is “a team player” in identifying and removing threats from our networks.