These are the slides from our 9/11/18 talk on performing a threat hunting beacon analysis.
Thank you to everyone who joined the webcast!
These are the slides from our 9/11/18 talk on performing a threat hunting beacon analysis.
Thank you to everyone who joined the webcast!
I briefly considered titling this blog entry “Why Threat Hunting Scope Creep Will Swallow Your Soul”, because I’ve spoken with quite a few folks that are completely overwhelmed with tasks the get labeled as “threat hunting”. Many feel like there are not enough hours in the day to provide full coverage of the typical corporate network. When you have one or more 500Mb links, and thousands of systems to keep track of, the raw quantity of data that needs to be reviewed can seem like an Impossible undertaking. However, I think much of this is actually a scope creep problem, and its only going to get worse unless we lock down what “threat hunting” actually means.
Imagine you are going through packet logs and you find evidence of a possible beacon. You see there is consistency in the timing and packet size. You investigate the external system, and it is not apparent why one of your internal systems would be communicating with this host. The service port being used is not a well-known port. You review the packet decode and the data appears to be encrypted or obfuscated with no obvious application headers. This makes you highly suspicious, so you jump to reviewing logs in your SIEM or go hands-on-keyboard with the system in question to run tools like Windows Resource Monitor, or the ‘netstat -on’ command in order to identify the process creating the connections.
Notice what we did in that last statement. We very quickly moved from executing a threat hunt to executing a forensic analysis. By conflating the two we have dramatically increased the time it will take to complete a successful threat hunt, potentially tained the forensic evidence trail, and possibly circumvented the incidence response process. We have created more work for ourselves while damaging the integrity of our response.
Note that in the previous example we were following a logical process. We were collecting evidence and simply followed the path that the evidence trail created. Unfortunately, our overzealousness caused us to charge boldly into other disciplines. While it is clear that a deeper analysis of the system is needed, we can do more harm than good by not properly changing gears mentally or involving the correct teams.
Cyber threat hunting has one specific goal, to validate the integrity of every system connected to the network. This is a huge undertaking, which can be extremely challenging given the plethora of hardware and operating systems one has to evaluate. As a process, threat hunting looks something like this:
Note that the central decision is identified as “Possible threat detected”. The operative word here is “possible”, as it is when we feel like we need definitive proof that we end up jumping down the rabbit hole into forensics. Forensics is an extremely time-consuming process. Think of threat hunting as the gatekeeper which identifies where those limited resources should be allocated. In this context, threat hunting becomes an extremely powerful tool with the capability to laser focus our forensic capability exactly when and where it is needed.
So imagine we’ve performed a threat hunt and we’ve identified five highly suspicious systems. What should we do next? At this point, our incident handling process should get triggered. The business threat of each of the potential targets should be evaluated and prioritized accordingly. For example, a system that appears to currently be exfiltrating data may rise to the top of the list. A system that is known to host company private data may take precedence over end-user systems.
Once the threats are prioritized, we can come up with the best plan of attack to address business risk as quickly as possible. For example, assume we have a very small forensic team and can only evaluate one system at a time. In that last example, we may decide to have them immediately focus on the system that hosts company private data, but simply retrieve the end user systems for later evaluation and have the help desk issue temporary loaner systems. This permits us to quickly mitigate all threats simultaneously while still working in the confines of our limited resources. This also helps to ensure that upper management is involved early in the process so that threats to the business are addressed holistically rather than just the security threat of one specific system.
Threat hunting is a new security discipline that is still evolving. During this growth stage, it is normal that its definition and boundaries are initially perceived as fluid. However, by tightly defining the discipline so that it does not overlap or conflict with other established processes, we can ensure that threat hunting is “a team player” in identifying and removing threats from our networks.
We’ve just released version 1.2 of AI-Hunter! There have been a number of tweaks and improvements, but there are three in particular that I wanted to highlight here.
We’ve had a number of folks express that AI-Hunter’s whitelisting capability is a real time saver. This allows you to quickly weed out false positives so that they don’t appear in later threat hunts. The problem was that once you have a few dozen whitelist entries, it can be easy to lose track of why each one was created. To resolve this issue, we’ve added the ability to add comments to whitelist entries.
You can use the comment space to identify why the whitelist entry was created. If you need to maintain a tight audit trail, you can even reference the ticket number responsible for creating the entry. The field is free-form, so you can add whatever additional info that you need.
We’ve also started including a recommended set of whitelist entries for folks to use. This will be available as a separate download via your Portal account. This is a list of the top conditions that we’ve found which may generate false positives. If you already have a set of whitelists you are using, you can merge our defaults in with your existing list. You can also use our list as a starting point and add your own entries as needed.
When analyzing beacon traffic, it can be extremely helpful to know what protocols and ports were being used during the communication session. In the past, you had to go back to the original Bro logs to retrieve this data. As of version 1.2, AI-Hunter now makes that info available directly in the beacons module. This should be a huge time saver!
For example, in the above screen capture, we can see that we are analyzing UDP/53 traffic, which contains DNS header information. We can also see that TCP/53 was used, but no DNS header is present. If additional protocols were used, we would see them listed here as well.
OK, this new feature is huge! The current modules in AI-Hunter are focused on threat vector. For example, you can check one module for beaconing behavior and another for connections that are constantly persistent. At the end of the day, however, we are looking to identify compromised systems, not necessarily threat vectors. Our new deep dive module is designed to tie all of these attack vectors together into a per system view.
Here’s an example screenshot:
There is a lot of data to unpack here, and I plan to expand on how to use this module in later blog entries but let me run you through the highlights:
In a previous blog post, I discussed the threat level rating scale and how our job as threat hunters is to move the needle as close to 0 or 100 as possible in order to disposition the session under analysis. This module is specifically designed to help you do that. By consolidating all of the data into a single view, it is now much easier to identify whether an internal system has been compromised and poses a threat to our environment.
The AI-Hunter user guide has a walkthrough on using this module, so please check that out after you upgrade. As mentioned, I’ll dig deeper into how to effectively use this module in later blog entries.
We are already working on the next release which will introduce even more goodies. Stay tuned to this channel for more info!
In part one of this two-part series, I described what is involved with performing a beacon analysis and why it is so important in catching the bad guys. In part two, I want to show you some open source and commercial tools that you can use to simplify the process of performing a network beacon analysis.
In part one I described the process for performing a beacon analysis during a threat hunt. Just to recap, those steps are:
While these steps are a challenge, we discussed that performing a beacon analysis is the only way to consistently identify malware calling home. However, there are tools that are designed to automate the above steps so that you can jump right into identifying which beacons may, in fact, be malicious.
Real Intelligence Threat Analytics, or RITA for short, is an open source tool that helps you identify compromised systems on your network calling home to C&C servers. It is designed to process Bro logs while running on Linux. It will even install Bro for you if it is not yet on the system. Once RITA collects 24-hours of Bro logs, it performs its analysis. You can read more about RITA here, or download it here.
RITA performs an extensive list of security checks, but one of the most unique is a beacon analysis. RITA breaks out the analysis based on sets of IP addresses. All communications are scrutinized for repeating intervals and even attempts to skew the results. The output is shown in the first figure.
The most important column is the first one, which is labeled “Score”, as this gives a score from 0-1 on the likelihood of communications between the two systems being a beacon. If you look closely at the first line, you’ll see that the score is .999774 between IP addresses 192.168.88.2 and 22.214.171.124. This is almost a perfect 1.0 score, so this is clearly beacon behavior. In fact, you may remember these IP addresses from the first half of this blog, as we identified communications between these two systems as being dnscat2.
From a threat hunter’s perspective, this is an extremely helpful tool. I simply start at the top of the list and work my way down. All of the hard work has been done for me.
AI-Hunter is an inexpensive commercial solution for threat hunting your network. It’s based on RITA, and also has an extensive toolkit for identifying compromised systems on your network. Unlike RITA, it has a Web based GUI interface which helps to expedite the threat hunting process. An example is shown in the second figure.
This is the same beacon we have been analyzing throughout this blog. Note that along with the beacon score, we also see a breakdown of the beacon frequency over time (bottom graph) as well as a breakdown of the interval dispersion (top right graph). If we switch to view 2, we can get a similar breakdown based on session data size rather than timing.
AI-Hunter also includes a number of tools that help expedite the threat hunting process. For example, I get to see metadata on the destination IP address. If I click the IP address, I can see if this address is on any known blacklists. Clicking the filter icon will let me whitelist these sessions if I deem them to be safe. If I want to see if any of my other systems have communicated with this external address, I can enter the IP address in the search dialog box in the top left of the screen, just below the title “Results”.
Beacon analysis is a critical threat hunting function. In some situations, it may be the only option available to identify a compromised system. While performing a beacon analysis manually is a huge chore, there are both open source and commercial tools available to expedite the process.
Beacon analysis is by far the most effective method of threat hunting your network. In fact, I would argue that if you are not checking your network for beacon activity, you have a huge gap in your defenses that attackers will happily leverage. In this two-part series, I’ll describe what is involved with performing a beacon analysis, why it is so important in catching the bad guys, and show you some open source and commercial tools you can use to simplify the process.
Malware infect desktops, servers and hardware can leverage a wide range of techniques to go undetected on the system. This is what makes host based threat hunting so problematic. Unless you know for sure the system is compromised, it is easy to miss any minor telltale clues. However, the one thing all malware strains have in common is that they need to be able to communicate with their author in order to execute their marching orders.
In the past, malware authors used to connect directly to the systems that they compromised. Today, most systems sit behind a firewall, limiting the ability to access them from the Internet. So if an attacker can fool one of your employees into infecting their own system, the attacker can’t count on having direct access to the system because a firewall will most likely block their access. This is the good news.
The bad news is, attackers have found a way around this problem. The solution is to use an intermediary server called a “Command and Control” (C&C) server. The basic setup is shown in the first figure.
When a system becomes infected, it generates an outbound connection across the internet to the attacker’s C&C server. Typically this connection will try and mimic normal traffic patterns by using HTTP, HTTPS or DNS. From a cursory view, the traffic will look like normal network activity. The intent of the connection is to inform the C&C server that a new compromised system has been activated and that the system is ready and waiting for marching orders. The process will then sleep for some period of time before repeating the check in process.
When the attacker wishes to activate the compromised system, they simply cue up a command on the C&C server. The next time the compromised system checks in, they have relayed the commands and execute on whatever marching orders have been given to them. These marching orders can be anything from stealing information off of the local system (data exfiltration) to attacking some identified host out on the Internet (DDoS attack).
Within the security industry, this behavior of calling home at regular intervals is referred to as “beaconing”. While on the surface beaconing can appear similar to normal network traffic, there are some unique traits we can look for as part of a network threat hunt. These traits revolve around the timing of the communications and the packet size being used.
As shown in the above example, a beaconing system calls home at regular intervals. This could be as quick as every 8-10 seconds or as long as a few times a day. It really depends on how patient the attacker is and how long they feel they can avoid detection. If the attack is concerned that their malware may be detected quickly, they may beacon more frequently in order to maximize system use prior to detection. There really is no specific time interval that all attackers use, which again contributes to the difficulty in detecting beacons.
Most network activity is random in its timing. For example, you may frequently use Google to perform searches, but it is unlikely that you use it exactly at the top of the hour, every hour. You leverage Google when you need it, not at some fixed time interval. So the predictable nature of beacon timing is one of the unique characteristics we can clue in on.
As noted in the first figure, the compromised system will spend a lot of time checking in only to find there are no marching orders for it to execute. This communication exchange of checking in and being told there is nothing to do uses a fixed set of commands. The result is that all of these sessions where the malware has nothing to do will result in identical amounts of data being exchanged. Even if the attacker takes steps to obfuscate the data, the size will remain consistent.
Most network activity is random in the amount of data exchanged in each session. For example, visiting multiple web pages on the same site will return images, text and code of various lengths. This will cause each session generated to transfer different amounts of data. So another predictable characteristic of beaconing behavior is consistency if the amount of data transferred per session.
Beaconing is a communication characteristic. It’is not good or evil, but just a way of describing the communication flow. While beaconing is heavily relied on by call home software, there are in fact times that legitimate software can exhibit beaconing behavior as well.
The most common false positive you will see is Network Time Protocol (NTP). NTP is used to ensure that the time on the local system remains accurate. NTP will beacon at a consistent interval in order to check the current time and ensure that the local system clock has not drifted. The beacon interval varies with different operating systems, but it is usually once every 15 to 60 minutes. Further, because NTP asks the same question each time (What’s the current time?), and gets back a fixed length answer, the amount of data transferred in each session is the same. So while NTP is the perfect example of beaconing behavior, it’s a vital tool that’s needed by every system and hardware device.
Luckily it is pretty easy to eliminate false positives. Our environment should be using a set of documented NTP servers to sync time. We can simply create an exception or “whitelist” these systems, thus ignoring any detected beacon activity. For example, let’s say our systems sync time with stratum2.example.com. We now know that we can safely ignore all UDP/123 traffic going to that system as part of our beacon analysis.
I’m not going to lie to you. Manually performing a beacon analysis is very difficult. There are a number of challenges that need to be overcome just to get the data into a format where a proper theat hunt is possible. Here are the basic steps:
Clearly just getting the data into a format where it can be used for threat hunting is pretty challenging. You need to tap the network, deploy hardware capable of writing packets at wire speed and have sufficient storage for two copies of all packets (one is the original capture, the other is the one manipulated for analysis). Multiply that by the number of days you want to store the info in case you need to go back through a threat hunt (if I find something evil today I may want to go back through old data to verify when it first arrived). You then need to do multiple passes through the data to sort and organize it in a way that helps highlight beaconing behavior. Whew!
The payoff is profound visibility into your network. Have a look at the second figure where I have used tshark to extract the interesting portions of a communication session between two IP addresses.
The columns from left to right are source IP, destination IP, protocol (17 is UDP), port number (53 is the well known port for DNS), the size of the packet, and finally the time gap between the packets shown on the screen. Two things should immediately jump out at you. First, every packet is the same size (89 bytes). This is an obvious predictable pattern. Second, the sessions are being consistently initiated about once per second, and are tightly grouped together around that one second mark. This makes the time interval extremely predictable as well. Finally, since this is UDP/53 traffic, we expect it to be DNS. DNS has no inherent beaconing behavior like NTP, which makes this pattern extremely abnormal.
From a threat hunter’s perspective, this pattern tells me that the source IP is most likely compromised and a host based forensic analysis is warranted. Further, I will want to ensure that my threat hunt looks careful for this behavior from any of my other systems. It’s worth noting that this is dnscat2 traffic. The tool creates an encrypted communication channel over DNS. All communications are DNS compliant, so it will happily tunnel through a DNS proxy server. Since all communications are encrypted, there is no pattern to try and find with your IDS or IPS. Shy of blind luck, performing a beacon analysis is one of your few options available for identifying when this tool is being used on your network without generating a huge number of false positives.
As you can see, manually performing a beacon analysis can be a huge chore. In part two I’ll talk about RITA, an open source tool you can leverage to dramatically simplify the process. I’ll also talk about AI-Hunter, a commercial solution, which can simplify the process even further.
We just performed another release this week and I’m psyched to fill you in on some of the new features! The two biggest are the addition of a module to analyze long communication sessions and a module that analyzes long URIs. Let’s jump into a bit about each.
As you may know, the beacons module analyzes network data in 24 hour chunks. This way, even when a system beacons home infrequently, we still collect enough data points to tag the system as a beacon. However, this begs the question, what if the malware does not beacon home? What if the attacker simply sets up a session back to the command and control server and leaves it active all of the time? This is where long connection analysis comes in.
By selecting the “long connections” module, you will be presented with a list of the longest duration connections taking place on your network. The list is sorted from longest to shortest, but you can change this using the sort option in the top left of the screen. You can also set a threshold for the minimum connection duration time you want to be shown.
There are times when long connections are expected behavior. For example, BGP sessions between routers tend to persist as long as possible in order to update route information. There are even times when long connections are not optimal, but not necessarily evil. A good example would be an SSH user that leaves their connections active even when they are done on the system. For the most part, however, excessively long connection times (more than an hour) should be scrutinized carefully. If they are not evil, they may at least be in breach of policy.
Our customers love AI-Hunter’s ability to identify when DNS is being used as a backdoor. Our new URL analysis module brings that same type of functionality to analyzing URIs. What may look like an internal system connecting to an external Web server may, in fact, be malware encoding information into the URI and passing that data out of your network. The url module is designed to allow you to quickly hone in on that type of activity.
Similar to the long connections module, the URL module displays the connections that took place with the greatest amount of URI data. Also, like the long connections module, you can change the sort option or the threshold on minimum URI length to report. Most users should be fine going with the defaults.
When URI exfiltration is taking place, it is not uncommon to see the data passed in the clear. However, using a simple XOR to obfuscate the data pattern, is a common technique as well. This means it can be helpful to extract the URI information for additional processing. If you mouse over the URI, a dialog box will appear showing the complete URI (this normally cannot be displayed due to its length). If you click on the report icon, you can download the data in comma separated format. This lets you perform additional processing in another tool.
A huge thank you goes out to our beta testers that helped us vet these new modules!
We have another new module in the works and wow, this one is a game changer. It really ties together the whole threat hunting experience! At ACM our goals are to simplify the threat hunting process so that:
This new module will go a long way toward accomplishing both of those goals. Intrigued? Stay tuned to this blog/channel over the next few weeks for further announcements.
We released another new version of AI-Hunter last week. It’s been so crazy busy, I have not had a chance to announce it until now. There are some features in this release that have me super excited!
We’ve had some cosmetic changes to clean up the interface. Those will be apparent when you login. What I want to dive into here are the features that will make threat hunting your network that much easier. Let’s start by talking about ASN whitelisting.
I’ve discussed AI-Hunter’s whitelisting feature in the past. Along with all the other ways you can identify systems as trustworthy, you can now whitelist based on the Autonomous System Number (ASN) associated with the source or destination address of a session. As a use case example, consider Microsoft’s network. You are probably willing to trust any beacon like activity that takes place between internal systems and Microsoft’s patching servers. That’s expected behavior. However, you may have an entirely different opinion of Microsoft Azure, as anyone can spin up servers in that space.
Microsoft has 23 different ASNs to route traffic to different portions of their network. By whitelisting ASN 8075, you’ll trust all of Microsoft’s patching systems, even when they change IP address, without trusting system running in Azure. This is much easier than trying to whitelist based on IP or subnet!
There is a new reporting mode that lets you easily copy/paste information out of the user interface without all of the HTML code. This is handy when you need to Slack the info to a teammate or open a Jira ticket to get a forensic analysis performed.
Finally, we’ve added a feature to display the hostname a system was trying to reach when it generated the session. This is not a simple lookup of the IP address to see what hostname is associated with it (PTR query). We actually look at the DNS A or AAAA record query generated by the host and pull the name from that session. Here’s an example:
In this example, the IP address is part of a cloud hosting provider. So looking up the IP address is simply going to show us whatever name the hosting provider assigned. However because AI-Hunter is capturing the name server queries, we get to see the actual FQDN our internal system was attempting to reach. This can go a long way towards helping us identify if we are looking at malicious activity.
We have another release in the works so please stay tuned!
Yet again we have received more awesome feedback from our customers!
AI-Hunter gives you the ability to “whitelist” IP addresses that you wish to exclude from your threat hunting analysis. It’s a popular feature, as you can whitelist based on individual IP address, subnets, or even full autonomous system numbers (ASNs). It’s a great way to move the stuff you understand out-of-the-way so you can focus in on the actual threats.
One of the requested features was the ability to share whitelists. For example, let’s say you have multiple analysts using AI-Hunter. You want to ensure they are whitelisting a consistent list of targets so that analysis becomes more consistent across multiple team members. We’ve even even had customers request a “best practice” list of systems to whitelist so they can quickly ignore targets that show beaconing behavior but are known to be safe (patching servers are a great example).
We are happy to announce that we have a new version going through testing that permits you to import and export whitelists. In fact, we took this feature one step further. With AI-Hunter, you will be able to load multiple whitelists and combine them together. For example, let’s say you are an international organizations with sites across the globe. There are some targets you wish to whitelist globally, but you have others that should only be whitelisted regionally. This new feature will permit you to create multiple whitelists and combine them as your needs require. Further, the save file is in JSON format. So you are free to edit and manage the lists outside of the AI-Hunter interface. Pro tip: Check these into your software repository (Github or similar) so you can maintain revision control.
We plan on including a default whitelist you can choose to load prior to analysis. This will include all of the IPs we’ve identified that exhibit beacon like behaviour but are actually legitimate (false positives).
As always we love hearing from our customers so if you have comments or questions on any of the features within AI-Hunter please drop me a line.