Version 1.2 Release

, , ,

We’ve just released version 1.2 of AI-Hunter! There have been a number of tweaks and improvements, but there are three in particular that I wanted to highlight here.

Whitelist Comments and Default Recommendations

We’ve had a number of folks express that AI-Hunter’s whitelisting capability is a real time saver. This allows you to quickly weed out false positives so that they don’t appear in later threat hunts. The problem was that once you have a few dozen whitelist entries, it can be easy to lose track of why each one was created. To resolve this issue, we’ve added the ability to add comments to whitelist entries.


You can use the comment space to identify why the whitelist entry was created. If you need to maintain a tight audit trail, you can even reference the ticket number responsible for creating the entry. The field is free-form, so you can add whatever additional info that you need.

We’ve also started including a recommended set of whitelist entries for folks to use. This will be available as a separate download via your Portal account. This is a list of the top conditions that we’ve found which may generate false positives. If you already have a set of whitelists you are using, you can merge our defaults in with your existing list. You can also use our list as a starting point and add your own entries as needed.

Protocol IDentification

When analyzing beacon traffic, it can be extremely helpful to know what protocols and ports were being used during the communication session. In the past, you had to go back to the original Bro logs to retrieve this data. As of version 1.2, AI-Hunter now makes that info available directly in the beacons module. This should be a huge time saver!


The syntax is:
(transport / port; service)

For example, in the above screen capture, we can see that we are analyzing UDP/53 traffic, which contains DNS header information. We can also see that TCP/53 was used, but no DNS header is present. If additional protocols were used, we would see them listed here as well.

New Deep Dive Module

OK, this new feature is huge! The current modules in AI-Hunter are focused on threat vector. For example, you can check one module for beaconing behavior and another for connections that are constantly persistent. At the end of the day, however, we are looking to identify compromised systems, not necessarily threat vectors. Our new deep dive module is designed to tie all of these attack vectors together into a per system view.

Here’s an example screenshot:

 

There is a lot of data to unpack here, and I plan to expand on how to use this module in later blog entries but let me run you through the highlights:

  • This view is system centric. You identify the internal IP you wish to analyze and we pull together data from all of the other modules as well as additional recon info.
  • The circular graphic shows all connections initiated by the system being analyzed. You can drag and zoom as needed to focus on different communication sessions.
  • You can click through on specific sessions to drill down on additional details.
  • The graph is color-coded to help draw your eye to potential threats.
  • You can review a summary of all communications, communications between a specific destination IP, and even step through info in unique sessions, all at the same time.

In a previous blog post, I discussed the threat level rating scale and how our job as threat hunters is to move the needle as close to 0 or 100 as possible in order to disposition the session under analysis. This module is specifically designed to help you do that. By consolidating all of the data into a single view, it is now much easier to identify whether an internal system has been compromised and poses a threat to our environment.

The AI-Hunter user guide has a walkthrough on using this module, so please check that out after you upgrade. As mentioned, I’ll dig deeper into how to effectively use this module in later blog entries.

We are already working on the next release which will introduce even more goodies. Stay tuned to this channel for more info!

Threat Hunting – Simplifying The Beacon Analysis Process

, , , , , ,

In part one of this two-part series, I described what is involved with performing a beacon analysis and why it is so important in catching the bad guys. In part two, I want to show you some open source and commercial tools that you can use to simplify the process of performing a network beacon analysis.

Beacon Analysis – Recap

In part one I described the process for performing a beacon analysis during a threat hunt. Just to recap, those steps are:

  • Capture all traffic flowing through a choke point to the Internet. The best choice is usually the internal interface of your firewall.
  • Capture and store enough traffic to record multiple instances of beacon activity. At a minimum, this is 12 hours of traffic. 24 hours is more ideal.
  • Whitelist out any traffic that may contain beacons that you know are safe. For example, any UDP/123 traffic going to known NTP servers.
  • Segregate the traffic into IP address pair combinations. For example, all traffic between 192.168.1.100 and 1.2.3.4 should go in one file, while all traffic between 192.168.1.100 and 1.2.3.5 should go in another. Note that we are not breaking out unique sessions. Each file needs to contain all traffic that was exchanged between the two identified IP addresses over the full-time interval of the capture.
  • Identify the time delta between the first packet in each session established between these two systems. Organize these by transport and target port that was used.
  • Identify the amount of data transferred in each session. Organize these by transport and target port that was used.
  • Review the data for any obvious patterns.

While these steps are a challenge, we discussed that performing a beacon analysis is the only way to consistently identify malware calling home. However, there are tools that are designed to automate the above steps so that you can jump right into identifying which beacons may, in fact, be malicious.

Threat Hunting Beacons with RITA

Real Intelligence Threat Analytics, or RITA for short, is an open source tool that helps you identify compromised systems on your network calling home to C&C servers. It is designed to process Bro logs while running on Linux. It will even install Bro for you if it is not yet on the system. Once RITA collects 24-hours of Bro logs, it performs its analysis. You can read more about RITA here, or download it here.

RITA performs an extensive list of security checks, but one of the most unique is a beacon analysis. RITA breaks out the analysis based on sets of IP addresses. All communications are scrutinized for repeating intervals and even attempts to skew the results. The output is shown in the first figure.

The most important column is the first one, which is labeled “Score”, as this gives a score from 0-1 on the likelihood of communications between the two systems being a beacon. If you look closely at the first line, you’ll see that the score is .999774 between IP addresses 192.168.88.2 and 165.227.88.15. This is almost a perfect 1.0 score, so this is clearly beacon behavior. In fact, you may remember these IP addresses from the first half of this blog, as we identified communications between these two systems as being dnscat2.

From a threat hunter’s perspective, this is an extremely helpful tool. I simply start at the top of the list and work my way down. All of the hard work has been done for me.

Threat Hunting Beacons With AI-Hunter

AI-Hunter is an inexpensive commercial solution for threat hunting your network. It’s based on RITA, and also has an extensive toolkit for identifying compromised systems on your network. Unlike RITA, it has a Web based GUI interface which helps to expedite the threat hunting process. An example is shown in the second figure.

This is the same beacon we have been analyzing throughout this blog. Note that along with the beacon score, we also see a breakdown of the beacon frequency over time (bottom graph) as well as a breakdown of the interval dispersion (top right graph). If we switch to view 2, we can get a similar breakdown based on session data size rather than timing.

AI-Hunter also includes a number of tools that help expedite the threat hunting process. For example, I get to see metadata on the destination IP address. If I click the IP address, I can see if this address is on any known blacklists. Clicking the filter icon will let me whitelist these sessions if I deem them to be safe. If I want to see if any of my other systems have communicated with this external address, I can enter the IP address in the search dialog box in the top left of the screen, just below the title “Results”.

Summary

Beacon analysis is a critical threat hunting function. In some situations, it may be the only option available to identify a compromised system. While performing a beacon analysis manually is a huge chore, there are both open source and commercial tools available to expedite the process.

Beacon Analysis – The Key to Cyber Threat Hunting

, , , , ,

Beacon analysis is by far the most effective method of threat hunting your network. In fact, I would argue that if you are not checking your network for beacon activity, you have a huge gap in your defenses that attackers will happily leverage. In this two-part series, I’ll describe what is involved with performing a beacon analysis, why it is so important in catching the bad guys, and show you some open source and commercial tools you can use to simplify the process.

Understanding Phone Home or Call Home Malware

Malware infect desktops, servers and hardware can leverage a wide range of techniques to go undetected on the system. This is what makes host based threat hunting so problematic. Unless you know for sure the system is compromised, it is easy to miss any minor telltale clues. However, the one thing all malware strains have in common is that they need to be able to communicate with their author in order to execute their marching orders.

In the past, malware authors used to connect directly to the systems that they compromised. Today, most systems sit behind a firewall, limiting the ability to access them from the Internet. So if an attacker can fool one of your employees into infecting their own system, the attacker can’t count on having direct access to the system because a firewall will most likely block their access. This is the good news.

The bad news is, attackers have found a way around this problem. The solution is to use an intermediary server called a “Command and Control” (C&C) server. The basic setup is shown in the first figure.

When a system becomes infected, it generates an outbound connection across the internet to the attacker’s C&C server. Typically this connection will try and mimic normal traffic patterns by using HTTP, HTTPS or DNS. From a cursory view, the traffic will look like normal network activity. The intent of the connection is to inform the C&C server that a new compromised system has been activated and that the system is ready and waiting for marching orders. The process will then sleep for some period of time before repeating the check in process.

When the attacker wishes to activate the compromised system, they simply cue up a command on the C&C server. The next time the compromised system checks in, they have relayed the commands and execute on whatever marching orders have been given to them. These marching orders can be anything from stealing information off of the local system (data exfiltration) to attacking some identified host out on the Internet (DDoS attack).

Beacon Characteristics

Within the security industry, this behavior of calling home at regular intervals is referred to as “beaconing”. While on the surface beaconing can appear similar to normal network traffic, there are some unique traits we can look for as part of a network threat hunt. These traits revolve around the timing of the communications and the packet size being used.

Beacon Timing

As shown in the above example, a beaconing system calls home at regular intervals. This could be as quick as every 8-10 seconds or as long as a few times a day. It really depends on how patient the attacker is and how long they feel they can avoid detection. If the attack is concerned that their malware may be detected quickly, they may beacon more frequently in order to maximize system use prior to detection. There really is no specific time interval that all attackers use, which again contributes to the difficulty in detecting beacons.

Most network activity is random in its timing. For example, you may frequently use Google to perform searches, but it is unlikely that you use it exactly at the top of the hour, every hour. You leverage Google when you need it, not at some fixed time interval. So the predictable nature of beacon timing is one of the unique characteristics we can clue in on.  

Beacon Packet Size

As noted in the first figure, the compromised system will spend a lot of time checking in only to find there are no marching orders for it to execute. This communication exchange of checking in and being told there is nothing to do uses a fixed set of commands. The result is that all of these sessions where the malware has nothing to do will result in identical amounts of data being exchanged. Even if the attacker takes steps to obfuscate the data, the size will remain consistent.

Most network activity is random in the amount of data exchanged in each session. For example, visiting multiple web pages on the same site will return images, text and code of various lengths. This will cause each session generated to transfer different amounts of data. So another predictable characteristic of beaconing behavior is consistency if the amount of data transferred per session.  

Beacon False Positives

Beaconing is a communication characteristic. It’is not good or evil, but just a way of describing the communication flow. While beaconing is heavily relied on by call home software, there are in fact times that legitimate software can exhibit beaconing behavior as well.

The most common false positive you will see is Network Time Protocol (NTP). NTP is used to ensure that the time on the local system remains accurate. NTP will beacon at a consistent interval in order to check the current time and ensure that the local system clock has not drifted. The beacon interval varies with different operating systems, but it is usually once every 15 to 60 minutes. Further, because NTP asks the same question each time (What’s the current time?), and gets back a fixed length answer, the amount of data transferred in each session is the same. So while NTP is the perfect example of beaconing behavior, it’s a vital tool that’s needed by every system and hardware device.

Luckily it is pretty easy to eliminate false positives. Our environment should be using a set of documented NTP servers to sync time. We can simply create an exception or “whitelist” these systems, thus ignoring any detected beacon activity. For example, let’s say our systems sync time with stratum2.example.com. We now know that we can safely ignore all UDP/123 traffic going to that system as part of our beacon analysis.

Beacon Analysis

I’m not going to lie to you. Manually performing a beacon analysis is very difficult. There are a number of challenges that need to be overcome just to get the data into a format where a proper theat hunt is possible. Here are the basic steps:

  • Capture all traffic flowing through a choke point to the Internet. The best choice is usually the internal interface of your firewall.
  • Capture and store enough traffic to record multiple instances of beacon activity. At a minimum, this is 12 hours of traffic. 24 hours is more ideal.
  • Whitelist out any traffic that may contain beacons that you know are safe. For example, any UDP/123 traffic going to known NTP servers.
  • Segregate the traffic into IP address pair combinations. For example, all traffic between 192.168.1.100 and 1.2.3.4 should go in one file, while all traffic between 192.168.1.100 and 1.2.3.5 should go in another. Note that we are not breaking out unique sessions. Each file needs to contain all traffic that was exchanged between the two identified IP addresses over the full time interval of the capture.
  • Identify the time delta between the first packet in each session established between these two systems. Organize these by transport and target port that was used.
  • Identify the amount of data transferred in each session. Organize these by transport and target port that was used.
  • Review the data for any obvious patterns.

Clearly just getting the data into a format where it can be used for threat hunting is pretty challenging. You need to tap the network, deploy hardware capable of writing packets at wire speed and have sufficient storage for two copies of all packets (one is the original capture, the other is the one manipulated for analysis). Multiply that by the number of days you want to store the info in case you need to go back through a threat hunt (if I find something evil today I may want to go back through old data to verify when it first arrived). You then need to do multiple passes through the data to sort and organize it in a way that helps highlight beaconing behavior. Whew!

The payoff is profound visibility into your network. Have a look at the second figure where I have used tshark to extract the interesting portions of a communication session between two IP addresses.

The columns from left to right are source IP, destination IP, protocol (17 is UDP), port number (53 is the well known port for DNS), the size of the packet, and finally the time gap between the packets shown on the screen. Two things should immediately jump out at you. First, every packet is the same size (89 bytes). This is an obvious predictable pattern. Second, the sessions are being consistently initiated about once per second, and are tightly grouped together around that one second mark. This makes the time interval extremely predictable as well. Finally, since this is UDP/53 traffic, we expect it to be DNS. DNS has no inherent beaconing behavior like NTP, which makes this pattern extremely abnormal.

From a threat hunter’s perspective, this pattern tells me that the source IP is most likely compromised and a host based forensic analysis is warranted. Further, I will want to ensure that my threat hunt looks careful for this behavior from any of my other systems. It’s worth noting that this is dnscat2 traffic. The tool creates an encrypted communication channel over DNS. All communications are DNS compliant, so it will happily tunnel through a DNS proxy server. Since all communications are encrypted, there is no pattern to try and find with your IDS or IPS. Shy of blind luck, performing a beacon analysis is one of your few options available for identifying when this tool is being used on your network without generating a huge number of false positives.

Simplifying The Beacon Analysis Process

As you can see, manually performing a beacon analysis can be a huge chore. In part two I’ll talk about RITA, an open source tool you can leverage to dramatically simplify the process. I’ll also talk about AI-Hunter, a commercial solution, which can simplify the process even further.

Portspoof

, , ,

In this video, John Strand goes through the setup and use of Portspoof. Portspoof is a great little cyber deception tool that can be used as a honeypot or as a way to frustrate attackers. You can configure Portspoof to listen on any TCP port of your choosing and even return service banner upon connection. The result is that port and vulnerability scanning tools will produce inaccurate results for any would-be attackers probing your environment.

Prevent Brute Force SSH Attacks in 30 Seconds

, , , ,

 

In this video, John shows off the “denyhost” utility from the “Active Defense Harbinger Distribution” security toolkit. The denyhost tool is extremely easy to deploy and blocks attackers from performing brute force SSH attacks. Here are all of the commands you need to run the tool:

sudo /etc/init.d/denyhosts start

To identify what IP addresses have failed their login attempt seven times and are now being blocked from access the server, type:

less /etc/hosts.deny

Should I Threat Hunt My Systems or the Network?

I made a comment in an earlier blog post that too much data without context can negatively impact the threat hunting process. This is why I’m a firm believer that you should hunt for adversaries using network data, and save host level logs for forensic analysis. In this blog post, I’ll dive into the reasons why I feel threat hunting should be a network based activity.

 

 

Network and Host Based Threat Hunting Defined

Just to ensure we are on the same page, I wanted to define a few terms. Cyber threat hunting is the process of searching the network infrastructure for compromised systems. The goal is to validate the integrity of every system connected to the network. Forensics is the process we leverage when we feel a compromise has been identified. While I will threat hunt every system and device on the network, I will only perform forensics on the systems I suspect of being compromised.

Network based threat hunting analyzes packets on the network. Typically you use a network tap or port mirroring to monitor all traffic flowing between networks of different trust levels. Typically, this is an internal network and the Internet, but the technique can also be applied to business partner connections. The goal is to look for signs of an internal system being compromised.

With host based threat hunting, I use logs entries that are generated by the hosts I wish to protect. Typically, these logs are forwarded to a centralized server where they can be correlated and analyzed. I then perform my threat hunt on the centralized server. Again, my goal is to find data in the logs that is indicative of a system being compromised.

Threat Hunting System Visibility

One of the core realities of threat hunting is that we need access to data to verify integrity. I cannot validate whether a system is still in a trusted state unless I have access to data which identifies that system’s current operational state. If I’m hunting on the network, visibility is a non-issue. Any compromised system calling home is going to use the local Internet connection. I just need tooling and process to distinguish that traffic from normal communication patterns.

With host based threat hunting, data access can be hit or miss. It can be extremely difficult to identify if you are collecting logs from all of your devices. In fact, you can be pretty certain you are not, given the number of BYOD, IoT and forgotten hardware devices (like printers) that end up on the typical network. Clearly, if I’m not getting logs from a device, I have no way of evaluating that device’s integrity.

Further, modern malware tries really hard to cover its tracks. So even if I am collecting log entries from a host, the malware that now owns that system may be altering logs in an effort to go undetected. This puts us in a catch-22 where we are relying on an untrusted source to provide accurate information in order to identify its integrity. As an analogy, imagine if the full extent of a police investigation was to simply ask various individuals if they committed the crime, and 100% trust the integrity of their word.

So think of network based threat hunting as the great equalizer. While an attacker may be able to hide the processes they are running on a system, they cannot hide their packets on the wire. The best they can hope for is to get lost in the rest of the traffic on the network.

Threat Hunt Data Consistency

One of the nice things about analyzing network traffic is that communications are fairly consistent. The IP protocols we use for communications are governed by RFC standards. This means that Secure Shell traffic originating from a Windows system is going to look the same as traffic originating from Linux, Mac OS, BSD or various IoT devices. This permits me to have one consistent process for analyzing these traffic patterns, regardless of operating system. Yes, each operating system can have its own nuances in the way it communicates, but these minor differences can actually be leveraged as part of the threat hunting process.

Reviewing system log data is anything but consistent. Most logging revolves around Syslog which is little more than a general framework. The result is a complete lack of consistency between operating systems and even applications running on the same OS. Windows and Linux record very different runtime messages. Two different Secure Shell applications are going to log different information, in slightly different formats, even if they are both running on the same OS.

The root cause is that Syslog leaves much of what it does define completely open to interpretation. Should a log entry for a failed login be assigned a severity of error, warning, notice or information? I’ve seen applications use all of these. The result is that I need intimate knowledge or every OS and application running on my network before I can even hope to detect suspect behavior. This also means I’m customizing my data processing techniques for every unique application. Add a new app and there is no guarantee your existing processes will validate it properly.

Sorting Through Data Noise

This brings me back to my original point that there is such a thing as too much data in threat hunting. I want to make a distinction here; there is a difference between actionable data which can be leveraged to make informed decisions and unrelated data that simply raises the noise level.

The first challenge is determining which data is actionable and which is unrelated. Is that Apache error a common runtime error or did someone just perform a buffer overflow? Since an application is going to log both actionable and unrelated information, you need to start by sorting through which is which before you can even begin threat hunting. This process can very quickly lead to decision paralysis.

Further, once you have determined which data is actionable and which is not, it can be a PITA to whitelist unrelated information from future threat hunts so you are not duplicating efforts. For example, you don’t want to ignore all Apache errors, just the ones that have nothing to do with an adversary compromising the endpoint. How exactly do you do that? You can’t, which means you end up ignoring a majority of log entries during every threat hunt. This can lead to anchoring bias, a common cause of ignoring actionable data even when it does appear in the logs.

With network based threat hunting, it’s far easier to sort through actionable versus unrelated data. All communications originating on the internal network and headed to the Internet are a potential threat. As a threat hunter, I need to understand and evaluate the business case for every one of those sessions. When a session is deemed safe, it’s a straightforward process to ignore those sessions from future threat hunts (ignore all HTTPS to www.google.com, ignore all NTP to my designated time servers, etc.). This allows me to iterate the threat hunting process over time. As I whitelist more and more sessions I understand as safe, the data set I need to review for potential threats gets smaller and smaller. This helps to speed up the threat hunting process over time.

When to Use Host Logs

This does not mean host logs have no value. When we threat hunt the network, we are trying to determine if each session is trustworthy or a potential threat. Sometimes the network does not have all the answers. For example, I may see that one of my internal systems is holding open a TLS protected session with an external host 24 hours a day. For me personally, I would be 75% certain that this is due to a compromise or an internal employee breaching policy. With this evidence in hand, I would now turn to host logs to identify what process is responsible for that session, and why it is running on the system. At this point, however, I’ve switched into more of a forensic analysis mode than threat hunting.

Summary

Many times we operate under the belief that if a little is good, too much must be just enough. That is absolutely not the case with data points when performing a threat hunt, as too many unrelated data points can unnecessarily extend the time it takes to perform a successful threat hunt, as well as lead to actionable data getting missed. When searching for threats within your organization, start with the network and only pivot to host logs when you think you’ve found something suspicious.

Pcap Paring

, ,

Intro

Filtering packets

Approaches

Full example

Additional notes

Resources

Compression tools

Intro

The first step in most network monitoring tasks is capturing network traffic from a network interface. In an ideal world where disk space and disk bandwidth are infinite, we’d save the files to disk as that allows us to run these packet files through multiple monitoring and forensics tools. Unfortunately, we run into the difficulty of saving  massive files. To save all the traffic from a fully saturated 1 gigabit/second ethernet link (1 gigabit/second in both directions) would require a constant stream of 250MB to disk every second and use up just under 22TB/day.

This commonly forces us to discard the files once they’ve been handed to all of the needed analyzers. There’s a middle ground, though, that allows you to keep some of the information without having to commit to massive disk purchases.

Instead of keeping the entire packet stream, let’s pare it down to just what we might need in the future. What might that include?

Filtering packets

What do we need to keep?

Let’s take a look at a few scenarios.

  1. We find an infected system on the network. The first question is how the attacker reached it. Could I go back to our packet logs and see what connections that machine has accepted just before the time of first infection?
  2. We get a subpoena for all information about a customer of ours. Can we use the information in the packet logs to identify any connections that customer made to our public servers?
  3. In response to a DoS attack, can we come up with statistics on what types of traffic are being thrown at us so we can know where to focus our mitigation efforts?
  4. A site on the Internet claims we’ve been attacking them. Could we use the packet logs to prove that we have not sent any traffic to them at all or learn that we did?

For the above scenarios, respectively, you’d need to keep:

  1. At least the SYN, SYN-ACK, FIN, and RST packets for all TCP connections.
  2. Like above, we’d need those same opening and closing packets to show that the client made connections at all; to show what happened on those connections, we’d need the Ack-only packets in the middle as well.
  3. Here we’d want as much of that traffic as we can handle. Since DDoS attacks tend to focus on a single type of traffic (though occasionally 2 or 3 types), we can capture just this/these ports as the attack starts and analyze the packets.
  4. Similar to 1, we’d want the opening and closing packets of all TCP conversations.
High level paring approach

You’ve probably already thought of a few I hadn’t considered. 🙂 That said, there’s a general approach to follow:
– Initial capture: this could be full packets or some subset that you need for initial analysis.
Immediately or soon after capture: compress pcap files to save space when practical.
– Early discard: once these pcap files have been processed by the tools you need, discard some portion of the data to save space but still accommodate the above scenarios.
– A few weeks/months in: Pare the files down again to the bare minimum needed to handle basic questions about network traffic.
– Even more months later: Delete files more than N months old.

What can we throw away after a while?

The right answer is this: If you look at a type of traffic and can say “There’s no chance I’ll need this more than N days after it was captured”, discard it after N days.

There are some general categories of things to discard:

  1. Chatty protocols like multicast DNS (UDP port 5353), netbios (UDP and TCP 137-139 and 445), NTP (123/UDP), and possibly DNS (UDP and TCP port 53). If you don’t feel these will be useful to you in the future, by all means discard them.
  2. In a TCP connection or UDP conversation, the bulk of the conversation is probably not of interest. In network analysis, you might want to hold on to the request (the early bytes going from client to server) and the headers of the response (the early bytes going from server to client), and discard the rest of the conversation. See Discard Acks after early ones if you want to keep these early bytes, and see Discard Acks if you don’t care about any of the payload. If you’re interested in the textual parts of the TCP connection but don’t want to save the middle portion, ngrep allows you to extract the pieces that look like text before you discard the Acks.

    In addition to the saved disk space, discarding the payload has the added benefit of privacy; in strict privacy settings, you may not be allowed to capture the conversation between client and server. By discarding that conversation immediately, you’re still left with the ability to identify that a conversation took place without conflicting with your organization’s policies.

Paring risk

One last warning before we get to actual paring. You’ll be making decisions on discarding traffic based on naive clues  like port number, using an assumption that – for example – every piece of traffic going over UDP port 53 is a benign DNS lookup or DNS response. Keep in mind that with tunneling, VPNs, encapsulation, steganography, and even the simple use of a different port, it’s possible to send almost any kind of traffic over almost any port. This makes that early analysis before you’ve discarded any traffic even more important; “hidden” traffic can no longer be detected once you’ve thrown away the packets in which it was hidden.

How to pare down

To do most of this we’ll use the “tcpdump” sniffer that comes standard on Linux, Mac OS, and other Unixes (windows note). We’ll use it in a mode where we tell it to read packets from pcap files, select only some of the packets, and write only those to an output file, like so:

nice -n 19 tcpdump -s 0 -r original.pcap -w filtered.pcap 'filter'

The ‘filter’ at the end is a Berkeley Packet Filter expression, or BPF for short. The string between the quotes states what packets should be kept; packets that do not match this filter will be discarded. We’ll use these filters below.

For other examples of BPF expressions, see the “EXAMPLES” section of the tcpdump man page, or “man pcap-filter” . pcap tools support this syntax, but note that wireshark and tshark use BPF only during capture but provide an (though richer) filter language for analysis.

Many of these commands have “nice -n 19” in front of them. This standard tool (included with Linux, Unix, and Mac OS, can be added to Windows) tells the computer to let other programs go first if there’s other work to do. This allows you to queue up lots of processing jobs without hurting any higher priority work (effectively everything except these nice’d processes.) On the other hand, if this system’s primary task is running jobs like these, remove “nice -n 19” in these commands.

Paring down compressed files

If the original file is compressed with bzip2 and you want to immediately recompress it after paring, use:

nice-n 19 bzcat original.pcap.bz2 | nice -n 19 tcpdump -s 0 -qtnp -r - -w - '{filter}' | 
nice -n 19 bzip2 -9 >filtered.pcap.bz2

The “bzcat” program decompresses the original compressed capture file, feeds that to tcpdump that reads from “-” (standard input) and writes the filtered packets to standard output “-w -” where the bzip2 program is ready to recompress it and save it back to a new filename.

Processing large numbers of pcaps

In the case where you’re dealing with tens of thousands of pcaps or more, it may not be possible to write a single to compress them like:

bzip2 -9 *.pcap

The list of pcaps may simply be too long to fit on a single command line. In that case, you can switch over to a loop:

ls -A1 | grep '\.pcap$' | while read onecap ; do bzip2 -9 "$onecap" ; done

This approach can be used for any command that’s expected to work with massive numbers of files.

Approaches

Compression

How: Use a tool such as zip, bzip2, or gzip to compress the file.

Pros: A compressed pcap file can take up <30% of the original file size. The file can be accessed without needing to uncompressing to disk first – see next paragraph. Ironically, accessing it may be faster as the CPU time needed to decompress may be less than the time needed to retrieve the uncompressed file from disk.

Cons: Compressing the file can take a lot of CPU time. To modify or read the file, it must be decompressed first.

Note that any task that would normally have wanted to read from a pcap file can probably be convinced to read directly from a compressed pcap file. In addition to taking less disk space, this uses less disk read bandwidth in exchange for a trivial amount of extra processing to decompress. As an example, replace:

tcpdump -r data1.pcap {other options}

with

bzcat data1.pcap.bz2 | tcpdump -r - {other options}

The following are some examples of ways to compress files. These examples are not intended to replace some well written articles giving comparisons of the time needed and final compressed size when using different compression tools. See https://www.google.com/search?q=compression+tool+benchmarks

Compression – fastest

How:

gzip -1 data1.pcap

Pros: This uses little CPU time to perform the compression.

Cons: The resulting compressed file could be 50% larger than one compressed with bzip2 -9 .

Compression – smallest

How:

bzip2 -9 data1.pcap

Pros: Better compression, so less disk space used.

Cons: Much more CPU time.

One approach that might make sense would be to use gzip -1 immediately after you’ve run the pcap file through your standard pcap tools, and then recompress it later with bzip2 -9 when the system is quieter or on another system with available processor time.

 Compression – using more than one processor

While most file compressors use a single processor core, “pbzip2” (a drop-in replacement for bzip2) can use as many processors as the system has. It’s “-p” option lets you set a maximum number of processors to use in case you want to limit this. When using N processors it’s almost N times as fast as using a single processor. The downsides are that the resulting file could be up to 0.2% larger than the same file compressed with bzip2 and the system may more RAM while compressing.

Paring

Now we’ll switch over to paring approaches that mostly use the tcpdump program to strip out unwanted parts of the pcap file. You’ll have to decide which – if any – of the following approaches are appropriate for your data retention requirements.

IPv4 only

This discards any IPv6 traffic. This might be useful on a network where systems support IPv6 but there’s no IPv6 connectivity to the outside world.

How:

tcpdump … 'ip'
IPv6 only

This discards any IPv4 traffic. This would allow you identify whether IPv6 is being used by any production servers and services.

How:

tcpdump … 'ip6'
Discard ACKs

More than 85% of the traffic flowing across the Internet is TCP traffic. A TCP connection begins with 3 early SYN, SYN/ACK and ACK packets with no payload, and ends with payload-less FIN or RST packets. All the packets in between those are ACK packets with their payloads holding the content going from client to server or server to client.

It might seem pointless to discard all those middle ACK packets – that’s everything said from client to server and all the server’s responses. The value here is that with the initial SYN and SYN/ACK packets and the FIN or RST packets at the end, we get the client and server IP addresses and ports, the start and end time, and the number of payload bytes transferred.

In short, this gives massive savings in disk space while still allowing you to know that a connection happened. The downside is that you can’t tell what data was sent or received.

How:

tcpdump … '(tcp[13] & 0x17 != 0x10) or not tcp'

There are two approaches that provide a middle ground between discarding all TCP payload and keeping it all. In the next section, we’ll look at keeping the earlier packets in the TCP conversation as that’s where much of the forensically interesting material is. That requires an extra tool, so if you’d prefer to stay with tcpdump, the “Shrink amount of payload stored” section keeps the beginnings of all packets. Think of the first approach as “Save the first 10 pages of this book and discard all later pages”, and the second as “Use a slicer to cut off the bottom 80% of every page and just save the the first few lines on each page.”

Discard ACKs after early ones

As we said in the “Discard Acks” section above, by discarding all acks we cannot see what request was placed or how the server responded (for example, the HTTP headers for “success” or “file not found”). By using a different tool, we can keep the early packets that commonly show request and response, but discard the ACKs that follow to save the majority of the space.

“pcap-modify” lets you keep the first C ACK packets in both directions of every connection seen in a pcap file with its “–ackcount C” parameter. It also allows you to ask to keep the first B bytes of every conversation with the “–ackbytes B” parameter. If you use both, packets that are either in the first C ACK packets or the first B bytes are kept. All remaining ACK packets are discarded. All SYN, SYN/ACK, FIN, and RST packets are kept. All non-TCP traffic is kept as well.
For reference, there’s a similar ability to keep early packets in UDP conversations with the “–udpcount Y” and “–udpbytes Z” parameters. These only affect UDP traffic.

How:

pcap-modify.py --ackcount 10 --ackbytes 4096 -r data1.pcap -w data1-earlypayload.pcap

Pros: Reduces disk use while still preserving the earlier packets that are likely to contain the protocol request and reply.

Cons: CPU and memory used.

Shrink amount of payload stored

Current versions of tcpdump capture the first 256K of a packet – in almost all circumstances that’s the entire packet as most network cables can’t even carry a packet that big.

One way to reduce the storage needs is to keep only the first N bytes of the payload. Add about 100 bytes to N for the headers at the beginning of the packet and you have what tcpdump calls the “snaplen”. Bytes that come after the first N+100 bytes are discarded.

While this does limit the storage needs from attempting to capture a DDoS attack with huge packets, it can make analysis difficult. By chopping off the ends of packets, it’s can be difficult to reconstruct the TCP payload. You might consider using pcap-modify instead.

How:

If you’re capturing packets from a network cable (as opposed to reading them out of a file), you can use the “-s snaplen” option with tcpdump:

tcpdump … -s snaplen

If you’re reading packets from a file, you’ll need to switch tools and use “editcap” (part of the Wireshark package that may be included with your operating system).

editcap … -s snaplen orig.pcap truncated.pcap

Pros: While this won’t save as much space as discarding all ack packets, it can give much of that benefit while still holding part of the conversation between client and server.

Cons: It is no longer possible to completely reconstruct the entire TCP conversation.

Discard specific protocols

BPF allows you to strip out entire classes of packets. The first 4 examples below are IP (specifically IPv4) protocols from /etc/protocols.

The last 3 examples show stripping out specific TCP or UDP ports (see “/etc/services”), letting you keep some kinds of TCP or UDP traffic while discarding less valuable or interesting conversations.

How:

tcpdump … 'not arp'
tcpdump … 'not egp'
tcpdump … 'not udp'
tcpdump … 'not tcp'
tcpdump … 'not tcp port 22'
tcpdump … 'not tcp port 80'
tcpdump … 'not udp port 5353'
“Known good” traffic – discarding specific protocols to specific hosts

One difficulty with “not tcp port 22” above is that we no longer keep any ssh traffic – good or bad. What we really want is to stop saving huge amounts of packets from known good ssh connections, such as the connections we use to transfer data daily from a business partner.

We can avoid saving these while still saving the packets to/from other hosts, allowing us to identify people scanning for open ports.

How:

tcpdump ... 'not (tcp port 22 and host download.partner.org) and not (tcp port 22 and host manager.example.org)
and not (tcp port 873 and host rsync.example.org)'

Pros: Assuming there’s a lot of data between us and these protocol/host pairs, we no longer save these to disk but still see attempts to use these ports from unauthorized users.

Cons: If one of these remote machines is hacked, the attacker might be able to come in over one of these protocols
without any logging at all.

Full example

Here is a sample script that manages a tree of pcap files, analyzing, paring and compressing as appropriate. It sets a simple lock so that it can be safely run out of cron without risking starting 2 copies at the same time. Script is at: http://www.stearns.org/pcap-modify/example-manage-pcaps

Additional notes

Paring during initial capture

Most sniffers and pcap analysis tools allow you to use the above filters not only during post-capture paring, but also when doing the capture in the first place. If you decided that you never wanted to grab the ACK packets in the first place, you could grab everything but ACK packets right off the wire with the following all on one line:

tcpdump -qtnp -s 0 -i eth0 -w /root/pcap/eth0.`hostname -s`.`date '+%Y%m%d%H%M%S'`.pcap '(tcp[13] & 0x17 !=
0x10) or not tcp'
Tradeoff

As with simple file compression, most of the above techniques trade additional 1) CPU time, 2) memory, and 3) disk bandwidth for a) reduced disk needs and b) the ability to perform at least basic forensics further into the past. Before moving forward with these approaches, take a survey of the system(s) that can access the pcap store; do they have enough available CPU, memory, and disk bandwidth to be able to do this paring?

tcpdump automatic rotation

tcpdump makes it straightforward to save packets to a file with the “-w” option we’ve been using above. It also recognizes that placing days or weeks worth of packets in a single file can produce a massive file that’s difficult to process, so it also offers the ability to “rotate” that file – stop saving to it once it crosses a certain size (“-C”) or after a certain number of seconds (“-G”). This simultaneously prevents files from getting too large or staying open for too long.

nice -n 19 sudo tcpdump -i en4 -qtap -C 50 -G 300 -w 'test.%Y%m%d%H%M%S.pcap' 'not port 22'

This rotates to a new output file when the old file would have crossed 50 million bytes (million, not mega) or has been open for 300 seconds. The “-w” option that specifies the output file lets you specify a time format to replace so each new file has its creation time embedded in the file name. The filename used above gives Year-Month-Day-Hour-Minute-Second. I got these on a test run:

...
test.20180705133513.pcap13
test.20180705133513.pcap14
test.20180705133513.pcap15
test.20180705134013.pcap
test.20180705134013.pcap1
test.20180705134013.pcap2
test.20180705134013.pcap3
test.20180705134013.pcap4
test.20180705134013.pcap5
test.20180705134513.pcap
test.20180705134513.pcap1
test.20180705134513.pcap2
...

This shows tcpdump changing the timestamp in the middle of the filename when it’s been capturing for more than 5 minutes, and appending a number to the end of the filename when it crosses 50 million bytes of data inside a single pcap.

tcpdump processing rotated files

tcpdump also makes it simple to process capture files when it switches to a new file. The “-z” option requires the name of a command to run when tcpdump rotates to a new output file; this command is given the name of the most recently closed capture file. For example, if you just wanted to do some light compression on the previous output file, you could add “-z gzip” to the tcpdump command line. While the new output file is being written, tcpdump will execute

gzip just_closed_output_file_name

in the background at low CPU priority.

Here’s an example:

sudo tcpdump -i en4 -qtap -C 50 -G 300 -w 'test.%Y%m%d%H%M%S.pcap' -z gzip 'not port 22'

If you’d like to do more complex processing of the previous save file, just make a shell script whose name you place after “-z” on the tcpdump command line. The file to process is the sole command line parameter.

ionice

On Linux systems, you can instruct the operating system to not only run these after other tasks have gotten all the CPU they need (the “nice -n 19” ‘s we used above), but you can also tell the operating system to make sure all other tasks have gotten all of the disk bandwidth they need first as well. This truly sets all of these paring tasks as ones that will not affect the other tasks on the system – doubly important when you may be capturing packets live.

To also use ionice, put

ionice -c 3

in front of any “nice -n 19” ‘s you see in above commands.

find … -mtime   in scripting

The “-mtime” option to the standard Linux/Unix “find” command lets you specify that you only want to look at files whose modification time is older than N days and/or younger than N days (you can also specify modification time in seconds, minutes, hours, or weeks by appending “s”, “m”, “h”, or “w” to the end of the number). Here’s an example of getting a directory listing of only files between 50 and 60 days old (files younger than 50 days old or older than 60 days old are ignored):

is -al `find pcap-modify/ -type f -size +0c -mtime +50 -mtime -60`

-rwxr-xr-x 1 wstearns wstearns 7278 May 8 18:25 pcap-modify/pcap-modify.py
-rwxr-xr-x 1 wstearns wstearns 7278 May 8 18:25 pcap-modify/pcap-modify.v0.4.py

For reference, the “-type f” requires that the object be a file, and “-size +0c” says the file size has to be
larger than 0 bytes.

Disk redundancy

For most types of data we want to have it stored on more than one physical disk, so if that disk dies we can still get the data from the second copy. You might consider placing the oldest pcap files on a storage area that only keeps a single copy of the data. That doubles your capacity, with the risk of losing those files if you lose one of the underlying drives.

Work off a policy

If your organization is large enough to need incident response capability, network monitoring, and storage for pcap files, it’s large enough to be governed by policies. Take the time to write up an addendum to your data retention policy stating for how long you’ll store full pcap captures, and at what time periods you’ll use the above techniques to pare them down and how. Get approvals, and then get paring!

Windows

tcpdump” does not come on windows systems, instead, use “windump”. When putting a filter on the command line, surround it with double quotes on windows instead of the single quotes used on Mac and Unixes:

tcpdump ... 'not port 22'

becomes:

 windump ... "not port 22"
Thanks

Many thanks to George Bakos, Chris Brenton, Melissa Bruno, Samuel Carroll, Logan Lembke, and Kris Chew for their time and help in putting this together. Also, thank you to John Strand and Active Countermeasures for sponsoring the time to write this.

Resources

IPv4 TCP/IP reference
https://www.sans.org/security-resources/tcpip.pdf

IPv6 TCP/IP reference
https://www.sans.org/security-resources/ipv6_tcpip_pocketguide.pdf

passer
http://www.stearns.org/passer/

pcap-modify
http://www.stearns.org/pcap-modify/

pbzip2
If not included with your operating system, see http://www.compression.ca/pbzip2/

Compression tools
https://en.wikipedia.org/wiki/Comparison_of_file_archivers
https://en.wikipedia.org/wiki/Lossless_compression#Lossless_compression_benchmarks

V1.1 Release

, , , , ,

Hey Folks!

We just performed another release this week and I’m psyched to fill you in on some of the new features! The two biggest are the addition of a module to analyze long communication sessions and a module that analyzes long URIs. Let’s jump into a bit about each.

Long Connection Analysis

As you may know, the beacons module analyzes network data in 24 hour chunks. This way, even when a system beacons home infrequently, we still collect enough data points to tag the system as a beacon. However, this begs the question, what if the malware does not beacon home? What if the attacker simply sets up a session back to the command and control server and leaves it active all of the time?  This is where long connection analysis comes in.

By selecting the “long connections” module, you will be presented with a list of the longest duration connections taking place on your network. The list is sorted from longest to shortest, but you can change this using the sort option in the top left of the screen. You can also set a threshold for the minimum connection duration time you want to be shown.

There are times when long connections are expected behavior. For example, BGP sessions between routers tend to persist as long as possible in order to update route information. There are even times when long connections are not optimal, but not necessarily evil. A good example would be an SSH user that leaves their connections active even when they are done on the system. For the most part, however, excessively long connection times (more than an hour) should be scrutinized carefully. If they are not evil, they may at least be in breach of policy.

Long URL Analysis

Our customers love AI-Hunter’s ability to identify when DNS is being used as a backdoor. Our new URL analysis module brings that same type of functionality to analyzing URIs. What may look like an internal system connecting to an external Web server may, in fact, be malware encoding information into the URI and passing that data out of your network. The url module is designed to allow you to quickly hone in on that type of activity.

Similar to the long connections module, the URL module displays the connections that took place with the greatest amount of URI data. Also, like the long connections module, you can change the sort option or the threshold on minimum URI length to report. Most users should be fine going with the defaults.

When URI exfiltration is taking place, it is not uncommon to see the data passed in the clear. However, using a simple XOR to obfuscate the data pattern, is a common technique as well. This means it can be helpful to extract the URI information for additional processing. If you mouse over the URI, a dialog box will appear showing the complete URI (this normally cannot be displayed due to its length). If you click on the report icon, you can download the data in comma separated format. This lets you perform additional processing in another tool.

A huge thank you goes out to our beta testers that helped us vet these new modules!

We have another new module in the works and wow, this one is a game changer. It really ties together the whole threat hunting experience! At ACM our goals are to simplify the threat hunting process so that:

  • More people can threat hunt their network
  • Those threat hunts take less time to complete

This new module will go a long way toward accomplishing both of those goals. Intrigued? Stay tuned to this blog/channel over the next few weeks for further announcements.

It’s not you, it’s your threat hunting tools

, , ,

Paul, John and myself had a very cool conversation regarding why threat hunting is so much harder than it has to be (spoiler: Our tools are not doing their job). I decided to write a blog entry about the points we were making so that I could incorporate some visual aids.

Consider the following threat hunting scale which describes your possible reaction to a session or set of log entries you are analyzing:

Think of this as a threat level rating where a “0” means “I am 100% convinced that this is perfectly safe” and “100” which means “I’m 100% convinced that this is absolutely evil”. Both of these extremes are great from a threat hunting standpoint because we know how to process the event (ignore it or initiate our incident handling runbook). It takes very little time to disposition and move on to the next event we need to analyze.

The problem is events that score somewhere in the middle. This is that nasty gray area where I am unable to perform a quick disposition. If I see an event and sore it as a 50, I then need to collect further data to move the score up or down the threat scale. This is where threat hunting becomes hard and time consuming. I need to move that needle closer to 0 or 100 so that I can move on to the next event that needs to be analyzed.

Where Most Tools Fall Short

Where most threat hunting tools fall short is that they are focused on collecting as much information as possible, not with helping you identify the relative threat scale of each piece of information being provided. If you think about it, this is actually counterproductive. If I increase from 10 to 1,000 data points but still don’t have context or threat rating, I’m actually in worse shape than when I started because there are a lot more events I need to sort through. I’ve increased the amount of noise around my signal which is never a good thing.

This is why threat hunting is such a tribal activity. It is heavily reliant on the skill set of the person performing the threat hunt. For example someone like John Strand, who has been doing threat hunting for many years, can probably quickly disposition threats between 0 – 40 and 60 – 100. That just leaves threats that fall between 40 – 60 that chew up his time. A junior analyst however, may have a gray area that runs closer to 10 – 90. This means they will need to deep dive on a lot more events.

Sorting Through The Noise

When we created AI-Hunter, we had one goal in mind, to bring threat hunting to the masses. By definition this means that we cannot simply throw lots of data at you and hope that some of it sticks. The tool needs to do the first pass at data analysis and add or subtract threat points to each of the systems being analyzed. This is the only way to reduce the time it takes to perform a successful threat hunt.

For example, AI-Hunter will sort through all of your communication sessions and assign a high score to the ones that most closely match a compromised system beaconing home. So rather than needing to review every communication session for clues of beaconing behavior, that work has already been done for you.  

All that’s left is to identify if this pattern is expected behavior. Is this an NTP session to a known NTP server? If so we can whitelist the target and move on to the next session for analysis. Is the target IP listed on one or more blacklists? If so we need to have someone investigate the internal system because we’ve probably been whacked.

What to Look For in a Threat Hunting Tool

Regardless of whether you are looking at AI-Hunter or some other threat hunting tool, there are three basic questions you need to ask yourself:

  1. Will the tool reduce the time we spend performing threat hunts, or simply give us more data that needs to be reviewed?
  2. Can a junior analyst be successful with the tool or will I be forced to surrender even more time from my senior staff?
  3. Can I write a runbook around using the tool so that different analysts will produce consistent results?

If you cannot respond positively to each of the above points, it’s time to look at other tools.

New Version in the Wild

, , ,

Hey folks!

We released another new version of AI-Hunter last week. It’s been so crazy busy, I have not had a chance to announce it until now. There are some features in this release that have me super excited!

We’ve had some cosmetic changes to clean up the interface. Those will be apparent when you login. What I want to dive into here are the features that will make threat hunting your network that much easier. Let’s start by talking about ASN whitelisting.

I’ve discussed AI-Hunter’s whitelisting feature in the past. Along with all the other ways you can identify systems as trustworthy, you can now whitelist based on the Autonomous System Number (ASN) associated with the source or destination address of a session. As a use case example, consider Microsoft’s network. You are probably willing to trust any beacon like activity that takes place between internal systems and Microsoft’s patching servers. That’s expected behavior. However, you may have an entirely different opinion of Microsoft Azure, as anyone can spin up servers in that space.

Microsoft has 23 different ASNs to route traffic to different portions of their network. By whitelisting ASN 8075, you’ll trust all of Microsoft’s patching systems, even when they change IP address, without trusting system running in Azure. This is much easier than trying to whitelist based on IP or subnet!

There is a new reporting mode that lets you easily copy/paste information out of the user interface without all of the HTML code. This is handy when you need to Slack the info to a teammate or open a Jira ticket to get a forensic analysis performed.

Finally, we’ve added a feature to display the hostname a system was trying to reach when it generated the session. This is not a simple lookup of the IP address to see what hostname is associated with it (PTR query). We actually look at the DNS A or AAAA record query generated by the host and pull the name from that session. Here’s an example:

In this example, the IP address is part of a cloud hosting provider. So looking up the IP address is simply going to show us whatever name the hosting provider assigned. However because AI-Hunter is capturing the name server queries, we get to see the actual FQDN our internal system was attempting to reach. This can go a long way towards helping us identify if we are looking at malicious activity.

We have another release in the works so please stay tuned!

Load More