Problems With Packet Capture

 

OK, I need to come clean with you. We talk a lot about capturing packets in our blogs and webcasts because it’s such a core piece of network security. The blogs and webcasts imply that this is simple, but packet capture and analysis can be annoyingly complex and has a bunch of things that can go wrong.

Here are some of the more common problems with packet capture, as well as some approaches to resolve them.

 

Can’t Tell Which Port to Listen On

When you look at the ethernet port names inside your operating system, these rarely match up with the names on the case next to each port. Here’s how to identify the internal port name to use for capturing packets:

https://www.activecountermeasures.com/on-which-interface-should-i-capture-packets/

 

Unable to Listen to the Ethernet Port at All

The packets flying by on the wire can carry very sensitive information. Even if current protocols don’t generally send usernames and passwords in plain text, there’s a lot you can learn about the local systems, remote systems, and traffic patterns by watching these packets.

For that reason, most operating systems restrict packet capture to the root/Admin user by default. To allow users other than root to capture packets, use one of the techniques in this blog post:

https://www.activecountermeasures.com/permission-to-capture-packets/

 

Having Trouble Capturing Packets Inside a Virtual Machine

This is a common problem.

For all their wonderful features, packet capture inside a virtual machine has a bunch of issues that make it a poor choice. We cover that in more detail at:

https://www.activecountermeasures.com/is-it-ok-to-capture-packets-in-a-virtual-machine/

 

No Packets Being Captured at All

If you start up a capture tool, you may get no output at all. First, confirm that you are not looking up hostnames. If you aren’t looking them up but you’re still not getting any output, it could be that you’re not listening on the right interface. See this blog to make sure you have the right capture interface:

https://www.activecountermeasures.com/on-which-interface-should-i-capture-packets/

 

Captured Packets Have Bad Checksums

Packets commonly have 1 or 2 checksums inserted in the IP and protocol headers to allow the receiving system to tell whether the packet has been modified in transit. In the case of packets created by this system (that are being sniffed on the way out), it’s possible that the capture happens before the checksums are added (the same may happen for packets arriving at this system).

The packet capture tool may see these packets as corrupted since the (missing) checksums don’t match the other parts of the packet. If you get this error, the simplest way to avoid it is to tell your capture tool to ignore the checksums. For Zeek, you can add “-C” to the command line to ignore the checksums. With tcpdump, use “-K” or “–dont-verify-checksums”.

 

Very Few Packets Being Captured

Another possibility is that your capture tool does see some packets, but nowhere near as many as you expect. These consist of broadcast packets and packets created by or destined for this system (but you don’t see the traffic for the other machines on the network.)

In this case, it’s likely that your capture port is connected to a port on your switch that isn’t a mirror, copy, span, or tap port. Trace the cable from your capture port back to the switch and note which switch port to which the cable is connected. Now go back to the management interface for your switch and confirm that it should be acting as a mirror, copy, or span port.

It’s also possible you’re capturing on the wrong interface, so I’ll link to this blog once more:

https://www.activecountermeasures.com/on-which-interface-should-i-capture-packets/

 

Some Systems/Connections Are Drowning Out the Traffic You Want to Analyze

Every time you go to capture network traffic you’re being handed 3 categories of packets: 1) the packets you definitely want to analyze, 2) the packets you’re not sure if you want to analyze, and 3) the packets you definitely don’t care about. It’s pretty common that the third category contains huge numbers of packets and they tend to drown out categories 1 and 2, not to mention placing a very heavy load on your processors, memory, and disk.

If you can identify some of these high volume streams, you can instruct your packet capture tool to ignore them. For example, if you’re looking for unencrypted traffic to see if any plaintext passwords are still being sent over your network, you could start with a filter that ignores the commonly used ports for SSH and HTTPS (which will generally be encrypted):

tcpdump -i eth0 -qtnp 'not tcp port 22 and not tcp port 443'

This type of filtering (called “BPF” or “Berkeley Packet Filter”) allows us to discard unwanted packets (like the above example) as well as other types of filtering. For more details, see the following blogs:

https://www.activecountermeasures.com/filtering-out-high-volume-traffic/

https://www.activecountermeasures.com/?s=BPF

https://www.activecountermeasures.com/packet-loss-or-why-is-my-sniffer-dropping-packets/

https://www.activecountermeasures.com/improving-packet-capture-performance-1-of-3/

https://www.activecountermeasures.com/improving-packet-capture-performance-2-of-3/

https://www.activecountermeasures.com/improving-packet-capture-performance-3-of-3/

https://www.activecountermeasures.com/why-you-cant-monitor-a-1-gb-connection-with-a-1-gb-span-port/

 

System/Disk Can’t Keep Up/High Packet Loss

Packet capture tools need to be able to keep up with fantastically high packet rates. They have to read a packet from the network interface, identify some basic features of the packet, (usually) do some analysis or update packet status, and possibly save some or all of the packets to disk. When watching a heavily used network, this can strain the processor, memory, and disk subsystems to the point that the capture system simply can’t keep up.

Here are some common performance problems and potential solutions.

 

Don’t Ever Look Up Hostnames During Capture!

As a convenience, many packet capture tools will read the two IP addresses in a packet, send out a request to turn them both into hostnames, and then show the hostnames instead of the IP addresses in the program output.

Unfortunately, this is really slow. During the time needed to turn just one IP address into a hostname, thousands or tens of thousands of packets may have flown by on the network cable and are never captured because the capture program is waiting to get the hostname reply.

In short, don’t ever look up hostnames while capturing the packets!

The command line option to disable this differs between programs. One common option to disable this is “-n”.

(As a side note, some capture programs will also offer to convert port numbers into port names. This task doesn’t take anywhere near as long as converting IP addresses to host names, but it does take some processing time with each packet, so it, too, is discouraged.)

 

Capture With a Program That Does Little Processing

By the time you get to wanting to capture packets, you almost certainly have a tool in mind that will perform the analysis you want. If you find that it can’t keep up, consider a slightly different approach.

Instead of running your preferred tool (let’s say it’s Wireshark), consider capturing the packets with tcpdump, saving the packets to a pcap file, then instructing Wireshark to read the packets from that file instead. That means that Wireshark doesn’t have to keep up with the incoming flood of packets at their original rate; it can read them from the pcap file at its own speed.

Tcpdump is just one example of a tool that can read packets from an interface and save them to disk with little overhead.

 

Can’t Run Preferred Tool on a Particular System

This same approach can be used to address another problem. Let’s say I can’t run my preferred analysis tool on a given computer. It might have too little memory, too slow a processor, a different processor architecture, or an underlying operating system or libraries that won’t run the program.

The above approach works well here too. I capture the raw packets on that computer using tcpdump or another program that will run on it, telling it to save the packets to disk. I can transfer that pcap file over to a system that will run my analysis tool and tell that tool to read packets from the transferred pcap file.

 

Reduce Amount of Processing per Packet

Imagine that your packet capture tool spends 1/1000th of a second to do all of its processing and statistics-gathering for each packet. Sounds fast, doesn’t it? Well, that tool has just limited itself to processing no more than 1000 packets per second. ☹️

If you want to work with more packets per second, you either need faster processors or you need to reduce the amount of work done at capture time. Think about what your tool does with each packet; basic statistics and saving it to disk are probably fine. Does that tool need to look up a hostname? Does it need to analyze the payload of each packet? Does it need to keep track of whether this packet is part of an existing connection or not? Does it need to extract the content of every file transferred across this network? Does the packet need to be displayed to a screen?

Consider these and all the other tasks your capture tool is doing at capture time. Do you need feature N at all? Even if you do, do you need feature N at capture time or could it be done later, long after the capture is complete?

Here’s where you’ll need to consult the documentation for your capture tool to see how to disable the features you don’t need. Sometimes this will be done in a configuration file, sometimes it will be a command line parameter, and sometimes you’ll be able to pick either.

 

Investigate System Bottlenecks

While many of the issues I’ve discussed above are focused on packet capture, it’s also worth considering the performance of the capture and analysis system. This blog covers the different forms of bottlenecks in a computer, as well as discusses some of the potential approaches to fix them:

https://www.activecountermeasures.com/why-is-my-program-running-slowly/

 

Reduce the Number of Packets Being Saved to Disk

If the above article points to your disk as the bottleneck on the capture system, this may be time to consider reducing the amount of data being written. Start with this blog to identify the traffic with the most volume, then stop analyzing that by using a BPF:

https://www.activecountermeasures.com/filtering-out-high-volume-traffic/

You could also take the approach of saving the packets to a RAID array.

Compressing the files will save disk space but not disk bandwidth, since the raw pcaps have to be written to disk once, then re-read at the end of the hour, compressed, and saved a second time in compressed format.

 

Hardware Offload

The Linux kernel knows when a network interface chip has the ability to perform some tasks that would otherwise be done by the main processor. For example, it can tell the chip that manages eth0 that the chip should do the work of calculating the checksums that are embedded in packets (so the packet recipient can tell if they were modified in transit.) These special capabilities (called “NIC offload”) reduce the load on the main processors when the system is under heavy packet load.

There’s a catch, unfortunately. Some of these offload features interfere with packet capture. When you’re doing capture on an interface with these enabled, you should turn these features off for that interface. (You’re still welcome to use these on the non-capture interfaces.)

The best summary of this problem we’ve seen is this article:

https://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html

 

Saved PCAP Files Grow Without End

Unless told otherwise, capture programs that save packets to disk will do that forever. This has a tendency to slow down other tools that need to read from or write to the disk, as well as fill it. 🙂

There are two primary ways to handle this: either kill the program from time to time and restart the capture, or — if the program supports it — tell the program that it should rotate the logs every N seconds. Killing the program is messy because it doesn’t always have a chance to write out the last packets captured and you get error messages each time you try to read that pcap file. Instead, let’s see how we can tell tcpdump to automatically rotate files.

The “-G” command line parameter needs a number after it, the number of seconds to use for each capture file. To start a new capture file 1 hour after starting (and every hour after that), use “-G 3600”. Note that this isn’t “start a new file at the top of every hour” – it’s “start a new file every 3600 seconds”, so if you start tcpdump at 12 minutes past the hour you’ll start a new file at 12 minutes past every hour from then on.

You also want to tell tcpdump how you want the files named. The “-w” command line option that usually gives a single filename to which to write the packets accepts a different format that’s useful when you rotate files like this:

tcpdump -i en3 -G 3600 -w '/packets/'`hostname -s`'.%Y%m%d%H%M%S.pcap'

Each time you switch to a new file, the codes starting the “%” will be replaced; “%Y” is the year at that moment, “%m” is the month at that moment, etc. (Run “man 3 strftime” to see the different available codes.)

If you also put “-z bzip2” on the command line, each file will be compressed by the bzip2 command in the background while the new packets are being saved in the next pcap file.

 

Not Enough Disk Space to Hold the PCAP Files

If your company has a data retention policy, see if it specifies how many days worth of packet captures need to be kept before they’re deleted. If so, are you finding that there simply isn’t enough disk space?

The simplest way to try to address this is to compress the pcap files in the background during capture; tcpdump will do that with the “-z bzip2” command line option. Unfortunately, pcap files don’t compress all that well.

To look at some more aggressive approaches to paring down these files, see:

https://www.activecountermeasures.com/blog-pcap-paring/