Where Do I Put My Zeek Sensor?
While AI-Hunter and RITA Threat Hunting tools can be placed almost anywhere you’d like – any available data center, DR site, or cloud – the Zeek sensor that feeds them network connection information needs to have a vantage point where it can see all the traffic going to and coming from the Internet. Let’s consider where the sensor should go.
Threat Hunting focuses very heavily on traffic going to or coming from the Internet as it’s looking for signs of communication with command and control servers there. This is called North-South traffic as the Internet is commonly placed at the top of network diagrams and your servers and client machines are at the bottom. (Traffic between your systems is called East-West traffic, and there’s very little of this involved in the initial detection of a potentially infected system).
Right off the bat, this gives us a way to reduce the amount of traffic we have to sort through. If we’re lucky enough to have a switch that handles all of our Internet traffic but has very little internal traffic, that’s a good candidate for where to place our Zeek sensor.
If there’s no choke point like that, we may have to place the Zeek sensor further back in the network, potentially on a switch that has a lot of internal traffic as well. In that case, we’ll want to tell Zeek to ignore all internal traffic at the kernel level.
To see how this works in practice, let’s assume that our internal systems use the 172.25.0.0/16 network range to support ~65,000 systems. We want to filter all traffic that has a source IP address in that range and a destination IP address in that range, with one exception. We want Zeek to see all DNS traffic, even if it’s from an internal system to an internal DNS server, so we’ll use the following filter:
port 53 or not (src net 172.25 and dst net 172.25)
The way to interpret that is “Kernel, hand up all packets that are either DNS packets or have some address outside of our network”. That leaves Zeek just the traffic we care about and will likely let it run faster.
To actually implement this, you’ll need to find zeekctl.cfg on your Zeek system and add this line to it:
zeekargs=-f "port 53 or not (src net 172.25 and dst net 172.25)"
Once saved, run:
sudo zeekctl deploy
to start using this filter.
We cover filtering in more detail in this blog here.
Zeek needs to have a full copy of all the traffic passing through a network. To get this we need a mirror port. Depending on the product you buy, this may be called a mirror port, span port, copy port, or network tap, but they all have the same goal of carrying network traffic as before while sending a second copy of all packets to this specialized port.
When you buy a switch, look in the list of features or the manufacturer’s web site for one of these terms. Look for “managed” or “unmanaged” for switches. An unmanaged switch will not have a mirror port – there’s no management interface to enable or disable the feature. You want a managed switch, but even then you want to check that it can do packet mirroring; not all managed switches do.
When you connect to the management interface to turn on packet mirroring you’ll be asked for both which port to monitor (source) and to which port to send the copied packets (mirror). For the source port, we recommend using the port that goes up to the Internet – that lets you focus on Internet traffic. If you’re setting this switch up for the first time, you might consider using port 1 for this. For the mirror port, you might consider using the highest numbered port.
What happens if you connect your Zeek sensor to a switch port that is not set up as a mirror port? You’ll only see three types of traffic: outbound connections created by this system, incoming connections headed to this system, and broadcast traffic on the network cable. As you Threat Hunt you’ll see a lot of traffic involving this system’s IP address, but very little for any other IP addresses. If this happens, double-check your switch to make sure the mirror feature is enabled and that the sniffing port on your Zeek sensor is connected to the mirror port.
When you set up the mirror port, you may also be asked if you want copies of just the packets going out of port 1, just the packets coming into port 1, or both. You’ll want to choose “both”.
In an example gigabit Ethernet switch, that could be as much as a full gigabit/second going out port 1, a full gigabit/second coming into port 1, for a total of 2 gigabits/second that we in theory want to push out the mirror port’s transmit pair – which is limited to one gigabit/second.
We talk about this more in this blog here.
In short, be aware of the total amount of traffic to monitor and be ready to get a different switch if you have more traffic to monitor than your mirror port can transmit.
Virtual Machine vs Physical Computer
I’m a big fan of virtualization – I love the idea that a single system can have isolation between processes while still sharing ram, cpu, disk, and network. Unfortunately, virtual machines are not always a good fit for packet capture for a couple of reasons.
- Setting up the networking in a way that allows you to capture packets on the host can be tricky or impossible.
- Virtual machines can add overhead to processing every packet, as the packet needs to transition from the host kernel to the sniffer application running inside the VM.
- Virtual machines are not always running – they share processor time with the other virtual machines on the system, and that means that a large number of packets to be processed could show up when the sniffer VM is not running. Ideally, the virtual machine platform knows this is possible and can buffer enough traffic to mitigate this, but this is not a guarantee.
In short, packet capture is best done on a physical computer instead of a virtual machine. If you still feel a VM is best for you, take the time to monitor how much packet loss you experience.
In most networks we’re using 10.x.y.x, 192.168.y.z, or 172.16.y.z-172.31.y.z addresses for our internal systems. There’s a device, commonly a firewall or router, that replaces these internal addresses with one public IP address as traffic heads out to the Internet. This device is performing NAT – Network Address Translation (It’s possible there are 2 or more public addresses, but that’s not material to sensor placement).
When you’re placing a Zeek sensor on your network you may have the option of placing this above this NAT device (where it sees all traffic with the one legal IP address) or below this NAT device (where it sees the same traffic, but with the large number of internal IP addresses). We strongly recommend placing the sensor below the NAT device so you can associate the traffic with the internal system that generated it. If you place the sensor above the NAT device, all you’ll see is a single outside IP address that’s talking with the Internet, with little or no ability to figure out what Internal system actually generated the traffic.
It’s common to use a proxy of some kind on your network to stand in the middle of network connections. Examples of these include an internal DNS server, an HTTP/HTTPS proxy, or a mail server. The two primary advantages of using these is that you 1) get to inspect the individual connections and possibly block some of them based on policy, and 2) can buffer up outgoing traffic and cache incoming results to improve speed and reduce latency.
When you use a proxy the traffic switches from:
InternalSystem1 ————————————————–> InternetHost1
InternalSystem1 —->ProxyIntIP ProxyExtIP ———->InternetHost1
The dashed lines could include any combination of routers, switches, and ethernet cables.
Where do we place a Zeek sensor in this case? If we place it on a switch on the left-hand side, we’ll see a bunch of internal systems all talking to the Proxy’s Internal IP address, but won’t see where that traffic will end up going (as that’s not included in the headers). If we watch the outside, we’ll see the Proxy’s outside IP talking to a lot of Internet hosts, but won’t see what internal system generated the request.
While you’re Threat Hunting, the first task is to decide if the traffic is malicious or benign. If the only thing you can see is the InternalSystem->ProxyIntIP traffic, it’s very hard to make that decision. On the other hand, if you can see the ProxyExtIP->InternetHosts traffic, at least you have some knowledge of where the traffic is going and that’s more likely to help you make that choice. When you do identify some malicious traffic you can refer back to the proxy logs to identify the Internal system that created the requests.
If you’ll be using a single sensor for Threat Hunting we recommend placing it above the proxy for this reason. If you have the flexibility to place 2 or more sensors, you could absolutely place one above the proxy and one below the proxy. You’d do your primary threat hunting with the sensor above the proxy, and refer back to the sensor below the proxy for reference or correlation.
If you decide to whitelist some of these connections, remember to whitelist either the InternetHost (as a destination) or the IP Pair (ProxyExtIP and InternetHost). Never whitelist just the ProxyExtIP; this will blind you to all other traffic flowing through the proxy!
Heavy Network Load
For every packet and connection that comes by, Zeek has to allocate both processing time and memory. As the number of packets per second and connections per second go up, the processing time and memory increases. Sooner or later you’ll reach a point where Zeek can no longer keep up with the load and it starts to miss packets and incorrectly classify some connections.
You certainly have the option of running Zeek on a system with more memory, more processor cores, and/or faster processor cores. We go over the process of diagnosing this and applying the right fix at https://www.activecountermeasures.com/why-is-my-program-running-slowly/ .
We cover additional approaches to speeding up packet capture in a 3 part blog series starting at https://www.activecountermeasures.com/improving-packet-capture-performance-1-of-3/ .
You also have the option of considering a commercial Zeek sensor such as Corelight’s appliances: https://www.corelight.com/products/appliance-sensors/ .
Interested in threat hunting tools? Check out AC-Hunter
Active Countermeasures is passionate about providing quality, educational content for the Infosec and Threat Hunting community. We appreciate your feedback so we can keep providing the type of content the community wants to see. Please feel free to Email Us with your ideas!
Bill has authored numerous articles and tools for client use. He also serves as a content author and faculty member at the SANS Institute, teaching the Linux System Administration, Perimeter Protection, Securing Linux and Unix, and Intrusion Detection tracks. Bill’s background is in network and operating system security; he was the chief architect of one commercial and two open source firewalls and is an active contributor to multiple projects in the Linux development effort. Bill’s articles and tools can be found in online journals and at http://github.com/activecm/ and http://www.stearns.org.