How Do Threat Hunting Tools Find Outbound Connections?
AC-Hunter and RITA, our two primary threat hunting tools, focus on traffic that’s exiting the network, looking for suspicious patterns that may indicate an infected system calling home for instructions. That leads to an important question: how do they tell whether a connection is exiting the network or not?
From a human perspective, that’s usually simple because we already have some context about where on the network we’re capturing packets. Unfortunately, the tools in this data pipeline (Zeek or some other capture tool, RITA, and AC-Hunter) don’t know where we’re capturing packets so they don’t have an instant way to tell if a connection goes from an internal system to an external system.
Networks That Use Reserved Addresses
The simplest way to tell is to realize that most networks use part of what are called “reserved” or “RFC1918” network addresses:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- fe80::/10
If you’re not familiar with that notation, just realize that they respectively include:
- the approximately 16.7 million addresses from 10.0.0.0 to 10.255.255.255
- the approximately 1 million addresses from 172.16.0.0 to 172.31.255.255
- the 65,536 addresses from 192.168.0.0 to 192.168.255.255
- the approximately 3.3×10^35 addresses from fe80:: to feff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
These addresses can be used on any private network. A router, proxy, or other network device turns these internal addresses into one or more external addresses so you can talk to computers on the Internet.
The nice part is that Zeek, RITA, and AC-Hunter all assume that you use these address ranges for your internal machines. When they see a connection from IP address “A” to IP address “B”, the test can go like:
- If A is a reserved address and B is not, then the connection is internal->external and we analyze it.
- If both are reserved addresses, then the connection is internal->internal and we ignore it.
- If A is not a reserved address and B is, then the connection is external->internal and we ignore it.
- If neither is a reserved address, then the connection is external->external and we ignore it.
If your network uses just these address ranges and nothing else, you don’t have to continue with this article; AC-Hunter and RITA already know about those address ranges and you don’t need to make any changes.
But There Are Exceptions to the Rule
Let’s get back to that phrase “most networks.” 🙂 Not all networks use this approach. Perhaps the network was set up in the days when IP addresses were plentiful and handed around like popcorn. “You need to set up 20 computers? No problem, here, have a block of 512 addresses in case you need to grow.” Networks like these will commonly use non-reserved addresses for their internal systems. Since your connections will all be from non-reserved IP addresses to non-reserved IP addresses, all the lines that Zeek provides will show traffic from A to B where:
- Neither is a reserved address, so the connection is external->external and we ignore it.
Oops. I hate it when that happens. 🙂
So what went wrong here? The problem is that we used a simple test to decide if an IP address was part of our internal network (“Is it a reserved/RFC1918 address? If so, it’s internal, otherwise it’s external.”) That test is wrong for networks that use non-reserved addresses.
Let’s say my company uses the address block 12.13.14.0/24 (the 256 addresses from 12.13.14.0 to 12.13.14.255) for our laptops. Instead of using the test “Is it a reserved/RFC1918 address?” I’d like to modify that test to be “Is it a reserved/RFC1918 or 12.13.14.0/24 address?” All three tools (Zeek, RITA and AC-Hunter) allow us to make this modification.
While this article shows examples of how to add a single subnet, you should add all the subnets your organization uses (one subnet per line).
Handling the Exceptions
Zeek
On your Zeek system, you’ll want to edit the file /opt/zeek/etc/networks.cfg like:
sudo nano /opt/zeek/etc/networks.cfg
The lines in that file contain a list of internal address ranges to which you’re able to add special ones for your company. Here’s an example of how the file looks:
# List of local networks in CIDR notation, optionally followed by a # descriptive tag. # For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes. 10.0.0.0/8 Private IP space 172.16.0.0/12 Private IP space 192.168.0.0/16 Private IP space
You can add your own to it, like so (the boldface is just to show the change):
# List of local networks in CIDR notation, optionally followed by a # descriptive tag. # For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes. 10.0.0.0/8 Private IP space 172.16.0.0/12 Private IP space 192.168.0.0/16 Private IP space 12.13.14.0/24 Our internal address space in the Fargo office
When you’ve added your address range, save the changes and restart Zeek with zeek restart.
AC-Hunter
AC-Hunter uses RITA in the background to handle importing new data, so it’s RITA that needs to know what your internal addresses are. For that reason, you’ll need to edit /etc/AC-Hunter/rita.yaml:
sudo nano /etc/AC-Hunter/rita.yaml
In that file, scroll down to a section labeled “InternalSubnets:” — here’s an example of how it starts:
InternalSubnets: - 10.0.0.0/8 # Private-Use Networks RFC 1918 - 172.16.0.0/12 # Private-Use Networks RFC 1918 - 192.168.0.0/16 # Private-Use Networks RFC 1918
When you get to editing this, be careful to keep the spacing exactly how it is in your file, and put the same number of spaces (not tabs) before your new entry. Web pages like this article may remove spaces entirely, so trust the formatting in your local file. Here’s an example of adding this new network:
InternalSubnets: - 10.0.0.0/8 # Private-Use Networks RFC 1918 - 172.16.0.0/12 # Private-Use Networks RFC 1918 - 192.168.0.0/16 # Private-Use Networks RFC 1918 - 12.13.14.0/24 # Our internal address space in the Fargo office
Like before, save and exit. To make this change take effect, you’ll need to run:
hunt down hunt up -d --force-recreate
RITA Standalone
If you’re using the RITA standalone program (as opposed to using AC-Hunter that happens to include RITA in the background; look to the previous section), you’ll need to edit /etc/rita/config.hjson on the RITA system:
sudo nano /etc/rita/config.hjson
The section to edit has “filtering: {” at the top. Inside that, scroll down to the “internal_subnets” line which ships like this (while the line may wrap in this article, it’s a single line in the file):
internal_subnets: ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "fd00::/8"], # Private-Use Networks RFC 1918 and ULA prefix
Edit that line and add your new subnet:
internal_subnets: ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "fd00::/8", "12.13.14.0/24"], # Private-Use Networks RFC 1918 and ULA prefix and Fargo
Proxies and Internal DNS servers
AC-Hunter
There’s one quirk we need to consider. If you have an internal DNS server or an internal proxy of some kind, Zeek, RITA, and AC-Hunter will see the traffic from your internal systems to that computer as internal->internal and will ignore it. To handle this, we need to edit /etc/AC-Hunter/rita.yaml on the AC-Hunter system.
sudo nano /etc/AC-Hunter/rita.yaml
Scroll down to the section marked AlwaysInclude. By default, it looks like this with a few sample addresses:
#AlwaysInclude: # - 192.168.1.2/32 # - 192.168.1.46
We’ll need to uncomment the header line and add the internal IP address(es) (the ones to which your client computers connect) of our proxy/proxies/internal DNS server(s) so it looks like:
AlwaysInclude: # - 192.168.1.2/32 # - 192.168.1.46 - 12.13.14.77/32 - 12.13.14.141/32
We don’t need to declare what those servers are, just that they’re DNS servers or proxies so AC-Hunter knows not to exclude traffic going to them. The same note above about keeping the spaces exactly the same and not using tabs applies here too.
Save your changes and exit. Now restart AC-hunter with:
hunt down hunt up -d --force-recreate
Zeek
There are no changes on the Zeek sensors for this.
RITA Standalone
For a system running RITA but not AC-Hunter you’ll need to edit /etc/rita/config.hjson once more.
sudo nano /etc/rita/config.hjson
In that same “filtering: {” section, go to the always_included_subnets line which ships empty like:
always_included_subnets: [], // array of CIDRs
Add your proxy/dns server internal IP addresses to this line like:
always_included_subnets: ["12.13.14.77/32", "12.13.14.141/32"], // dns server and web proxy in Fargo
Save and exit. This change will be applied to new databases the next time you import logs with RITA.
Network Scanners/Busy Systems
OK, we have one more category to consider. Does your network include any of the following?
- Network monitoring systems
- Scanners that make thousands of outbound connections
- Very heavy use DNS servers, mail servers, or proxies
If so, you may want to filter these high-bandwidth (or high numbers of connections/second) systems from AC-Hunter processing.
Wait a Minute, Didn’t We Just Force Watching Those?
In the last section we just added proxies and DNS servers to AlwaysInclude so that we’d always process these connections, and now I’m suggesting we stop doing that. What gives?
Proxies and DNS servers sit between client machines and the Internet like this:
[Client computers (a)] —-> [(b) DNS or proxy (c)] —-> [(d) Internet systems]
There are two connections in most cases; the client connects to the proxy with a connection from IP address (a) to IP address (b). We want to inspect this traffic because it shows which client(s) might be making lots of connections that end up at systems on the Internet (with IP addresses (d) ). Those connections are always turned on with the “AlwaysInclude” changes we made above.
The other connections – the second arrow from the DNS/proxy IP address (c) to systems on the Internet (d) – aren’t useful for threat hunting. If we watch these, we just see thousands or millions of connections a day that all look like they’re coming from IP address (c). That’s not useful for threat hunting so we want to stop looking at those.
Network monitoring systems and scanners are slightly simpler:
[Scanner/monitor (c)] —-> [(d) monitored systems]
We still have the (c) address that generates way too much traffic that we no longer want to process.
TL;DR: we want to use AlwaysInclude to watch the connections from client computers to the DNS server/proxy, and we want to use NeverInclude to stop watching the high volume outbound connections.
Stop Watching the Outbound Connections in AC-Hunter
To stop watching the external connections, we need to edit /etc/AC-Hunter/rita.yaml on the AC-Hunter system.
sudo nano /etc/AC-Hunter/rita.yaml
Scroll down to the section marked NeverInclude. By default, it looks like this with some addresses that should never be watched:
NeverInclude: - 0.0.0.0/32 # "This" Host RFC 1122, Section 3.2.1.3 - 127.0.0.0/8 # Loopback RFC 1122, Section 3.2.1.3 - 169.254.0.0/16 # Link Local RFC 3927 ...
We’ll need to add the external IP address(es) ( (c) in the above example) of our proxies, internal DNS server(s), and monitoring systems so it looks like:
NeverInclude: - 0.0.0.0/32 # "This" Host RFC 1122, Section 3.2.1.3 - 127.0.0.0/8 # Loopback RFC 1122, Section 3.2.1.3 - 169.254.0.0/16 # Link Local RFC 3927 - 17.18.19.5 # Fargo DNS 1 - 17.18.19.6 # Fargo DNS 2 - 17.18.19.20 # Fargo web proxy - 17.18.19.7 # Fargo network monitoring system ...
The same note above about keeping the spaces exactly the same and not using tabs applies here too.
Save your changes and exit. Now restart AC-hunter with:
hunt down hunt up -d --force-recreate
Stop Watching the Outbound Connections in RITA
For a system running RITA but not AC-Hunter you’ll need to edit /etc/rita/config.hjson yet a third time.
sudo nano /etc/rita/config.hjson
In that same “filtering: {” section, go to the never_included_subnets line which ships empty like:
never_included_subnets: [], // array of CIDRs
Add (c) addresses to this line, remembering that even if it wraps in this article it’s a single line in the file:
never_included_subnets: ["17.18.19.5/32", "17.18.19.6/32", "17.18.19.20/32", "17.18.19.7/32"], // array of CIDRs
Save and exit. This change will be applied to new databases the next time you import logs with RITA.
And in Case You Were Wondering…
…If a connection has both a NeverInclude and an AlwaysInclude address, AlwaysInclude wins and the traffic will be analyzed.
Thanks
Many thanks to Keith, Aaron, Faan, and Kassie for their help with reviewing and publishing this post.
Bill has authored numerous articles and tools for client use. He also serves as a content author and faculty member at the SANS Institute, teaching the Linux System Administration, Perimeter Protection, Securing Linux and Unix, and Intrusion Detection tracks. Bill’s background is in network and operating system security; he was the chief architect of one commercial and two open source firewalls and is an active contributor to multiple projects in the Linux development effort. Bill’s articles and tools can be found in online journals and at http://github.com/activecm/ and http://www.stearns.org.