NAT and Packet Capture
The shortage of IPv4 addresses has forced us to use smart techniques to reallocate address blocks, speed up adoption of IPv6, and reduce the number of addresses we use. We’re going to focus on this last one as it affects how packets are captured.
Using Reserved Addresses
In our example network we’ll use 172.25.0.0/16 to provide 65,534 addresses to the systems in our building (from 172.25.0.1 through 172.25.255.254). All traffic between those internal systems will happen normally.
The 172.25.0.0/16 network is one of the reserved address blocks, also called RFC1918 addresses after the document that made them available, https://www.rfc-editor.org/info/rfc1918 . The good news is that these addresses can be used anywhere for internal communication. They can be – and are – used on millions of networks around the world.
This raises an interesting question; given a reserved IP address such as 172.25.3.19, to what system is that assigned and how would I get a packet to it? Unfortunately, there’s no answer to those questions since potentially millions of computers could all have that IP address – to which one should we send the traffic?
Since we have no way to direct traffic to one of these systems with a reserved address, we need to handle packet routing differently.
NAT
NAT stands for Network Address Translation. At its simplest level it’s the act of accepting a packet and modifying the source or destination address in that packet to some other address before sending it on its way. Let’s look at how it’s used.
Masquerading
Masquerading (also called “source nat” or “snat”) is the simplest of the Network Address Translation tools. There’s a single routable IP (IP version 4) address assigned to the outside interface of the firewall or router leading to the Internet. In this blog we’ll use 203.0.113.7 as that address. As we mentioned above, we are handing out 172.25.0.0/16 reserved addresses to all devices in our building.
Here’s an internal system trying to communicate with a machine out on the Internet:
UDP 172.25.2.94:18311 -> 8.8.4.4:53
This is a DNS request going to one of Google’s public DNS servers. Unfortunately we can’t send this request to the Internet because there’s no way for the Internet routers to get the replies back to us. So we’re going to modify the packet. Since we have a non-reserved address assigned to the outside of our router we’ll borrow that address and replace the source address with it:
UDP 203.0.113.7:18311 -> 8.8.4.4:53
Since the modified packet is now using non-reserved addresses we can send it on to the destination just fine and the response:
UDP 8.8.4.4:53 -> 203.0.113.7:18311
can successfully make it back to our router.
Handling Replies
There’s a quirk in this; what do we do with the reply when it gets back? We need some way to get it to the internal system that made the original request (172.25.2.94). To make that happen we need to replace the destination address (currently 203.0.113.7) with 172.25.2.94 and then send that modified reply back:
UDP 8.8.4.4:53 -> 172.25.2.94:18311
To make this possible the router has to remember what replacement it made so it can reverse the process on the reply. It will keep an internal table with one entry per conversation; here’s the entry for our DNS request:
protocol:UDP, source IP:172.25.2.94, source port:18311, destination IP:8.8.4.4, destination port:53: ==> replacement source IP: 203.0.113.7
Whenever a packet arrives back from the Internet that router needs to consult this table to see if it’s one of these modified conversations so it knows to replace its own destination IP address with the correct internal system IP address.
(If you’re familiar with masquerading you’ll notice that I oversimplified the above example – I’ll cover the complexities at the end.)
Packet Capture Placement
AC-Hunter and RITA use a packet capture tool called Zeek to read the raw packets and summarize the conversations going on. Zeek (like all capture tools) needs to find a spot on the network from which it can capture copies of packets.
On a network with masquerading where should Zeek listen?
Option 1: we capture packets just outside the NAT router on the line heading to the Internet. All of these packets will contain the single non-reserved address assigned to the NAT device:
UDP 203.0.113.7:18311 -> 8.8.4.4:53 UDP 203.0.113.7:6112 -> 8.8.8.8:53 TCP 203.0.113.7:3001 -> 174.143.211.58:80 UDP 8.8.4.4:53 -> 203.0.113.7:18311 UDP 8.8.8.8:53 -> 203.0.113.7:6112 TCP 174.143.211.58:80 -> 203.0.113.7:3001 ...
AC-Hunter and RITA want to report on traffic from your systems – which internal systems are making lots of connections in specific patterns. If we go with Option 1 we’re already running into a problem there – it looks like there’s a single IP address (203.0.113.7) that’s making tons of connections. That makes it really hard to look for regular connection patterns because it’s just a huge mash of connections from one IP.
Let’s say your particular capture tool doesn’t care about traffic patterns but instead you’re looking for malicious payloads or suspicious packet characteristics. Once you’ve found one or the other, how do you identify which internal system generated that packet? (In theory you could interrogate your router and ask it for the translation table it’s keeping, but this is an additional step and is generally not possible if you’re investigating traffic that’s more than an hour old.)
Option 2: we capture packets just inside the NAT router on the switch leading to it. Zeek will see packets with the original reserved addresses like these:
UDP 172.25.2.94:18311 -> 8.8.4.4:53 UDP 172.25.111.222:6112 -> 8.8.8.8:53 TCP 172.25.6.6:3001 -> 174.143.211.58:80 UDP 8.8.4.4:53 -> 172.25.2.94:18311 UDP 8.8.8.8:53 -> 172.25.111.222:6112 TCP 174.143.211.58:80 -> 172.25.6.6:3001 ...
By looking at the packets inside the NAT router we get to see the IP addresses of the internal systems that are actually generating the conversations and receiving the replies. That makes it easier to do both threat hunting and track down a system that’s sending something suspicious. This is why we strongly urge you to do your packet capture inside (before) any NATdevice.
NAT Complications
My example above – where the NAT router simply replaced the internal system IP with its own – is how masquerading does work but it ignores some complications that show up when you actually try to implement this in software. If you read this blog just to learn about where to capture packets you’re welcome to skip this section.
1. Source port conflicts
If two internal systems happen to pick the same source port when talking to the same service on the Internet we need some way to keep those two conversations distinct:
UDP 172.25.2.94:18311 -> 8.8.4.4:53 UDP 172.25.7.12:18311 -> 8.8.4.4:53
When this happens we also need to substitute the source port with a replacement and store both the source IP and source port in the table for these connections:
UDP 203.0.113.7:18311 -> 8.8.4.4:53
UDP 203.0.113.7:31952 -> 8.8.4.4:53
Like the simplified example above, both the source ip address and port need to be replaced when the response packets come back.
2. Table size and scan time
The table that remembers these connections can grow to be massive on a busy NAT router. When that table fills all the available memory (quite possible on routers with small amounts of ram) the router may need to discard some of them, leading to broken connections. Even when the table stays small enough it can still take a substantial amount of time to search through this table to find the right connection, a search that has to happen on every packet.
3. Timeouts
We need a way to remove entries from that table to keep it from growing too big. Routers commonly set timeouts – the amount of time to wait for another packet to show up before considering the connection abandoned and remove it from the table.
The default values for these timeouts tend to work reasonably well but there are situations where these timeouts can keep connections around too long (leading to heavy memory and CPU usage) or not long enough (leading to broken connections).
4. FTP and others that stuff the port inside the payload
I’ll just mention that there are a small number of protocols that decided to place the source port inside the payload of the connection as well. When this happens and that source port is modified, it can break that protocol and make it impossible to use it through nat. Luckily some firewalls recognize that this happens and modify the payload too to allow that protocol to continue working.
The most common of these is FTP. This is not generally a problem as FTP is itself becoming less commonly used.
Bill has authored numerous articles and tools for client use. He also serves as a content author and faculty member at the SANS Institute, teaching the Linux System Administration, Perimeter Protection, Securing Linux and Unix, and Intrusion Detection tracks. Bill’s background is in network and operating system security; he was the chief architect of one commercial and two open source firewalls and is an active contributor to multiple projects in the Linux development effort. Bill’s articles and tools can be found in online journals and at http://github.com/activecm/ and http://www.stearns.org.