Why You Can’t Monitor a 1 GB Connection With a 1 GB Span Port
I hadn’t expected my bank teller to tell me this: “I’m sorry, we do show $2,000 in deposits, but we only kept half of them so your account only has $1,000”.
What’s the Problem?
Capturing packets flying by on a network is our bread and butter; we use multiple programs to analyze those packets and report on different Threats, but the first step is always getting them off the network. And that’s where we have to admit we skipped a step. To explain what the problem is, you need to know a few basic facts about networks.
When we refer to a Gigabit Ethernet network interface or switch, each port involved has a pair of wires that transmit data and a pair of wires that receive data. The transmit pair can send up to 1 billion bits per second (aka 1Gbps * or 1 gigabit/second, which is equivalent to 1,000,000,000/8 = 125 megabytes/second or 125 MBps *). The receive pair can accept the same amount of data: 1 gigabit/second.
Let’s say I ask my switch to take all of the data moving into or out of port 1 (we’ll assume that’s the port leading to the Internet) and push a second copy of all of those packets out the transmit pair on port 9 (to which my sniffer is connected). Now my sniffer can see all the packets going to the Internet and all the packets coming from the Internet. But there’s a problem.
In the worst-case scenario, I could have as much as a full gigabit of traffic heading out to the Internet at the same time as I have a full gigabit of traffic coming back from the Internet – a total of 2 gigabits per second. The switch has been told to push a copy of all of those packets out the mirror port (port 9) to the waiting sniffer. But wait; port 9 can only transmit 1 gigabit per second! Like the bank teller spilling the beans above, that’s our dirty little secret; in the worst-case, we can only mirror and capture half the traffic! 🙁
If what we’re asking the switch to do is physically impossible, how can we capture traffic at all? There are a couple of reasons:
1) We’re not always using full bandwidth in both directions. In fact; it’s a far more common scenario that even if one of the directions is moving packets at full throttle (close to 1 Gbps in this example), the other direction is usually far less busy; we might only be seeing 10 or 12 megabits/second the other way. The combined bandwidth of both may be – and commonly is – less than the 1 gigabit/second limit on the mirror port.
2) The switch may have the ability to buffer a small amount of traffic. During a fraction of a second where there is more than a gigabit per second to push out the mirror port, it may be able to hold on to a few packets that it can send if there’s a brief gap in the traffic.
3) Packet capture tools assume they will not always be able to capture everything. They need to be able to deal with a scenario where some packets that actually did get to their original destinations weren’t captured.
So How Do We Fix It?
In the above example, both the port being watched (port 1 heading out to the Internet) and port 9 (the mirror port getting all of port 1’s traffic) were both Gigabit Ethernet ports. Nothing says they have to be the same speed.
Let’s change our example to say that port 9, the mirror port, is running at 2.5 Gbps (like our 1Gbps port above, it can simultaneously send 2.5Gbps and receive 2.5Gbps on its transmit and receive wire pairs). In our worst-case where there is simultaneously 1 Gbps going out of port 1 and 1 Gbps coming into port 1, the switch can send all of that out of port 9 (a total of 2Gbps) with bandwidth to spare. Obviously, the packet capture machine connected to port 9 needs a 2.5Gbps Ethernet port and needs to be able to process 2 Gbps. To keep things simple we’ll connect the packet capture machine directly to port 9 with a standard Category 7 Ethernet cable that can handle that speed.
Once we’ve upgraded the ports in question, we can keep up with a totally saturated Gigabit Ethernet port 1.
The Practical Side
The choice of “2.5Gbps” was not an arbitrary number greater than 2. There are multiple Ethernet standards for connections faster than 1Gbps, including 2.5Gbps, 5Gbps, 10Gbps, 40Gbps, and beyond. The 10Gbps standard has been around for a long time and is commonly used for network backbones at larger companies, but its high price makes it uncomfortably expensive for connecting all systems or use at home. Because of that, the 2.5Gbps and 5Gbps standards have stepped in to fill in the middle ground between “1Gbps is too slow for this network” and “10Gbps is too expensive”. And this is where you have multiple options for the gear needed.
For my sniffer, I’m going to go with the Sabrent USB-2.5GbE Interface. It will connect with a USB C or USB A interface (if using USB A, make sure you have a USB 3.0 or higher connector with a blue plastic center as they’re fast enough to carry this traffic).
For the switch, I’m going for the managed ZyXel XGS1210-12 switch that has 8x 1Gbps, 2x 2.5Gbps and 2x 10Gbps ethernet (SFP+) ports. You must choose the managed switch; the unmanaged alternative will not mirror packets. You’ll need to log in to the web interface on the switch to tell it to mirror all packets from the 1Gbps port 1 to the 2.5Gbps port 9.
I’ll also encourage you to use a Category 7 Ethernet cable on the 2.5Gbps port (though a Category 6 or 6a cable might work as well).
For a longer discussion and references to other hardware options, see the “Hot-rod” article in the References.
Once we’ve made these relatively minor changes, the 2.5GbE port has taken away the original bottleneck in the mirror port’s 1Gbps transmit pair. The only thing left to do now is file a report with my state’s banking board… 🙂
Many thanks to Jon Jacobi for his article “Hot-rod your home network with multi-gig wired ethernet—for far less coin than you might think”
While you’re obviously welcome to use any parts you like, the parts I mentioned above seemed like cost-effective ones for even a hobbyist environment:
* “gigabit per second” uses the acronym Gbps; note the lowercase “b”. “gigabyte per second” uses GBps with an uppercase “B”. We’ll mostly use gigabit per second (Gbps), but if you prefer to think in bytes divide that number by 8 to get Gigabytes per second or multiply by 125 to get Megabytes per second). “2.5 GbE” refers to the “2.5 Gigabit per second Ethernet” standard.
In all of these numbers we’re skipping over the difference between 1000 and 1024 in measuring bits and bytes per second; see https://en.wikipedia.org/wiki/Orders_of_magnitude_(data) for the gory details.
Interested in threat hunting tools? Check out AC-Hunter
Active Countermeasures is passionate about providing quality, educational content for the Infosec and Threat Hunting community. We appreciate your feedback so we can keep providing the type of content the community wants to see. Please feel free to Email Us with your ideas!
Bill has authored numerous articles and tools for client use. He also serves as a content author and faculty member at the SANS Institute, teaching the Linux System Administration, Perimeter Protection, Securing Linux and Unix, and Intrusion Detection tracks. Bill’s background is in network and operating system security; he was the chief architect of one commercial and two open source firewalls and is an active contributor to multiple projects in the Linux development effort. Bill’s articles and tools can be found in online journals and at http://github.com/activecm/ and http://www.stearns.org.