Cloud Network and Virtual Machine Monitoring
We’re all aware of the advantages of hosting systems and services in the cloud; reduced hardware costs, speed of creating servers, and ease of scaling services. Unfortunately, by placing your servers on somebody else’s physical network, you lose the ability to plug a network monitoring system into a span port, mirror port, or tap to look for unwanted or malicious network traffic.
There are still some ways to monitor network traffic at Cloud providers and in Virtual machine environments, but they’re not universally available. Here we’ll look at the primary ways to make that happen and consider the limitations of each.
Approaches That Require Configuration on the Monitored System
With these, the monitored system is configured to copy traffic to the monitoring system. The difference between this and a hands-off approach below might seem academic, but does matter for security. Any approach that requires the monitored system to send off its traffic to a monitor can be disabled from within the system itself. That means that when a system has a break-in, the attacker could turn off the network monitoring from inside the attacked host.
iptables or TC on Linux
Stephen Reese has excellent coverage of 5 approaches using iptables or tc on a monitored Linux system at https://www.rsreese.com/network-traffic-capture-in-virtual-enviroments/ . He discusses the pros and cons of each.
While these techniques only work when the monitored system runs Linux and requires local changes, they do work with any environment; Cloud, local hardware, or both.
When routing this type of traffic between a monitored and a monitoring system, remember to set up a second network interface on both systems for routing this traffic that’s separate from the main interface leading to the Internet. Ideally these second interfaces get their own isolated network segment, both for performance and security reasons. You also don’t have to worry about getting into an infinite loop where each wrapped packet gets re-wrapped because of a configuration error!
Approaches That Capture Traffic with No Changes on the Monitored System
Amazon VPC Traffic Monitoring
Amazon’s AWS provides the ability to monitor one cloud server’s traffic on another cloud server. There are a few requirements to use this feature:
- The monitored systems must be in a VPC, and each must be a Nitro instance.
- The monitoring software needs to be able to extract the original packets from the VXLan wrapper before analysis (Zeek/Bro and Suricata both know how to do this).
- For each system to be monitored you must set up an individual Traffic Monitoring Session that specifies that you want to capture some/all traffic from an interface on the monitored system and send that traffic to the monitoring system. This approach does not support monitoring an entire VPC at once.
The network interfaces at both ends of this session can be either the network interfaces already in use on the monitored and monitoring cloud servers, or can be Elastic Network Interfaces (ENIs). The advantage of the latter is that one can move an ENI from one virtual machine to another while they’re running (allowing me to redirect the captured traffic from one sniffing cloud server to another sniffing cloud server to do a different type of analysis). I found it helpful to capture traffic on the existing network interface of the monitored system and see all the traffic it sends through its default route, and send that to an ENI that I attach to the monitoring system (giving me the flexibility to move the latter if needed).
We’ve written up the steps needed to make this happen at AWS VPC Traffic Mirroring. Going through the entire thing is somewhat time consuming, but you can short-circuit any of the steps you’ve already done:
- In your Amazon AWS account, create a VPC in which to place the systems
- Create a monitoring system and install Bro/Zeek 2.6 or higher
- Create a system to watch
- Create an ENI for the monitored system (optional)
- Create an ENI for the monitoring system (technically optional, but helpful)
- Create a Traffic Mirror Target in AWS
- Create a Traffic Mirror Filter in AWS
- Create a Traffic Mirror Session in AWS
- Confirm that you’re receiving VXLan traffic at the monitoring system
- Send the Bro/Zeek logs from the monitoring system to AI-Hunter
Routing One or More Cloud Servers Through a Router
Normally we think of Cloud servers as being standalone virtual machines that talk almost directly to the Internet. If your Cloud provider offers the ability to cluster a group of machines and route all their traffic through a gateway cloud server that you control, you can run your monitoring software on this machine directly and watch all traffic flowing through.
Strictly speaking this isn’t a span/mirror/tap approach; we’re not making copies of the packets and sending them somewhere else. Instead we’re simply watching those packets as they flow through a gateway/router to the Internet.
- No changes needed to the individual cloud servers
- Can monitor multiple machines at once (all the cloud servers behind the cloud gateway)
- Requires that your cloud provider offer this way of routing outbound traffic (and banning internal systems from reaching the Internet in any other way)
- Requires that you have access to install software on the cloud gateway
- Cloud gateway can be placed under heavy load, potentially impacting bandwidth or latency for the communication headed to/from the Internet.
Amazon AWS offers this ability, but as we have instructions on how to do this with VPC Traffic Monitoring above, we won’t cover how to do this here.
Virtual Network TAP on Azure
Microsoft’s Azure offers a Virtual network TAP as a preview. Traffic generated by this appears to be vxlan traffic sent to udp port 4789 on the collector. As they do support Linux cloud servers, this approach may allow us to analyze traffic there in a way similar to how we work on Amazon.
VMWare Cloud on AWS Port Mirroring
Nicolas Vibert covers how to use VMWare Cloud’s Port mirroring to monitor a single system or an entire subnet at https://cloud.vmware.com/community/2019/04/25/port-mirroring-vmware-cloud-aws/ .
Virtual Network Monitoring on VMWare
When we consider local virtualization we have multiple network monitoring techniques on VMWare. Ben Rossi covers 5 approaches in https://www.information-age.com/5-ways-copy-virtual-application-traffic-monitoring-123461226/# . Kent Ickler also covers how to run Bro in ESXi at https://www.blackhillsinfosec.com/monitor-network-traffic-virtualized-bro-2-51-ubuntu-16-04-2-esxi-6-5/ .
OpenStack Tap-as-a-Service (TaaS)
OpenStack offers Tap as a Service. Details at https://docs.openstack.org/openstack-helm/latest/install/plugins/deploy-tap-as-a-service-neutron-plugin.html and https://opendev.org/x/tap-as-a-service/src/branch/master/specs/mitaka/tap-as-a-service.rst .
As of 10/31/2019, I was not able to find any reference to configuring a virtual span port on DigitalOcean, Google Cloud Platform, Linode, or Rackspace. You would still be able to do network monitoring on Linux cloud servers on these and other cloud platforms using the iptables or TC approach above.
Bill has authored numerous articles and tools for client use. He also serves as a content author and faculty member at the SANS Institute, teaching the Linux System Administration, Perimeter Protection, Securing Linux and Unix, and Intrusion Detection tracks. Bill’s background is in network and operating system security; he was the chief architect of one commercial and two open source firewalls and is an active contributor to multiple projects in the Linux development effort. Bill’s articles and tools can be found in online journals and at http://github.com/activecm/ and http://www.stearns.org.