AWS VPC Traffic Mirroring

November 18, 2019 General, Network Tools, Technology, Threat Hunting

Intro

This document walks you through using AI-Hunter to analyze traffic coming out of an AWS VPC for Beacons and other potentially malicious traffic signatures.

Amazon has provided the Cloud equivalent of a network span port or tap, called Traffic Monitoring. By setting this up you have the ability to run a sniffer on one cloud server that looks for interesting traffic going into or coming out of another cloud server.

Requirements

The cloud instances you wish to monitor inside the VPC must be EC2 Nitro instances (more details). The following list of Nitro instance types was copied from that page on Oct 29, 2019:

  • A1, C5, C5d, C5n, G4, I3en, M5, M5a, M5ad, M5d, M5dn, M5n, p3dn.24xlarge, R5, R5a, R5ad, R5d, R5dn, R5n, T3, T3a, and z1d
  • Bare metal: a1.metal, c5.metal, c5n.metal, i3.metal, i3en.metal, m5.metal, m5d.metal, r5.metal, r5d.metal, u-6tb1.metal, u-9tb1.metal, u-12tb1.metal, u-18tb1.metal, u-24tb1.metal, and z1d.metal

Your analysis system needs to run Bro or Zeek 2.6 or higher. 2.6 was the first version that analyzes the VXLAN traffic used in Amazon’s Traffic Monitoring.

You can use another sniffer program, but that tool needs the ability to accept incoming VXLAN traffic on UDP port 4789 and unwrap the outer VXLAN header to expose the original packet within before doing its analysis. If your tool doesn’t have this ability, all it will report are a lot of udp port 4789 sessions between the monitored and monitoring systems.

 

Implementation

The following are step-by-step instructions to create a VPC, set up both a monitoring station and a simple traffic generation system, and connect the two with Amazon’s traffic monitoring approach.

All of the above are starting suggestions for how this can be set up. Feel free to adjust them to fit your needs and skip any that are already done.

Setup Process

  1. Create a VPC if you don’t already have one
    a) Create one in Amazon’s console . Pick a region close to you before creating one.
    b) “Create VPC”.
    c) Choose IPv4 address block to use (such as “192.168.201.0/24”), and select “Amazon provided IPv6 CIDR block”.
    d) Create and attach an Internet Gateway.
    e) Add routing table entries for 0.0.0.0/0 and ::/0 through your Internet Gateway (https://console.aws.amazon.com/vpc/ , select “Route Tables”)
    f) Make sure your Security Group allows inbound ssh at a minimum.
  2. Create the Zeek packet analysis instance in Amazon’s EC2 configuration .
    a) For simplicity we recommend starting the Zeek instance in the same VPC as the instance(s) you wish to monitor (see Amazon’s documentation for details on how to set up cross-VPC monitoring).
    b) Start up an AWS EC2 instance running Centos 6.x (or Redhat Enterprise Linux 6.x) or higher. “Launch Instance”, Amazon Marketplace: “CentOS 7 (x86_64) – with Updates HVM”, “t3a.xlarge”. On the “Configure Instance Details” page, select your VPC in the “Network” option, and chose “Enable” for ipv4 and ipv6. On the “Add storage” page, increase the size of the root partition to 32GB or more figure 500MB/day of Bro logs on a moderately busy server).
    c) Install Bro/Zeek 2.6 or higher on that instance (versions 2.5 and below will not work). Make sure it’s running and set to start on next boot.
    d) To configure Centos 7: i) ssh to the system: 
    ssh -i yourkey.pem centos@instance_ip_address

    ii) Once on the system:

    cd /etc/yum.repos.d/

sudo curl -s -O https://download.opensuse.org/repositories/security:/zeek/CentOS_7/security:zeek.repo

sudo yum check-update

sudo yum -y install zeek tcpdump deltarpm nc lsof

sudo yum -y update

[ -L /opt/bro -o -d /opt/bro ] || (cd /opt ; ln -s zeek bro)

ifconfig #Look up primary network interface

sudo vi /opt/zeek/etc/node.cfg #Change eth0 to interface name

sudo vi /opt/zeek/share/zeek/site/local.zeek

    iii) Remove the “# ” in front of “@load policy/protocols/conn/mac-logging”. (No action needed to enable vxlan processing, it’s part of base/frameworks/tunnels/main.zeek which is loaded by default)
    iv) Save and exit.

    echo 'export PATH=$PATH:/opt/zeek/bin/' | sudo tee /etc/profile.d/zeek.sh >/dev/null

    v) Place zeek.service in /usr/lib/systemd/system (/lib/systemd/system on Ubuntu systems) as root. Then run:

    sudo systemctl daemon-reload

sudo systemctl enable zeek.service

sudo systemctl start zeek.service

    vi) For repository files to use for other distributions, see the subdirectories of https://download.opensuse.org/repositories/security:/zeek/ , and specifically, the security:zeek.repo file in them.
    e) Make sure the security group for this instance allows incoming UDP port 4789 traffic from the instance(s) you wish to monitor.

  3. If you don’t have an instance to monitor yet and want a test instance to watch:
    a) In Amazon’s EC2 configuration
    b) Start up an AWS EC2 instance running Centos 6.x (or Redhat Enterprise Linux 6.x) or higher. “Launch Instance”, Amazon Machine Image: “Ubuntu Server 18.04 LTS (HVM), SSD Volume Type – ami-0d5d9d301c853a04a (64-bit x86)”, “t3a.medium”. On the “Configure Instance Details” page, select your VPC in the “Network” option, and chose “Enable” for “Auto-assign Public IP” and “Auto-assign IPv6 IP”.
    c) Log in over ssh 
    d) mkdir $HOME/bin

    e) Save the following lines as $HOME/bin/tame-beacon

    #!/bin/bash

while sleep 34 ; do

echo -n '.'

ping6 -n -c 1 2604:a880:800:10::19:8001 >/dev/null 2>&1

echo 'Infected' | nc -n -u -q 1 -6 2604:a880:800:10::19:8001 999

curl -s https://www.activecountermeasures.com/ >/dev/null

done
    chmod 755 $HOME/bin/tame-beacon

    f) Add the following line to /etc/rc.local (ubuntu/debian) or /etc/rc.d/rc.local , above any “exit” line, if there is one:

    screen -S tame-beacon -t tame-beacon -d -m bash -c /home/ubuntu/bin/tame-beacon

    g) Reboot

  4. Create an ENI for the existing instance you wish to monitor (in this example, we’ll call it eni-capture-source) in Amazon’s EC2 configuration . Remember that this must be a Nitro instance.
    a) Under EC2 Dashboard/Network & Security, select Network Interfaces . Choose “Create Network Interface”. You’ll need to pick the subnet in which your existing instance lies. For IPv4 Private IP and IPv6 Setting, choose Auto-assign.
    b) Once created, select it and press “Attach”. Pick the system you wish to monitor. This ENI will show up as an additional network interface on that system – you can see it with “ifconfig -a” .
  5. Create an ENI for the packet analysis instance (in this example, we’ll call it eni-capture-destination) in Amazon’s EC2 configuration .
    a) Under EC2 Dashboard/Network & Security, select Network Interfaces. Choose “Create Network Interface”. You’ll need to pick the subnet in which your existing instance lies.
    b) Once created, select it and press “Attach”. Pick the packet analysis instance. This ENI will show up as an additional network interface on that system – you can see it with “ifconfig -a” .
    c) Reboot
    d) Run “ifconfig -a” again to get an updated list of interface names.
    e) Make sure you have an /etc/sysconfig/network-scripts/ifcfg-name file for all your interfaces from “ifconfig -a” above, especially the new one. If you have to rename one of the files to match the interface name, make sure you edit the “DEVICE=” line to match the interface name too. To bring the interface up without an IP address, the following lines should be added to that file: 
    BOOTPROTO="none" #formerly dhcp

IPADDR=0.0.0.0

NETWORK=0.0.0.0

NETMASK=255.255.255.255

    f) On that system, tell it to listen on this new interface. “sudo vi /opt/zeek/etc/node.cfg” and put the new interface name on the “interface=” line. Save, exit, and run:

    sudo /opt/zeek/bin/zeekctl deploy

    g) You can check the status of Zeek anytime with:

    sudo /opt/zeek/bin/zeekctl status
  6. Create a traffic mirror target.
    a) In http://console.aws.amazon.com/vpc/ at the bottom of the left-hand menu, choose “Mirror Targets” under traffic mirroring.
    b) Click “Create traffic mirror target”. Name it “mt-test”.
    c) You’ll need to pick which interface to which to route packets. To see the interface names, go to the EC2 console in a second tab, choose Instances, pick a system, and click on one of the network interfaces in the details pane below. The interface ID will show up there. Once you’ve picked the right interface, choose “Create”.
  7. Create a traffic mirror filter that sends all traffic.
    a) In http://console.aws.amazon.com/vpc/ at the bottom of the left-hand menu, choose “Mirror Filters” under traffic mirroring.
    b) Choose “Create traffic mirror filter”. Name it “tmf-test”.
    c) Add inbound rules that match what you want to inspect. If you want to inspect everything, you’ll need two rules; one with “All protocols” and “0.0.0.0/0” in both Source and Destination CIDR blocks, and the other with “All protocols” and “::/0” in both Source and Destination.
  8. Create a Traffic Mirror Session. In it, specify the source (eni-capture-source), the target (eni-capture-destination), and the filter (capture-all).
    a) In http://console.aws.amazon.com/vpc/ at the bottom of the left-hand menu, choose “Mirror Sessions” under traffic mirroring.
    b) Click “Create traffic mirror session”. Name it “tms-test”.
    c) You’ll need to pick which interface you want to mirror. To see the interface names, go to the EC2 console in a second tab, choose Instances, pick a system, and click on one of the network interfaces in the details pane below. The interface ID will show up there. Note: you may wish to use the Primary network interface of the system you wish to monitor as this is the interface over which packets will flow by default.
    d) Pick a Session number (1, unless you’re creating and prioritizing multiple sessions)
    e) Next, select the mirror target you created above (“mt-test”).
    f) Finally, select the mirror filter you created above (“tmf-test”) and press Create.
  9. Confirm that you’re receiving VXLAN traffic at your monitoring machine – see the Troubleshooting section below.
  10. The final step is to carry the Bro logs from your Zeek instance to your AI-Hunter system. Within a few hours you should see the traffic on the monitored system showing up in AI-Hunter.

 

Troubleshooting

To confirm that mirrored packets are arriving at your Zeek instance, install tcpdump if not already installed:

sudo yum install tcpdump || sudo apt-get install tcpdump

, then run the following command on the Zeek instance:

sudo tcpdump -i eth1 -qtnp 'udp port 4789'

Once that’s running, generate some traffic coming from or going to the monitored instance. If no lines show up in the tcpdump window, then there’s an issue with the mirroring. If you see lines of traffic then you have the traffic mirroring set up correctly.

 

Reference

Terms

AWS: Amazon Web Service, their cloud service
VPC: Virtual Private Cloud

More Information

Introduction article: https://aws.amazon.com/blogs/aws/new-vpc-traffic-mirroring/

More details on Traffic Mirroring: https://docs.aws.amazon.com/vpc/latest/mirroring/what-is-traffic-mirroring.html

In particular, there are step-by-step instructions at: https://docs.aws.amazon.com/vpc/latest/mirroring/traffic-mirroring-getting-started.html

VXLAN is used to carry mirrored packets from the source to the Zeek instance. VXLAN RFC: https://tools.ietf.org/html/rfc7348

zeek.service file contents:

#Run “sudo systemctl daemon-reload” after placing this in /usr/lib/systemd/system/
#(rpm-based) or /lib/systemd/system/ (debian/ubuntu)

 

#sudo systemctl start zeek.service #start right now
#sudo systemctl stop zeek.service #stop right now
#sudo systemctl enable zeek.service #start on next boot
#sudo systemctl disable zeek.service #do not start on next boot
#Also restart, try-restart (restart only if running), reload, status, is-active (running now?), is-enabled (will run on next boot?)
#sudo systemctl list-units –type=service –all

 

[Unit]
Description=Zeek network security monitor
Documentation=https://www.zeek.org/documentation/index.html
After=network.target

 

[Service]
ExecStartPre=-/opt/zeek/bin/zeekctl cleanup
ExecStartPre=/opt/zeek/bin/zeekctl check
ExecStartPre=/opt/zeek/bin/zeekctl install
ExecStartPre=/opt/zeek/bin/zeekctl cron enable
ExecStart=/opt/zeek/bin/zeekctl start
ExecStop=/opt/zeek/bin/zeekctl stop
ExecStopPost=/opt/zeek/bin/zeekctl cron disable
Type=oneshot
RemainAfterExit=yes
TimeoutStopSec=600

 

[Install]
WantedBy=multi-user.target

 

Related Post:

Cloud Network and Virtual Machine Monitoring

 

 

Interested in threat hunting tools? Check out AC-Hunter

Active Countermeasures is passionate about providing quality, educational content for the Infosec and Threat Hunting community. We appreciate your feedback so we can keep providing the type of content the community wants to see. Please feel free to Email Us with your ideas!

Share this:
Tags:
AC-Hunter Datasheet
GET THE PDF
AC-Hunter Personal Demo
SCHEDULE HERE
What We’re up To

SUBSCRIBE

Recent Posts
Categories
Archives