Threat Simulation Overview and Setup

Intro

No software project is complete without testing. In this blog series, we’ll cover how to test if your Threat Hunting platform can detect common threats.

The approaches we give should be usable with any threat hunting software, including home-grown tools and using a SIEM. If you’re evaluating a threat hunting package, use these steps as a way to 1) confirm that it works at all, 2) confirm that it can detect each of the threats, 3) evaluate its ability to sort threats with the most likely malicious ones at the top, and 4) evaluate the amount of manual effort needed to find the threats. 3) and 4) are important – a threat hunting package could technically detect a threat, but it may be buried 27 pages down in a list of other potential threats. Worse yet is a threat hunting package that simply cannot detect one of these threat types, such as C2 communication over DNS.

This series will be posted to our blog on Tuesdays and Thursdays with the last post on April 16th, 2020.

 

Setup

We assume you’re using Linux on both the Internal and External systems; we provide installation commands for Debian/Ubuntu-based and Fedora/Redhat Enterprise/Centos-based platforms. If you prefer, you should be able to use a Mac OS or one of the BSDs for the Internal system, and one of the BSDs for the external system. You may be able to use Windows for them, though you’ll very likely have to use Cygwin or some other external package tool to provide the needed tools.

If your system responds that it cannot find one of the requested tools, try these alternative package names:

For netcat, try: netcat, nc, nmap-netcat
For zeek-cut, try: bro-cut

If you’re using RHEL 8 or Centos 8 for either system, please enable the EPEL (Extra Packages for Enterprise Linux) repository as it includes the “screen” tool we’ll need:

sudo dnf -y install epel-release
sudo dnf config-manager --set-enabled PowerTools

 

External System (“Ext”)

We need a machine out on the Internet to act as the command and control server; we’ll refer to that as Ext or External. The simplest way to do this is to create a Cloud server; it can be spun up and down on a moment’s notice, can be as small or as large as you need, and can run any packages needed for this testing.

It should have both IPv4 and IPv6 addresses so you can test that Threat traffic is detected on both. When we refer to these addresses we’ll use Ext4 and Ext6.

You should install the following packages: netcat, screen, and tcpdump
To install these on a Debian or Ubuntu-based system, run:

sudo apt install -y netcat screen tcpdump

To install these on a Fedora or RHEL-based system, run:

sudo yum -y install nmap-ncat screen tcpdump

To perform the DNS checks, you’ll want to install the bind nameserver. To perform the Certificate Issues and Client Signatures checks, you’ll want to install a webserver:

To install these on a Debian or Ubuntu-based system, run:

sudo apt install -y bind9 apache2

To install these on a Fedora or RHEL-based system, run:

sudo yum -y install bind httpd

 

Internal System (“Int”)

This system will act as our infected/infiltrated system on the inside of the network. This will be referred to as Int or Internal, and its IP addresses will be Int4 and Int6.

You should install the following packages: netcat, screen, dnsutils/bind-utils, tcpdump, wget and curl.

To install these on a Debian or Ubuntu-based system, run:

sudo apt install -y netcat screen dnsutils tcpdump wget curl

To install these on a Fedora or RHEL-based system, run:

sudo yum -y install nmap-ncat screen bind-utils tcpdump wget curl

If you’ve not used “screen” before, I made an introduction blog.

For a number of these checks we’ll need the ability to ssh from the Internal system to the External system. We’ll set this up to use ssh keys so we don’t have to supply a password each time.

As a non-root user (*) on the Internal system, run the following commands. First, create an ssh key for connecting if you haven’t already:

ssh-keygen -b 2048 -t rsa -N "" -f "$HOME/.ssh/id_rsa_threattest"

Now we can push this key over to the external system. To do this you’ll need to supply your password on the External system this one time:

cat "$HOME/.ssh/id_rsa_threattest.pub" | ssh Ext4 'mkdir -p .ssh ; cat >>.ssh/authorized_keys ; chmod go-rwx ./ .ssh/ .ssh/authorized_keys'

And confirm that you can now successfully connect without a password.

ssh -i "$HOME/.ssh/id_rsa_threattest.pub" Ext4 'echo success'

* We encourage you to use the same user for all these checks. Since we assume this is a throwaway system, you could certainly use your normal account, or you could create a throwaway user under which these checks are done.

 

Threat Hunting Platform

Your threat hunting platform should be on the network between the Internal and External systems. If it’s running directly on a router on the path between the two, you should be all set to capture traffic between the two systems.

If your threat hunting software is listening to a network segment between the two, remember that you need to be using a span port, tap, copy port, or mirror port to get a copy of the packets. Simply plugging your sniffer into a switch port that is not one of the above will not get you the traffic you need to identify threats.

If you’re trying to do your sniffing inside a virtual environment like a cloud provider’s network, it gets more complicated – see our video and our blog posts 1 & 2.

For the purpose of testing, you can instruct your threat hunting software to only report traffic between these two systems. This could be as simple as using the filter:

'(host Int4 or host Int6) and (host Ext4 or host Ext6)'

This does allow you to confirm that the individual threats are detected at all. However, since you’re only seeing traffic between the two systems, you can’t tell where your threat hunting software would show this in a display that includes all the normal traffic on your network. If you do this we encourage you to rerun the tests later without this filter in place so you can find out if the Threat traffic is buried below normal traffic.

Make sure you remove this filter when you go back into production!

 

IPv6

If your network, Internal system, and/or External system do not support IPv6, skip any instructions that need it. For example, the filter in the previous section can be replaced with:

'host Int4 and host Ext4'

 

Firewalls

If you’re running any firewalls (this includes perimeter firewalls, routers with ACLs, cloud platform security groups or firewalls, individual firewalls on the Internal or External systems, or IPS systems), you’ll want to make sure they allow traffic between the Internal and External systems. If your firewall rules allow you to put in a rule that also allows all responses to that traffic, use rules like:

Allow all traffic from Int4 to Ext4 (and allow replies)
Allow all traffic from Int6 to Ext6 (and allow replies)

If your firewall has no ability to allow all replies, you can do:

Allow all traffic from Int4 to Ext4
Allow all traffic from Int6 to Ext6
Allow all traffic from Ext4 to Int4
Allow all traffic from Ext6 to Int6

When your testing is complete, make sure you remove these rules.

 

Performing the Test

The other blogs in this series give you the steps to perform. For example, “Threat simulation – beacons” gives you all the steps needed to confirm that your Threat Hunting system can detect beacons.

Keep in mind that for each of these you may have to wait anywhere from a few minutes to a good part of a day for these results to show up. Because of this delay, you may want to start them all up and come back the next day to check that they were all detected.

Using tcpdump and Zeek in Troubleshooting

This process should be usable with any Threat Hunting platform. That said, we want to provide you with steps you can use to troubleshoot problems such as not detecting the particular threat.

The first tool we’ll use is tcpdump. This is available on any Linux or Macintosh system (and while we won’t cover it here, it is available as “windump” for Windows systems). It’s not intended to be a full Threat Hunting package but is sufficiently strong to help diagnose problems.

Our second tool is Zeek. This, too, is available for most Linux distributions and MacOS (see https://zeek.org/get-zeek/ ).

If Zeek is not currently available for your system, see if “Bro” is included in the list of available packages. Bro is the older name, and while it might be missing a few features, can certainly stand-in for this type of troubleshooting. If you do this, any command we use that has “zeek” in the name will need to be renamed to use “bro”, such as replacing “zeek-cut” with “bro-cut”.

 

Evaluation

Once you’ve checked for the different types of threats, there are a few things to consider as you finish the evaluation:

1. How many of the threats were found?
2. If your platform ranks threats according to relevance, severity, or in the order in which they should be investigated, where did these test runs show up? Was this appropriately high for the risk?
3. Did you feel your platform gave support to the follow-up investigation?
4. Does the threat hunting platform make it easy to remove/filter/whitelist conversations that you feel are legitimate (or do not require any followup)?
5. Does the platform successfully identify the traffic when it’s obfuscated?
6. Is obfuscated traffic still shown appropriately high on the list of potential conversations?
7. For any conversations that were either not caught or were so low on the investigation list that they’re effectively hidden, do you have other tools or mitigation techniques as a backup?
8. Is the package confused by or otherwise unable to handle encrypted traffic?
9. Is the package restricted to just capturing packets live on an interface, or can you tell it to analyze packets that were saved earlier in a pcap file?

 

Links to All Posts in “Threat Simulation” Series

1. Unexpected Protocol on Non-Standard Port
2. Long Connections
3. Client Signatures (User Agent)
4. DNS
5. Beacons
6. Threat Intel
7. Certificate Issues
8. Client Signatures (TLS Signature)