Threat Simulation – Long Connections

Intro

This article is number 2 of 8 in a series on testing Threat Hunting software to make sure that it’s configured correctly and working successfully. If you’ve not already read the “Threat Simulation Overview and Setup” article, start there and return here to test whether your Threat hunting platform can detect connections that are left open for hours.

 

What Traffic Are We Trying to Find?

The infected system in your network needs to be able to connect to its Command and Control (C2) server to both receive commands and send back results. The two primary ways of doing that are to 1) open a connection at regular intervals, say every 5-300 seconds, and 2) open up a connection and just leave it open for as long as possible.

While this is technically legal to do – a TCP connection could stay open for years – it’s somewhat uncommon. When you read your mail, download a web page, or do most of the tasks that happen over the Internet, the connection is opened up just long enough to get the data, and then closed. There are exceptions (downloads of massive files, some database replication tasks, routing table transfers, and ssh connections that are left open), but short (<5 minute) connections are the norm.

For this reason, anything that’s left open for a long time should be checked, whitelisted if intended, and investigated if not.

 

What Tools Are Needed

We’ll use a few tools that can leave a connection open for a long time. We’ll use ssh (installed by default) and netcat (installed in the overview article). Note that netcat may be named “nc” or “netcat” depending on the operating system; if you get “command not found” with nc, use netcat instead.

Make sure you’ve set up the SSH key as mentioned in the “Threat Simulation Overview and Setup” article – we’ll need it for these next steps.

 

What to Set up on the External System

We’ll set up netcat to listen on TCP port 8877. First, we start screen so that we can leave this task running in the background:

screen -S testlongconn -R

You’ll be given back a new command prompt running inside screen (so you can disconnect from it and reconnect to it at any time). In this new prompt, run:

nc -k -l 8877

Once you’ve run that, you can press “Ctrl-A”, then “d” to disconnect from the screen session and let netcat run in the background. You can reconnect at any point with:

screen -S testlongconn -R

and disconnect again with “Ctrl-A”, then “d”.

 

What to Set up on the Internal System

We’ll open three connections from the Internal server to the External. We’ll leave the first ssh connection open for 4 hours, and the second ssh connection for 12 hours. The nc connection will stay up until we take it down. Since Zeek doesn’t add a connection to its logs until the connection closes, we want something that is both long enough to show up in a Long Connections report, but won’t cross a day boundary or take forever to get logged at all.

Start a screen session to hold the three commands:

screen -S testlongconn -R

You’ll be given back a new command prompt running inside screen (so you can disconnect from it and reconnect to it at any time). In this new prompt, run:

ssh -i "$HOME/.ssh/id_rsa_threattest" Ext4 'sleep 14400 ; echo slept 4 hours'

Now open a new terminal in this screen session with “Ctrl-a”, then “c”. In this second terminal, run:

ssh -i "$HOME/.ssh/id_rsa_threattest" Ext6 'sleep 43200 ; echo slept 12 hours'

Now open up one more new terminal in this screen session with “Ctrl-a”, then “c”. In this third terminal, run:

nc Ext4 8877

Once you’ve run these, you can press “Ctrl-a”, then “d” to disconnect from the screen session and let those commands run in the background. You can reconnect at any point with:

screen -S testlongconn -R

and disconnect again with “Ctrl-A”, then “d”.

 

What You Should See on the Threat Hunting Platform

For this one you’ll have to wait 5 and 13 hours, respectively, to see all three connections show up in the Long Connections report. In there you should see the two ssh connections, one with the IPv4 addresses of the two systems, one with the IPv6 addresses of the two systems, and the netcat connection on port 8877.

How to Troubleshoot It If You Don’t

With tcpdump

Seeing the connections between the two hosts is not too complex if you start the packet capture before you launch the ssh and nc commands (run this on the Internal system and substitute the name of the Internal system’s ethernet port for eth0):

sudo tcpdump -i eth0 -qtnp '(host Ext4 or host Ext6) and (tcp port 22 or tcp port 8877)'

and if it is, run the same command on the External server (substitute the name of the External system’s ethernet port for eth0):

sudo tcpdump -i eth0 -qtnp '(host Ext4 or host Ext6) and (tcp port 22 or tcp port 8877)'

You should see at least one IPv4 and one IPv6 conversation on that port between the two pairs of IP addresses for those systems. If you don’t, see if there’s a firewall that’s blocking this traffic.

If you have command-line access to the Threat hunting system, run the same command there, changing “eth0” to the name of the interface that’s capturing traffic:

sudo tcpdump -i eth0 -qtnp '(host Ext4 or host Ext6) and (tcp port 22 or tcp port 8877)'

If you start up any of the above three packet captures after you’ve launched these ssh and nc commands, you may not see anything at all. Both ssh and nc can fall back on a quiet mode where they’ve set up the TCP connection but there’s no data to actively send; watching the wire when this happens may not show any packets.

To trigger some actual traffic to see, log back in to the Internal system and run:

screen -S testlongconn -R

to reconnect to the window holding the 2 ssh commands and the nc command. You should be looking at the nc command. Type a couple of characters, which nc will send across to the server (If you log into External and run the same screen command you’ll be able to see the characters show up there). Now you can look at all the tcpdump commands; all three should have seen this netcat traffic on port 8877. By pressing “Ctrl-d” you’ll close the netcat connection and now the connection tracking logic in your IDS/Threat Hunt software can see and log a closed connection.

If the traffic shows up on the Internal and External systems, but not when you run the sniffer command on the Threat hunting system, recheck that the Threat Hunting system is plugged in to a span/mirror/copy port on a switch that sees the traffic from Internal to External.

With Zeek

Most connections in Zeek have a “duration” value we can extract and sort by. Here’s an example of pulling the 8 longest connections out of all the Zeek logs for a given day:

This command grabs all the lines from the connection logs for this day, extracts and formats the connection duration, source IP, destination IP, destination port, protocol, and service. We discard any that have less than 10-second duration and sort by that field with the longest connections up top, showing the top 8.

 

Links to All Posts in “Threat Simulation” Series

1. Unexpected Protocol on Non-Standard Port
2. Long Connections
3. Client Signatures (User Agent)
4. DNS
5. Beacons
6. Threat Intel
7. Certificate Issues
8. Client Signatures (TLS Signature)