Threat Simulation – Unexpected Protocol on Non-Standard Port


This article is 1 of 8 in a series on testing Threat Hunting software to make sure that it’s configured correctly and working successfully. If you’ve not already read the “Threat Simulation Overview and Setup” article, start there and return here to test whether your Threat hunting platform can detect traffic running on an odd port.


What Traffic Are We Trying to Find?

On a network where access to the Internet is carefully firewalled, an attacker may find that the systems they’ve infected can no longer reach the Internet via their preferred ports. For example, the meterpreter shell that’s part of Metasploit connects back to its command and control server on TCP port 4444 by default; if outbound connections to this port are blocked by a network firewall, meterpreter needs to get sneakier.

The meterpreter shell and the C&C server could agree to use, say, TCP port 143 for their conversation instead. Assuming that the firewall allowed outbound TCP port 143 connections to allow users to connect to remote IMAP mail servers, this might allow meterpreter to connect to its command and control server.

How would we detect this? The meterpreter conversation, while it would be on port 143, doesn’t look anything like the conversation that an IMAP client and server would have – it would look like HTTP. For that reason, we want our Threat hunting platform to identify this as an “Unexpected protocol on well-known port” (in this case, HTTP on TCP/143).


What Tools Are Needed

In this example, we’ll be running SSH on port 110. The Threat hunting platform should flag the SSH conversation running on a port that should normally be carrying POP3 (mail) traffic.

The Internal machine will need both the openssh client (“ssh”) and the screen program installed. The External system does not need any additional packages installed.


What to Set up on the External System

We need to reconfigure the External ssh server to accept ssh connections on ports other than 22 (while continuing to accept conversations on port 22). To do this, follow the instructions here, using port 110 instead of port 2022. To confirm that it’s successfully listening on port 110, run the same command you’d normally use to ssh to External, but add “-p 110” to the ssh command line and verify that you can ssh in.


What to Set up on the Internal System

Make sure you’ve set up the SSH key as mentioned in the “Threat simulation overview and setup” article – we’ll need it for these next steps.

Now set up the background task that will repeatedly connect to the External system with ssh over port 110:

screen -S testbadport -R

You’ll be given back a new command prompt running inside screen (so you can disconnect from it and reconnect to it at any time). In this new prompt, run:

while sleep 207 ; do ssh -i "$HOME/.ssh/id_rsa_threattest" -p 110 Ext4 
"echo success4" ; ssh -i "$HOME/.ssh/id_rsa_threattest" -p 110 Ext6 
"echo success6" ; done

Once you’ve run these, you can press “Ctrl-A”, then “d” to disconnect from the screen session and let those commands run in the background. You can reconnect at any point with

screen -S testbadport -R

and disconnect again with “Ctrl-A”, then “d”.


What You Should See on the Threat Hunting Platform

Your Threat hunting software should report that it’s seeing non-imap traffic on port 110. If it’s able to successfully identify that the traffic is ssh traffic, excellent.

Note that the TCP conversation is happening every 207 seconds, so these connections should show up as beacons as well.

How to Troubleshoot It If You Don’t

With tcpdump

If you are not seeing this traffic reported as unexpected protocol on well-known port, check that the conversation is successfully being started on the Internal server (substitute the name of the Internal system’s ethernet port for eth0):

sudo tcpdump -i eth0 -qtnp '(host Ext4 or host Ext6) and port 110'

If it is, run the same command on the External server (substitute the name of the External system’s ethernet port for eth0):

sudo tcpdump -i eth0 -qtnp '(host Ext4 or host Ext6) and port 110'

Within 10 minutes you should see at least one IPv4 and one IPv6 conversation on that port between the two pairs of IP addresses for those systems. If you don’t, see if there’s a firewall that’s blocking this traffic.

If you have command-line access to the Threat hunting system, run the same command there, changing “eth0” to the name of the interface that’s capturing traffic:

sudo tcpdump -i eth0 -qtnp '(host Ext4 or host Ext6) and port 110'

If the traffic shows up on the Internal and External systems, but not when you run the sniffer command on the Threat hunting system, recheck that the Threat Hunting system is plugged into a span/mirror/copy port on a switch that sees the traffic from Internal to External.

With Zeek

Zeek does report on the port, protocol, and type of service seen in a given connection. For example, if I ssh to a remote system on the default port of 22, zeek will include “22”, “tcp”, and “ssh” in the log line for that connection.

Ideally, we’d like to know when someone’s using a non-default port, so we’ll have to discard ones on default ports. Here’s an example of this that includes a reasonably complete list of standard ports:

That’s a bit dense; here’s what it’s doing:

– Go to a directory with Zeek connection logs

– Spit out all the lines from all logs whose names start with “conn” and end in “log.gz”

– In those lines, only grab the following columns: source ip, destination IP, destination port, protocol, and service. An example of these would be:  8801     udp    dtls

– Reformat those lines so that port, protocol, and service are separated by colons, looking like:  8801:udp:dtls

– Now we want to discard all connections where the service is normally found on that port, like “22:tcp:ssh”, “80:tcp:http”, and “443:tcp:ssl”. The egrep does that for this starter list of ports, along with discarding any ftp-data lines (which can show up on any port), and any lines where the service is “-“; this happens when Zeek doesn’t recognize the service.

– Finally, throw away duplicates and order the remaining unique lines in IP order.

Remember that this is a starting list of common services and their default ports. You’ll almost certainly have to add entries that we’ve not included.


Ways to Obfuscate the Beacon Behaviour

Because the requests are always 207 seconds apart, this is a clear clue to threat hunting packages that this is a beacon. To obscure that, replace “sleep 4” with:

sleep $[ $RANDOM / 1000 + 190 ]

which will sleep for a random number of seconds between 190 and 222 . This won’t totally hide the beacon behaviour, but will make it a little harder to classify.

The ssh commands we used all run a simple command: “echo success4” or “echo success6”. The payloads for these connections will have similar sizes.
To give them different sizes, replace the ‘echo success4’ / ‘echo success6’ with a command of random size. We can put some random padding (a random number of random characters) at the end of our “echo success4” like:

"echo success4 #mVO9XkZrxsphMwUtd8JoHtF8ToGm5x8lTP"

The shell at the other end will simply ignore the characters from “#” on.

Here’s an example with both a random time interval and a random payload size:

while sleep $[ $RANDOM / 1000 + 190 ] ; do ssh -i 
"$HOME/.ssh/id_rsa_threattest" -p 110 Ext4 "echo success4 #`dd 
if=/dev/urandom bs=1 count=$[ $RANDOM / 50 ] 2>/dev/null | tr -dc 
'A-Za-z0-9'`" ; ssh -i "$HOME/.ssh/id_rsa_threattest" -p 110 Ext6 
"echo success6 #`dd if=/dev/urandom bs=1 count=$[ $RANDOM / 50 ] 
2>/dev/null | tr -dc 'A-Za-z0-9'`" ; done

The 2 blocks from “dd” to “tr” are executed on the Internal system since they’re surrounded by backquotes. That means we extract a random number of bytes (0-656) and discard any that aren’t alphanumeric, stuffing the remaining random letters and numbers after the “#” and before the double quote. That means the command we’re sending across varies in size from 10 to 666 bytes.

The Threat Hunting software may reduce its certainty that this new traffic is a beacon, but it should still be recognized as such.


Links to All Posts in “Threat Simulation” Series

  1. Unexpected Protocol on Non-Standard Port
  2. Long Connections
  3. Client Signatures (User Agent)
  4. DNS
  5. Beacons
  6. Threat Intel
  7. Certificate Issues
  8. Client Signatures (TLS Signature)



Interested in threat hunting tools? Check out AC-Hunter

Active Countermeasures is passionate about providing quality, educational content for the Infosec and Threat Hunting community. We appreciate your feedback so we can keep providing the type of content the community wants to see. Please feel free to Email Us with your ideas!

Share this:
AC-Hunter Datasheet
AC-Hunter Personal Demo
What We’re up To

We're always up to cool threat hunting stuff at Active Countermeasures. Sometimes we want to share our cool stuff and we'll mail it to this list. We are not spammy, and no salesy junk!

* indicates required

Choose the Email Categories You Want to Receive:

(un-check any you don't want)

We're always up to cool threat hunting stuff at Active Countermeasures. Sometimes we want to share our cool stuff and we'll mail it to this list. We are not spammy, and no salesy junk!

* indicates required

Choose the Email Categories You Want to Receive:

(un-check any you don't want)