Threat Simulation – Certificate Issues

Intro

This article is number 7 of 8 in a series on testing Threat Hunting software to make sure that it’s configured correctly and working successfully. If you’ve not already read the “Threat Simulation Overview and Setup” article, start there and return here to test whether your Threat hunting platform can detect connections that are left open for hours.

What Traffic Are We Trying to Find?

When a user browses a legitimate web site, there’s a very high likelihood that the site is able to serve pages over TLS and has a valid, signed certificate. We can flag web sites that don’t have valid certificates as potentially suspect and list them for investigation.

An “invalid certificate” error doesn’t automatically mean that malware is in use. This can show up when:

– A server is installed but is using a self-signed certificate that’s generated during the installation process.
– The valid, signed certificate was incorrectly installed.
– The certificate has expired and needs to be replaced.
– The certificate is using one or more hash or encryption algorithms that are not allowed by local policy.
– The request being placed to a machine is for a domain not covered by its certificate.

While the first might be appropriate for a development or private server, it would still be worthwhile to keep an inventory of these. The remaining reasons are still worth investigating, along with any malware-caused communication to hosts with invalid certificates.

In this test, we’re going to look for that first kind of certificate problem – a self-signed certificate (more details here) The techniques we use should be able to catch other kinds of certificate or TLS errors if present on the server.

What Tools Are Needed

We’ll use the same tools we used in the Client Signatures section – “wget” and “curl” on the Internal system, and a web server on the External system (Apache and nginx are commonly available in most Linux distributions and relatively easy to install).

What to Set up on the External System

We need to add a self-signed certificate to the External web server. The steps needed vary depending on which flavor of Linux you have, so here are some tutorials to follow:

• Apache: Ubuntu Linux
• Apache: Debian and Ubuntu Linux
• Apache: generic Linux and Windows
• Apache: Centos Linux 7 and RHEL 7
• Nginx: Ubuntu, Debian, Centos, and RHEL

No matter which web server you use or what operating system it’s on, make sure you restart it after adding the certificate. Once it’s restarted you’ll also want to confirm that the web server is listening on TCP port 443; use one of the following commands on the External system:

sudo ss -tanp | grep 'LISTEN.*443'

LISTEN     0 128 :::443   :::* users:(("httpd",pid=32073,fd=6),

 

sudo netstat -anp | grep ':443.*LISTEN'

tcp6       0 0 :::443   :::* LISTEN 9723/httpd

What to Set up on the Internal System

We’ll place HTTPS requests to the External system. First, start a screen window that holds the running curl and wget processes:

screen -S testelfsigned -R

You’ll be given back a new command prompt running inside screen (so you can disconnect from it and reconnect to it at any time). In this new prompt, run:

while sleep 144 ; do curl -s https://Ext4 >/dev/null ; wget -q https://Ext6 -O /dev/null ; done

Once you’ve run these, you can press “Ctrl-A”, then d to disconnect from the screen session and let those commands run in the background. You can reconnect at any point with:

screen -S testselfsigned -R

and disconnect again with “Ctrl-A”, then “d”.

What You Should See on the Threat Hunting Platform

Your Threat Hunting software should monitor connections from Internal systems to External systems, and flag any SSL/TLS connections that are to servers with invalid certificates (including the self-signed certificate connections we’ve generated above). Within a few hours you should have at least two alerts; one for the Ext4 IP address and one for the Ext6 IP address.

How to Troubleshoot It If You Don’t

With Zeek

To detect a self-signed certificate with Zeek, look at the “notice*” and “ssl*” logs. To look for matching lines for this server, go to a directory with Zeek logs and run:

zgrep Ext4 notice* ssl* | less -S

In the notice* logs you’ll find lines with:

SSL::Invalid_Server_Cert        SSL certificate validation failed with (self signed certificate)

In the ssl* logs, you’ll see:

self signed certificate

Checking With a Web Browser

To simply confirm that the server has a self-signed certificate, open up a link to https://Ext6 in a web browser (note that this must be https). Chrome will report:

NET::ERR_CERT_AUTHORITY_INVALID

Subject: localhost

Issuer: localhost

Firefox’s advanced screen shows:

Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT

Checking the Certificate Attributes

To see the attributes of a particular certificate, you can either run a local script or have a web service do the analysis for you.

To run it locally, download “testssl.sh” from Here (source is Here). Here’s how you download the tool and run it against an HTTPS web server:

$ mkdir testssl
$ cd testssl
$ git clone --depth 1 https://github.com/drwetter/testssl.sh.git
Cloning into 'testssl.sh'...

...
$ cd testssl.sh
$ ./testssl.sh www.whitehouse.gov

If the certificate is self-signed, you see this in the output:

Issuer                       self-signed (NOT ok)

To use a web service go to https://www.ssllabs.com/ssltest/. To use their service, the system you’re testing must have a dns hostname; you can’t use this with just an IP address.

Both approaches give you details on insecure attributes.

Links to All Posts in “Threat Simulation” Series

1. Unexpected Protocol on Non-Standard Port
2. Long Connections
3. Client Signatures (User Agent)
4. DNS
5. Beacons
6. Threat Intel
7. Certificate Issues
8. Client Signatures (TLS Signature)

Bill Stearns

Bill has authored numerous articles and tools for client use. He also serves as a content author and faculty member at the SANS Institute, teaching the Linux System Administration, Perimeter Protection, Securing Linux and Unix, and Intrusion Detection tracks. Bill’s background is in network and operating system security; he was the chief architect of one commercial and two open source firewalls and is an active contributor to multiple projects in the Linux development effort. Bill’s articles and tools can be found in online journals and at http://github.com/activecm/ and http://www.stearns.org.