Safelist Synchronization

Overview

“safelist-sync” is a tool that copies safelist (formerly named whitelist) additions between AC-Hunter servers. This allows you to enter a new safelist entry on one AC-Hunter server and have it automatically show up on a group of others.

By running the synchronization tool in the background, you do not need to manually distribute safelist changes among your AC-Hunter servers; the sync tool will handle the replication.

 

How Does It Work?

safelist-sync is a program that runs on a Linux system. Every 300 seconds (configurable) it connects out to AC-Hunter servers, asking for their current safelist. It looks for safelist entries that haven’t been sent to all the servers that need them and pushes them out to each one.

 

Scenarios

All Safelist Entries Copied to All Servers

Safelist-sync is given a list of AC-Hunter servers. No matter which of these servers the safelist entry was created on, it is sent to all other servers within 5 minutes:

safelist-sync.py -s host1 host2 host62

“-s” specifies a list of sources. These same hosts will also get the changes from each other, so the changes are pushed bidirectionally.

This approach is especially helpful when you have multiple AC-Hunter consoles that monitor similar types of traffic, or when you have primary and secondary/disaster recovery servers.

 

Entries Pulled From a Limited Number of Systems and Replicated

Only a subset of the systems are considered sources of new entries, and there are others that only receive safelist entries. This would be more appropriate if host1 and host2 were considered trusted, and host3 and host4 should get new entries but not send any back.

safelist-sync.py -s host1 host2 -r host3 host4

If I make a change on host3 or host4, they will not be sent back to host1 or host2; host3 and host4 are just recipients. Because both host1 and host2 are sources, they will bidirectionally replicate just like before (in addition to sending their changes to host3 and host4.)

 

Pushing Changes From a Central Host to One or More Replicas

A single central machine accepts new safelist entries, then pushes them out to one or more recipients.

safelist-sync.py -s host1 -r host3 host4

 

Only Certain Safelist Entries Should Be Replicated

I may have a group of servers that monitor networks with lots of Windows machines. I’d like to share some safelist entries between them, but not with the other AC-Hunter servers I manage. To do this, I’m going to pick a name to describe the items of interest, and for this example I’ll pick “syncwindows”. I need to use this in two places; somewhere in the comment field of any safelist entries I want to share, and on the safelist-sync command line after the “-f” option:

safelist-sync.py -s host1 host2 host44 -f 'syncwindows'

This safelist entry – and any others that have “syncwindows” in the comment field – will be sent to the other hosts. Any safelist entries on these systems that don’t have “syncwindows” in the comment will be ignored by this copy of safelist-sync.

 

Other Groups of Systems Share Other Types of Safelist Entries

I can run multiple copies of safelist-sync, each focusing on a specific group of machines and keyword:

safelist-sync.py -s host1 host2 host44 -f 'syncwindows'
safelist-sync.py -s host1 host44 host75 -f 'syncntp'
safelist-sync.py -s host44 -r host107 -f 'syncsmb'

If I create a safelist entry on host44 that has all of these keywords in its comment field, that safelist entry will go to host1, host2, host75, and host107. If the comment has none of these keywords, it will not be replicated anywhere by these 3 running programs. If I have some of these keywords, but not all of them, the new entry will be sent to some (but not all) of these hosts.

 

Collection/Backup Server

One last one to consider is a backup server (host999 below) for all your safelist entries from all other AC-Hunter servers.

safelist-sync.py -s host1 -r host999 -w 3600
safelist-sync.py -s host2 -r host999 -w 3600
safelist-sync.py -s host3 -r host999 -w 3600
safelist-sync.py -s host4 -r host999 -w 3600
safelist-sync.py -s host44 -r host999 -w 3600
safelist-sync.py -s host62 -r host999 -w 3600
safelist-sync.py -s host75 -r host999 -w 3600
safelist-sync.py -s host107 -r host999 -w 3600

We can’t merge these into a single running copy of safelist-sync – if we did and called them all “sources”, they would not only send their changes to host999 but also would send all their changes to each other. Instead we use one sync program for each remote host to pull that host’s safelist entries back to host999.

We added “-w 3600” to say that this synchronization only needs to happen once an hour.

When set up this way all of the safelist entries are merged together into host999’s safelist with no direct way to separate them.

 

How to Set This Up

1. First, log into a system that can ssh to each of the target AC-Hunter hosts. It’s fine to use either an AC-Hunter system (see “Running on an AC-Hunter system”, below) or a Zeek sensor, as long as it can ssh to each target. You should log in as a non-root user.

2. Next, get a copy of safelist-sync from https://github.com/activecm/safelist-tools :

wget https://raw.githubusercontent.com/activecm/safelist-tools/main/safelist-sync.py
chmod 755 safelist-sync.py
sudo mv safelist-sync.py /usr/local/bin/

3. For each target host, edit ~/.ssh/config under that user’s account. You’ll need to add a section for each host (or edit the existing section if you have one or more). The lines you’ll want to add are in bold. You’ll need to edit anything in italics to match the host in question.

Host achunter1
        Hostname 12.13.14.15
        LocalForward 41001 172.23.0.4:8080
        ServerAliveInterval 180
        HostKeyAlias achunter1

4. In particular, the “172.23.0.4” on the LocalForward line is the IP address of the API component of AC-Hunter. To find out what IP address to use, ssh to that AC-Hunter system and run this (all on one line):

sudo docker inspect achunter_api | grep -i '"IPAddress": "[0-9]' | sed 's/.*"IPAddress": "\(.*\)",/\1/'

This will return an IP address that you’ll bring back to the sync system and place in ~/.ssh/config for that host. Note that this address may change between AC-Hunter systems or you may find they all use the same IP address.

5. The port number on the LocalForward line (“41001” in the previous example) must be unique for each AC-Hunter host. That means that the second AC-Hunter console’s section might be:

Host achunter-dr-site
        Hostname 85.86.87.88
        LocalForward 41002 172.17.0.3:8080
        ServerAliveInterval 180
        HostKeyAlias achunter-dr-site

6. On the sync system as the user under which you’ve been setting this up, start up a copy of screen:

screen -S tunnels -R

This will return a prompt. Start an SSH session to the first host with:

ssh -N achunter1

Now create a new prompt – which leaves that “ssh -N achunter1” running – with

Ctrl-a c

(press “Ctrl” and “a” , release both, then press “c”). Inside the new prompt, open an SSH connection to the second host:

ssh -N achunter-dr-site

Repeat until you have open connections to all hosts.

7. Now start up safelist-sync, one for each group of hosts with their filter strings. Note that when we talked about “host1”, “host2”, “host33” above, these are not the hostnames of the systems; these will be in the form “127.0.0.1:41001“, where 41001 is the associated port. Here’s an example command line where our two example systems synchronize bidirectionally, but only for new safelist entries with “syncthese” in the comment field:

safelist-sync.py -s 127.0.0.1:41001 127.0.0.1:41002 -f 'syncthese'

You can also run these inside the screen session or in separate windows as you choose.

8. To test this, log in to the web UI of one of the systems in the “-s / –sources” section. Find something benign to safelist and do so, but before you finally press submit make sure the comment field includes your filter string (like the “syncthese” in the previous section). Wait 5 minutes, then check one of the other systems in this group (either in sources or in recipients). Go to the dashboard, then the Gear in the upper right, then the safelist tab. Press “View/Edit” to see the safelist, and type “syncthese” in the Search box. You should see your new safelist entry there.

9. You can also create a new safelist entry that doesn’t have “syncthese” in the comment field on either host. After 5 minutes has passed, check the safelist on the other host; the entry should not have been carried over.

 

Running on an AC-Hunter System

There are 2 side notes if you run these scripts on an actual AC-Hunter system that’s taking part in this replication. First, once you run the command that finds the API IP address (

sudo docker inspect achunter_api | grep -i '"IPAddress": "[0-9]' | sed 's/.*"IPAddress": "\(.*\)",/\1/'

, you’ll use that IP and port 8080 instead of the forwarded port numbers (like 41001) we talked about above.

safelist-sync.py -s 172.18.0.3:8080 127.0.0.1:41002

Second, you won’t set up a section in ~/.ssh/config for this host, and you won’t start an ssh session with “ssh -N localhost” for it either.

 

Additional Notes

If 5 minutes is too long to wait for the entries to be replicated, the “-w” / “–wait” option lets you specify a number of seconds to wait between checks. While this delay could be as small as 1 second, that would give an appreciable load to the AC-Hunter servers. 30 seconds is probably a good lower bound. You can also raise this up to an hour (3600) or even a day (86400) if the replication is not time sensitive.

To get more detail about what’s happening, add “–devel” to the command line. In early testing use both –devel and –dryrun , which goes through most of the local processing but doesn’t send any actual changes to any host.

This is an excellent time to set up ssh keys with the private key on the sync system and the public key copied to all the AC-Hunter servers. It may also be good to load your key into ssh-agent.

Starting up synchronization does not affect safelist entries that already exist on one of the servers. It also does not affect items added to one of the servers that aren’t part of the replication (either because this server is a recipient or because the new safelist entry does not have a filter string that’s being mirrored.)

Any references to adding new safelist entries includes adding entries by hand in the web UI as well as adding entries via the other API scripts found at https://github.com/activecm/safelist-tools .

When you’re picking a filter string (like the “syncwindows”, “syncntp”, and “syncsmb” examples above), we encourage you to start each string with something that indicates that it’s involved with synchronization (like “sync…”).

Our setup above assumes you can ssh directly from the sync system to all AC-Hunter systems. If direct ssh connections are not possible, you can use any of ssh’s alternate approaches for connecting from sync to each AC-Hunter console:

• Going through a VPN
• Reverse tunneling where the AC-Hunter system connects to the sync system with Remote forwarding to carry SSH traffic back to the AC-Hunter system.
• Going through a bastion host
• Going through a socks or other type of proxy

You also have the option to use some other approach for carrying the traffic from a local port (like 41001) to port 8080 on the API docker container. If you do, please use an approach that encrypts the traffic and uses some form of strong authentication of both ends of the connection. We strongly discourage using port forwarding on the AC-Hunter servers as this is neither encrypted nor authenticated, and opens up the API to outside attackers.

To help remember what hosts the port numbers point at, you can create entries in /etc/services (on the sync machine) for each of them:

achunter1 41001/tcp
achunter-dr-site 41002/tcp

Now, when you start safelist-sync, you can use these port numbers:

safelist-sync.py -s 127.0.0.1:achunter1 127.0.0.1:achunter-dr-site

 

Project Status

• Current version: 0.0.4
• safelist-sync will duplicate new safelist entries from any source system to the other source systems and all recipient systems
• The filter functionality correctly limits replication to just those safelist entries that have that string in their comment field.
• It is not known whether modified safelist entries will also be replicated at this time.
• Safelist deletions are not correctly replicated at this time; in fact, safelist-sync will in most cases recreate the deleted entry. A future release is expected to fix this. As a temporary workaround, shut down the appropriate safelist-sync script with Ctrl-C , delete the entry from all systems in that group, then restart safelist-sync.

 

References

How to use screen: http://www.stearns.org/doc/screen-for-detachable-sessions.html

Using the ssh “config” file: https://www.activecountermeasures.com/ssh-no-more-memorizing-ip-addresses/

Quickly pushing the ssh public key to a server: http://www.stearns.org/ssh-keyinstall/quick-ssh-keyinstall