03-26-2019 ACM Webcast: Network Decoding Command and Control Channels

Timeline:

Files:

Transcript of This Webcast:

So yes, brought to you by Black Hills and ACM, yay, and as John was saying, we’re really thinking of making this a making this a series. So if folks are into that, give us a heads up let us know.

There will be some information on you know how to go about getting a hold of us first, but I wanted to kind of run through like why we do these, because this is something that like, John and I chat about a lot because when you… and it’s funny because we’ll like attend somebody else’s webcast on occasion and it’s very, very different.

We were talking about the like, missing banter, they also tend to be, you know, hey this is what we sell and come buy what we sell. And you know, we talked about that a little bit… we got to keep the lights on, but at the same time we try to focus more on the tech side.

And you know, my analogy is down at the bottom it’s like a car. If you don’t know how your car works when it makes a noise, you’re going to pay a lot more to get that fixed than you probably had to if you don’t understand how it works.

So what we’re trying to do is convey that information you need to make good choices about the tools that you find. I, as my kids heard their whole life growing up, you know, we need to trust you to be able to make good choices.

So like I said, thinking of making this a series, and if we do make it a series, would love some ideas on which ones to cover next.

dnscat2 is one of my personal favorites, which is hey, why we’re covering it first, but there’s some other cool ones out there that we can go through and talk about. And as John said I’m gonna share a link at the end of this, so that you can grab the pcaps that I use. You can go back through everything that I’ve gone through and done if you’d like to just try and reproduce it. Maybe you find stuff I didn’t, that would be awesome. I would love to hear about that, because you know one brain only finds so much.

But we’re going to start off talking about dnscat2, and one of the things I love about dnscat2, is that it’s really good at getting out of environments. You know when you think about an attacker has gone to all the trouble to compromise one of your internal systems, they want to make sure that software can reach back to the command and control server. Your networks’ a black box, they don’t know for sure how they can get out, so what can they rely on, well so long as you’re sending DNS compliant messages to the local DNS resolver that will probably relay it out to a command-and-control server so you can get out from there, so dnscat2 leverages that capability.

As a bunch of different query types it goes in and cycles through we’ll look at some of that as we go through, but that’s what allows it to communicate back to the command and control server it can communicate with it directly and circumvent all the DNS but those that’s much easier to tag it’s when it follows the normal DNS trail that it becomes really hard to try and track.

And by follow the normal DNS trail, I mean exactly that, so I have my internal compromised system here, it’s sending a DNS query to the local resolver because its DNS compliant the resolver looks at that and says, okay were you trying to go, alright I don’t know where the name servers are the authoritative for that domain let me go to the root name servers, you know, work my way down from there until it finds the name servers for that remote domain and those name servers are actually the command and control servers. so the query that comes in lets the command and control server know, hey that’s one of my minions and my minions looking for marching orders “what do you have me to do next”, so it will either tell it you know “go back into sleep mode” or “hey go grab all the local Excel spreadsheets” or whatever it is you want they have a thing to do.

So here’s an example of what it looks like when it’s doing a direct connect. so this is some tcpdump output, you can see I’m connecting to an IP address out on the Internet, in this first packet it’s not connecting to the local resolver, it’s an MX query so it’s doing a mail query that’s kind of interesting, as I’ve said it does cname it also does text. We’ll look at those a little bit later too but a couple of giveaway signs here, one is when it does a direct connect it always puts DNS cap right in the payload so any IDs would be able to kind of pick up on this.

Another thing that’s kind of interesting with this, is the response that comes back, you get one answer to your question. so we did an MX query against a hostname. Number one; that’s really odd today, it’s RFC compliant and in fact, I’m old enough to remember when we used to have mail deliver it that way. You would actually have mail delivered directly to your local system, but we don’t do it that way anymore, it’s always to the domain itself. So for someone to do an MX against a specific hostname, it’s legal but it’s weird, it’s something that’s probably worth taking a look at.

The other thing that’s kind of odd in this is if I look at my name server records, my additional records that both set to zero, again it’s legal, it’s just odd for a system to not hand that information back just in case the host may need it so it doesn’t need to do an additional lookup. So this is DNS compliant, but there’s a couple of telltale signs in here we can kind of key in on.

Now here’s a connect going to the local resolver. So this is you know, east-west traffic staying on the same network, this one’s cname query, this one’s a text query and you can see I’ve got a really long hostname in here that I’m going in and query, and we’ll talk about that one later. out at this domain here honestimnotevil com. Yes I could not believe that one was available to register, so I just snag that one really quick, but it’s doing a cname query, again when the answer comes back on this one no additional records which is which is especially weird for a cname because with the cname I’m saying, hey I want to know what’s associated with this alias. Well, you kind of give the additional information to get that – you’re gonna be doing multiple queries. So again, a lot of odd stuff in here that we might be able to kick in. So looking into the headers, sometimes into the payload, we can probably tag this stuff. MX’s that are not going to the domain but a go in a specific host we said those are weird texts going to a specific host is not really really odd but it is a little odd that’s another one that might kind of pay off are you a little bit.

I’ve got a question on that one you said that text going to a host is not really odd but it is somewhat strange. Yeah, um, what are some legitimate situations where you would see a text record actually being used?

I have seen there are hosts out on the internet and it’s funny because I drag this out of Bill Sterns’ database, there are hosts on the internet that still publish text records associated with the host to identify what services are on that host itself. So if there’s a host out there that you don’t know, you know is the web server is an FTP server what ports are open some people actually populate the text files to be associated with that not saying that’s a great idea and I’m not saying it’s common but I have seen that on occasion.

Well, and for people that are interested, Bill does a tremendous amount of research for Active Countermeasures and he’s released a tool called Passer. Yeah, I just put that out in the text in the chat so you guys can look at what Bill does so whenever we say Bill found this out in the wild it’s usually with something like Passer or a tool that he’s released open-source so check that out as well, yes.

And we all have faster internet in the US this week because Bill is over in Tokyo.

That’s actually not a joke, like how many wireless routers has he killed just visiting your house?

Ah… two he has killed, actually it was a border router and a wireless access point he’s killed visiting my house. There’s been a number of public events where all the bandwidth went away and they’re trying to figure out who it is and I just went over and told Bill stop it, and when he shut his machine the problem went away. Yeah Bill, like Bill’s a madman. Bill’s a madman. Which is why we hired him.

So yep, and Mike just popped up and he said that Cisco actually uses the text records and their expressway products as well. Good to know, good to know. Hey so while we’re here, any other questions? No, so far so good. Cool.

So one of the ways we can go about trying to tag this stuff, is to look for beacon-like activity.
Look for these things calling home on a repetitive basis.

You know, let’s start with like, what is a beacon? A beacon is an automated non-human process that’s connecting out to some host out on the internet. Doesn’t necessarily mean it’s evil, and I’ll give you a great example in NTP. If you’re on a Linux system, actually specifically if you’re on Ubuntu – every 15 minutes it will call out to an NTP server, and it will do it with exactly the same session size every time. So on a very specific time interval, it will send a very specific message size. That’s like the epitome of a beacon, but that’s also the epitome of a beacon that is not actually evil.

So we will need to be able to whitelist out stuff like that, but when it’s system gets backdoored, that’s the type of behavior we’re actually looking for. We’re looking for that software, calling up to some host out on the Internet and they’ll be all the characteristics involved with it obviously. You know you won’t be able to identify the application layer as being NTP, obviously. It may you know, vary its interval and do some other fun stuff that we’ll go through and we’ll kind of talk about as we go through.

But one of the ways to start looking for these things is to look for any signs of beacon activity on your network. Now I’ve got a couple of pro tips down the bottom, and it’s basically start with the most obvious things first and you know I know a lot of folks probably sitting back and saying well if I wrote my backdoor or I would make sure I didn’t show up on that. Hey, that’s great but you figure you gotta start somewhere, you might as well start with the most obvious, because that’s the easiest to go in and kind of pick apart.

You know, and that’s really interesting. There was a huge Twitter conversation, I don’t think I’ve had a chance to talk to you about this yet – there was a huge Twitter conversation on domain fronting, and a lot of these different techniques and it basically boiled down to, Tim Malcombetter at Walmart kind of kicked this thing off, and it seems from what we’re seeing and what people are talking about that a lot of the techniques that are being used by red teams and penetration testing firms, are way far on the more difficult detect spectrum than the actual attacks that we’re seeing.

So you know if people are always playing that game of well, an attacker can make it beacon once a day, yeah they could but that’s actually a useless beacon to most attackers. So you know, like, like you said it’s start with the low-hanging fruit and work your way up. Don’t automatically start with the assumption that it’s the most difficult thing in the world to detect.

Yeah, and you know quite honestly, going after the most difficult stuff first is going to take you the longest. The stuff that’s really noisy, that’s usually the easiest if it’s not evil, it’s the easiest to make go away and it’s the most obvious.

That reminded me of one, and John I know you were involved with this, we had a customer that had HVAC systems that we’re calling into a central console. Just basically to say, hey I’m online here’s what I think the current temperature is, here’s what I’m set to you know and I’m in a run state. And someone did a misconfiguration, where it was supposed to call in once per minute and it was calling in once every millisecond. I hate it when I get that decimal point wrong, but it was literally the most noisiest things on the wire, but they you know number one it was easy to kind of run down what was doing it, number two was also easy to fix, and they got some free bandwidth out of it which was kind of nice.

So, don’t forget about the side benefits of these things. You know sometimes it might not be evil but you might be able to make the IT world a little bit better, within your environment as well.

All right, so we’re gonna start with one of my favorite tools; tshark, yay. I love tshark, don’t get me wrong I’ve been a tcpdump user for years, anybody’s gone through my SANS classes probably learned more tcpdump than they ever thought they would ever need to use. But I really love tshark and I love tshark’s ability to be able to say, these are the specific pieces of information I’m interested in, ignore everything else to show me these things.

So what I’ve done in this here, my -T fields allows me to identify these are the specific fields I want to say, so its destination IP address, okay that’s pretty straightforward, I’m also going in and I’m saying I want you to show me the protocol that’s being used using protocol 17 that’s UDP, which port number and for our purposes beaconing we’re looking at what’s the session size now since this is UDP it’s just the size of the packet and what is the time spread interval between each transmission that’s taking place. Now notice my top line is zero, I’m going to need to normalize that out before I go in and do any types of statistics. so everything else is that time gap that took place between it and the session that was seen just prior to that. And in this case here I just, you know, ran it through head so we could see a couple of lines of output. But ideally what you want to do is take this and redirect that to a file, because once it’s in a text file, hey, you can do all sorts of nasty things with it like leverage the cut command to go in and say okay I want to look at this one specific field within that file and identify for me how many packets am I seeing associated with each protocol. It’s kind of nice to be able to go in and get a quick protocol breakdown, You know, what ports are being used now when I look at this file I see most of its UDP I do have two packets that are based on TCP, okay that makes it interesting, I’m going to want to take a closer look at that one in a second but they’re all associated with DNS.

So you know, where do I start? For me the most obvious let’s go in and kind of start with the TCP. Now when I go in and I look at the TCP I can see my internal system was sending to port 53 on some host out on the Internet, okay that’s odd, my system, my own my local host should not be doing direct connects to DNS servers out on the internet, so that kind of makes this one stand out a little bit. But also notice I’m seeing two packets or two syn flag packets I don’t have a syn ACK coming back, which tells me my system was trying to connect to that port there was a firewall in the way or there was something that was making the syn packets go away before it got to that end point because I don’t see a reset packet coming back telling me the ports closed.

So what I did here is I use grep to say okay that text file I created – V any TCP traffic, so comma 6 comma, get rid of all the TCP traffic and now I’m just gonna look at the beacon data set as part of the UDP trail. So as I said, the first thing I want to do is go in and delete that first line when I go in and look at my stats for timing that’s gonna kind of skew things a little bit so I want to make sure that doesn’t happen. So I just went in and I said okay VI the file delete that first line we save it and now we’re ready to go in and do an analysis.

Now, there’s a couple of ways you can go about doing this, you can import it into a spreadsheet you can play around with it if you want to. I like to keep it on the command line as much as possible, just because that means I can go through and I can actually do an analysis and then save the data raw without it taking any manual entry. You know, in other words it’s really easy to go in and kind of automate a lot of this stuff.

So what we’re looking at first is we’re looking for this just oh we’re looking at the statistics based on the session size. So we’ve got a minimum session size of 89, we’ve got a maximum session size of 290, we’ve got a mean which is behind my camera, eighty-nine point nine eighty-nine point eight okay. So the first thing that grabs me here, is my mean is very close to my minimum value, what does that mean? Well, we had a hundred and eight thousand data points here, that means that an overwhelming majority of those hundred and eight thousand data points are closer to that minimum size value. Now stop and think about how a beacon normally works.

A beacon is calling home saying “Hey do you have anything for me to do?” “No go back to sleep.” “Do you have anything for me to do?” “No go back to sleep.”

Okay, that tells me that, if my mean is close to that if a majority of the packets are close to that that couldn’t very well be a beacon signal that I need to go into look at and notice my standard deviation is pretty small, which means that my cluster around that mean is pretty small as well. Which again, leads me to feel like yeah, at least based on size I think I’m looking at a beacon here. Now my max size is 290, so something was much larger than that. What does that tell me? Well, if I’m seeing a min size of 89 and I’m seeing a majority of the time my mean I’m around that 89 size but I have a 290 in there, that’s kind of an oddness into the data set, that tells me that at least once this beacon may have been activated over the previous day. I don’t know where or when or how or anything else yet I’d have to go back into the pcap to go digging for that to find it, but at least now I know there’s something interesting there based on size.

Which, by the way, when you look at a lot of the tools that are out there that go looking for beacons, size tends to get ignored. You know, we were geeks we know this you don’t ignore size. Timing seems to be the thing that everybody keys in on and yeah, timing is important but you got to look at the size of the whole thing as well.

Suppose speaking of timing, yeah so I apologize my example here, I went in and should have deleted off the zero when I didn’t, so my minimum time gap is some small amount less than zero but notice my time gap goes up as high as 861. So it’s pretty high. I’ve got a mean that’s close to my min which again tells me, yeah okay this could have some beacon-like indicators and my standard deviation is small, which again is another indicator that I’m probably looking at a beacon. So we talked, we kind of analyzed size to say yeah because of all of these data points it looks like we got a beacon when we look at timing it’s not quite as clear but it’s still leaning towards that. Yeah I think we got a beacon we got to pay attention to here.

Questions at one point what’s in the beacon test file again? Let me back that up just a second it is this information so this file is that what I created with tshark to go through and just say, give me the IP addresses but most importantly give me the size, give me the timing. That’s what she said.

Yes of course, and let’s see was there any other questions in here I should jump on. Johnny Jason now the one that we just had pop up we have wouldn’t it wouldn’t using d-o-t or do H means one cannot do any situ detection on an external device DNI traffic will be encrypted hence the detection device would be useless. Well, so encrypted traffic doesn’t it doesn’t really matter you know so when we’re going in and we’re doing this analysis based on size and based on timing notice we don’t care if the payloads clear-text obfuscated or encrypted we’re simply doing a size based analysis on it. So you know again encrypt the packet stream it doesn’t really matter and this is kind of nice because some folks some sites actually have privacy concerns to worry about where if you’re looking at other people’s data in their packets you know that could be problematic. You can actually just go through extract metadata and do all this type of analysis never look at the payload and stay clear of those privacy concerns and we honestly stayed away from looking at the payload with RITA just because there is so much encryption.

Gary asked a question; said what about standard deviation how small is small? Chris, correct me if I’m wrong but, the actual size of the deviation doesn’t matter as much as what is the disbursement if it’s a flat bell curve or a balanced bell curve. That’s more interesting to us than how wide that distribution actually is. Absolutely and quite honestly there might be ways to pull that out of our that I have not found yet and if anybody has, oh my god please drop me a tweet because I’ve just like I’ve played around with this tool a little bit but it there’s a whole lot more I know I could learn so there may be a way to kind of figure out that dispersion a little bit clearer. You can graph it and that makes life easier and we’ve got a tool to graph it and I’ll show it later but to be able to do it on the command line, yeah, that would be kind of nice.

Yep, and we’ve had a bunch of questions about the encryption and once again the encryption doesn’t matter because we’re looking at the payload. We’re looking at the payload size we’re looking at the disbursements of the connection intervals and even the payload sizes as well, so just basically get the hell out of trying to look at the pay because we can encrypt it we can also use steganography and hide it in things like you state parameters so the further you get away from actually trying to analyze and make sense of payloads the more sane your life is gonna be. Yeah because again it would still yeah if I’m again if I’m hiding it inside of something the timing the size that’s still something I can go in and do an analysis on.

So cool, so hey, we had a way to detect a beacon so just script this and we’re done right? Well, maybe this scales nicely for your home network – not so much when you’re dealing with hundreds of millions a day, because again, we did we just went through one IP peer set, one internal system talking to one external host and that was it. We would need to do a similar analysis for every other session that took place across our perimeter over the previous day. Oh my god, that’s a lot of work. So are there ways we can go through and kind of simplify this?

Well yeah, that’s what RITA was for. So, one of the reasons we came up with RITA was to be able to do this type of analysis and actually have it be something that’s scaled. So, this is an open source tool that was created on the BHIS side, It’s now maintained on the ACM side and it’s specifically designed to go in and identify command and control channels specifically beaconing which will go in and we’ll take a look at now. So, RITA does not ingest pcap files directly you’ve got to give its metadata. So the easiest way to do that is just use Bro, and Bro is another awesome awesome open source tool so you can just simply take a pcap file or you can just run it and have it listen on the wire. Ingest that into Bro once you have Bro logs, now just go in and put them into RITA. And once RITA has them, RITA can go in and do an analysis on them.

And this looks like it’s not doing much but hey this just I did this just answered all those questions we had about the mean and the standard deviation and everything else we’re doing. All that math in the backend and more importantly people who are smarter than with math and I am doing this in the math on the back end. So this very first value that you see here in the output this is how likely is this a beacon. You know, because it’s not really a yes or no question, because it’s a gray area so they do this on a scale of 0 to 1 so if it’s a zero nope, definitely not a beacon. If it’s a 1 yes it’s definitely a beacon most of the stuff comes through it like you know point 6 something a point 8 something this one’s come comes in right at one so this is something that we two looked at and said oh no this is absolutely a beacon and you can see some of the values we’ve been talking about like the number of packets that took place the size that was involved it shows up in this output as well.

And we also have AI-Hunter, which is a commercial tool. It’s cheap, but it is a pay for use that allows you to go through and do a similar type of analysis. And you know, really what it comes down to is; it’ll ingest Bro logs as well but there’s a lot more visualization with this, so not only looking at “Is it a beacon yes or no?” but, “Is it actually a threat?” We’ll go through and we’ll gauge that, and we also try to go through and give you some good visualization tools to figure out if it’s a beacon. You know, this is a 24 hour period of time here… if I can draw a flat red line like I did here across all of my 1-hour quantities of traffic that took place, yeah that’s definitely a beacon. I need to go in and pay attention to we also mentioned that when we were in looking at the our stats.

“Hey maybe this backdoor was activated?” That’s something AI-Hunter’s designed for – here’s my heartbeat, here’s my little signal telling me yeah somebody activated it and transferred 300 bytes. So again, we don’t look at payload so we can’t tell you what it is, but 300 bytes we’re running about 90 off of a heartbeat anyway so there’s maybe about 250 bytes transferred here. That to me sounds like a directory listing or maybe a listing of what’s running a process where processes were running in memory. It’s definitely not “hey here’s a new toolkit I’m uploading” or “hey let me grab all the Excel spreadsheets”, so we’re still at a point where somebody’s in a discovery mode on the system they compromised on you. If we can get them off now, forensics becomes a whole lot easier. Okay, so that is.

dnscat2 – but let’s kind of play around with this one a little bit. Let’s say someone goes in and says okay I want to try and make dnscat2, it won’t be as functional it won’t be as easy to use but it’ll be harder to detect we’re going to go in vary the timing during this, we’re only going to have it run during normal business hours. So again, I might want to inject a command to the system that I’ve compromised and if it’s not business hours yet, it isn’t going to come back. Not like dnscat2 which responds very quickly when you say I want to execute this command on the remote system. Further, it may, you know, take the command and wait awhile before it actually gives a response back. So this was just, hey can I can I keep dnscat2 in a functional state but make it actually harder to go in and detect?

So this will be the second file that you get as part of the tar archive we’re going to give you at the end of this. A suspect DNS pcap, so what this does is it’s very similar traffic, you know, as you can see my size is a little different, the timing at least initially looks about the same although it is starting to widen up a little bit more. That’s actually the time dispersion, gets wider and wider the longer it runs, but it’s the same type of thing we did before. I’m taking a pcap file, I’m saying what fields in each packet are interesting to me. So, it’s destination IP most importantly size and timing and now I’m going to redirect that to another file so I can go in and I can use my file manipulation tools to go in and do an analysis. So like before, the first thing I go in and do is look to see okay, what’s in here what do I have for traffic. And this time I can see it’s all UDP traffic and I’ve got, you know, 200 2074 packets sitting in there.

Cool, so I’ve got something I can go in and I can play with. To go in and look at just this one particular channel. So again, running this through R in order to go through and do a statistical analysis. My min, my max, fairly close together size wise but also notice my mean. So my mean on size is pretty close to my minimum but look at my standard deviation. So this says, yeah, my average is pretty close to the heartbeat but when I kind of plot out how everything kind of jumps around that mean, it’s a much wider dispersion that it was before. It’s not huge, but this is one of those gray area ones. This is one of those instances where you’d look at this and say, well it doesn’t really look like a beacon but it doesn’t not look like a beacon. I’m not sure what to make of that and then you’d go look at the timing and here’s what the timing looks like.

So I’ve got a minimum time gap that’s very small. I’ve got a maximum time gap that’s much much larger. I’ve got a mean that’s kind of close to the minimum but not really. You know, our R minimum is 50 milliseconds and this is saying almost two and a half seconds is my average or is my mean. It’s not that close and you know again my standard deviation kind of says it’s all over the place. So when I try and do an analysis on the timing, you know, if so when I look at the analysis on size I said maybe-maybe-not. I’m not too sure when I go in I look at this, this wouldn’t convince me even further. Yeah this is probably not a beacon. I’ve got other stuff to look at that’s even closer. It would be really easy going in and doing this type of an analysis to look at this and say yeah, this isn’t something I need to worry about.

Cool, okay so we’re at the command line. We’re not sure. Let’s pull it in RITA. This is what RITA was designed to do. So let’s import it into RITA, and RITA looks at it and says that’s not a beacon. So yes, I purposely built this to defeat our own tool. I know that sounds weird but I wanted to make a point of this. So yeah, RITA didn’t detect it as a beacon, it didn’t figure out it was backdoor so are we out of luck. Is there anything else we can try and do here? Well yeah, if we can’t detect it as a beacon… remember back at the beginning of the webcast I was kind of talking about there’s oddities we can go in and looking for? You know, things like hey, you’re doing an MX query against a hostname or a fully qualified domain name instead of just a domain name. That’s kind of weird, we may be able to go in and kind of look for those types of things.

One of the biggest ones we can key in on is how many fully qualified domain names are being resolved as part of that remote domain. And, give me a second to kind of talk you through this one, because that this is kind of a key point with this stuff because when you look at you know, we’re talking about your dnscat2 and we said it uses C names, it uses text files that uses MX records, but in the wild we’ve seen ipv6 quad-a queries being used for command control. Meaning that, what comes back looks like an IP version 6 IP address and then when you look, you see no way no one’s been allocated that yet or it’s a private address that’s something that we’d never get handed out and it turns out, oh no wait, that’s command and control. We’ve seen people doing the query against DNS keys you know so DKIM they’re going in and saying, hey I want the keys to be able to authenticate that domain and what comes back is invalid keys. What it actually is is again a command and control channel.

So we were talking about cname txt and that type of thing but, oh my god, there’s just so many ways to skin that same cat here with DNS. You know, as I said, you know domain name keys I could use that ipv4 ipv6 addresses. I can use that so how do we try and find a methodology that will let us catch everything consistently? Well, we need to find the weakness in using this as a command and control channel and the weakness is DNS caching. The weakness is, as a compromised system I need to make sure I’m always talking to the command control server. So I’m going to take a DNS query, I’m going to send it to the local resolver and it’s supposed to forward that request on. Well, I got to make sure that resolver doesn’t catch it. Now one of the things I could try and do as the bad guy, is I could say, okay when you did a look-up on this fully qualified domain name I’m gonna set the TTL to that to one second. You know, some small period of time so that the local resolver knows: don’t cache this information.

Well, here’s the problem with that: DNS software ignores those settings for the most part, at least the minimums and the maximums. You can go in and you can say you know page this out in one second and the local resolver will say, yeah the admin configured me to remember everything from minimum of 5-10 minutes. So even though the TTL comes in set to one second, it’ll reset it to a five-minute period of time. Okay now I can’t talk to my command and control server anymore, now I’m only talking to the local resolver. I’m broken. You know, the other thing you can do is, you can say hey remember that record for, you know, 30 days. And again, the local DNS server might say, no no my admin configured me not to remember anything for that long. Not supposed to remember anything more than I think Microsoft’s default is like 24 hours. So you know again, so even though you can set TTLs, local resolvers can override those.

So how do you make sure the resolver can’t break your C&C? Well, the only way to do that is to change the lookup you’re doing every single time. Meaning that, I’m looking up a different fully qualified domain name each time. Well, this creates an interesting situation because think about how many fully qualified domain names will normally be advertised as being part of a specific domain. You know, they could think about like your own environment and you can probably like, count on two hands maybe just one how many hosts that you advertised. People on the Internet right, you’ve got a web serve,r you’ve got a couple of DNS servers, you’ve got a mail server, maybe a customer portal and probably not much more than that. So that 10 to 12 range that’s it.

And then you get into the domains that everybody’s heard of; Akamai, Amazon, Google. They offer services on the internet, they’re huge, everybody knows who they are. And they’re probably in like the five to six hundred range if you are using them for their public cloud services. It might be up in like the 800 range depending upon how many VMs you’re running at any given time, but again, certainly less than a thousand. So anytime we see it, and again this is not how many do they have in their records per se, it’s how many of these fully qualified domain names find it necessary to go query within a 24 hour period of time. So again to say anything more than a thousand is suspicious, yeah that’s gonna be suspicious. That’s something I need to be concerned about.

So could we leverage that to try and figure out when somebody’s gone in and created a command-and-control channel? So I’m back at tshark again, yay tshark! And I’m going in and I’m saying, all right I’m doing the same output I did before although quite honestly I don’t really need it here. I just kept it in the slide for consistency. The important fields are these two new ones I’ve added, DNS query type and DNS query name because that allows me to go in and say hey, you guessed it… the query type as well as what was the fully qualified domain name that was actually being queried. So I noticed, we’ll look at a dnscat2, remember I said it mixes things up. I’ve got a cname here, I’ve got a text here another text another C name and an MX record, so it’s revolving through a bunch of different things that may be something that’s worth going in and taking a look at. But I’m gonna take this output now, I’m gonna redirect this out to a file and once I get it out to a file I can go through and I can start manipulating with it now.

Just because you can do a thing, doesn’t mean you should do a thing, and I do that all the time.

So, we were talking about Bill Stearns earlier. One of the things that’s awesome about Bill is he is the nicest person you will ever meet in your life. And Bill looks at some of the stuff on the command line I do and he’s really good at “oh wow Chris that’s awesome, yeah that’s really creative, that’s cool, here’s how to do it in ten characters with awk, it’s a real simple way to skin that same same cat, so here’s a lollipop for your efforts.” Yes, exactly, exactly, I get a lot of lollipops from Bill. Actually yeah, it’s a running gag I used to have a whiteboard that identified all the things I’ve been able to teach Bill about Linux since I met him. I met Bill in 1998, I think that number’s up to six. Just a very smart guy.

But, so I redirect all that output into this file let me kind of walk you through the individual commands here. So first I’m running cut and I’m telling cut my comma is my delimiter for each field, so as you can see here anytime I’ve got a comma changing from like the source IP to the destination IP and so on from there. So anytime you see a comma, that’s gonna be you know, a delimiter on a field and I want you to go after field number eight okay. What was field eight? Well if we count these through field eight, is this fully qualified domain name. So it’s this really long convoluted “I can definitely tell a person didn’t create that host name” followed by the domain name itself. Now this is where things are going to get a little bit weird because what I want to do is I want to go in and I want to look at anything that’s associated with honestimnotevil.com. So, I could go in and now say, okay make period my delimiter, but how do I make sure I only grab honestimnotevil.com without catching the fully qualified domain name in the front especially since there could be multiple subdomains?

So that’s where rev comes in. So I go in and I kind of sort this out and I run it through rev. So what rev is gonna do is it’s gonna take all that data and just change it to MOC period L IVs blah blah blah all the way out to the end now. I said okay, cut, this time the delimiter is the period and you guessed it now that I have the domain name written backwards. I just say, grab field 1 and to switch it back again so I can actually read it as a human sort it run it through unique dash C and give me a count on what you say so what this is. So what all of this is telling me is that in this period of time when we did this analysis, which I think this runs over about two and a half hours, was all we were able to get. Again, this thing had office hours, I had two thousand and seventy four fully qualified domain names that appeared within this domain. Okay, it’s not Amazon, it’s not Google, gee that’s really suspicious. So do I know for certain what’s going on here? Well no, but I definitely know this something weird taking place here and I need to pay attention.

So Chris, we had Shawn asked, wait a minute, why reverse it again? So the reason to reverse, if you can read it backwards you’re fine. The reason we’re reversing it back again is so that it’s actually still legible. So if you think about if I didn’t have this in here if I didn’t have this excuse me this one here to reverse it back again what I’m going to end up with is a line that says MOC period Li ve at T oh and blah blah blah all the way down to the numeric value which the numeric value will also be backwards too. We don’t want to have to read that, so I’m reversing it so that my COM comes first then my domain name and then any subdomains and then the host that’s associated with it. So now I can go in and say just grab me the first two. Now a qualifier with this if mixed into this data was another domain that was honestimnotevil.com.uk? Okay, this kind of breaks down so you may have to do multiple passes to kind of weed this thing out correctly. If you want to go through and try and do it this way, but I at least wanted to find some method to give you to be able to do this in the command line just with the raw data. This goes through and this gives it for you make it send Sean okay. I think we’re cool, all right, so let’s yeah, so you could do that Sean. So Sean saying, hey what if you cut and you went in and you went looking for fields two through three you could do that you could ignore calm that would still give you the kind of unique stuff even if there’s a you know country identifier in there as well but it does make it much much harder when sometimes you want to match on the first two field sometimes you want to match on like the first three fields.It definitely takes a little bit more to kind of sort through this so I made it look the only reason I brought that up is because I do this and it kind of seems like hey, and look this is easy and it is easy for Mike and data in the real world.

There’s a couple of bumps you’re gonna run into you’ll need to kind of work through, and that’s kind of why I wanted to bring that up. But the easier way to do this, run it through RITA again. RITA is open source, anybody can download this and use it. So we just take it as Bro data, pump it into RITA which we had already done and remember we didn’t detect this as a beacon. But RITA did look at this… this time instead of saying show me beacons, I’m saying show exploded DNS. RITA is saying the this 274 fully qualified domain names associated with that domain. Hmm… that might be interesting. And then RITA goes on to list them all out for me, so I can go in and see them if I need to.

So you know, again, RITA might be a good choice here and I can pull it out of AI-Hunter as well. So, the same type of data, just a different visualization you can go through and use.

So what have we learned? Well, we’ve learned that back doors like dnscat2 can have beacon characteristics, but sometimes they might not, so we need to look for beacons but we need to have other things to go looking for to be able to catch this stuff as well. And it can’t be patterns you know, it can’t be. I mean I love snort, you know Marty awesome guy, I have you snort forever. You know the IDS definitely has its place, but as soon as you all know, as soon as somebody changes the pattern you start missing things. So how do we do this in a way that if they mix up the pattern we can catch it with DNS anyway? So long as we’re looking for those fully qualified domain names and counting those up, that’s a pretty consistent way to comb through and catch that back to our regardless of the type of query that they’re going through and doing so you know it’s kind of that, you know, wax-on wax-off grasshopper type of thing you know leverage their weakness against them that’s really what we’re doing here.

So, to kind of give you a wrap-up, so drop a tweet to us if you want to see this as a series. As we mentioned at the beginning, I was originally going to do three different backdoors I got up to 80 slides and I was not done and said okay, I don’t think I can cover 80 slides in an hour and still leave time for Q&A. So we need to trim this back a little bit. I’m happy to try to turn this into a series folks want to so we can go through different command and control channels. if you like this and you tell us, hey yeah I like that idea, please also tell us if you there’s a command and control channel you want to cover, dump that out there as well.

Showed you a couple things AI-Hunter. John usually saves that till the end. I kind of mixed it in, I apologize for the little sneaky still stuff as we go through, but if you want to see a demo drop “demo” in the chat.

Movie to help you out, but more importantly RITA open source. Here’s a link to get to it, so if you go to that link you can grab a free version of RITA and you can go through and use the pcaps I used in this webcast. There’s the link to go through and get those so webcast – files – downloads. Go to our website, go to that link, you’ll be able to go in and grab the files from this and then you can go back through the slides which will make available in a couple of days and you can go through and repeat all of this stuff.

By the way, if you’re watching this on YouTube as opposed to watching it live, that’s okay. Don’t feel like you missed anything you know, tweet it out, throw it in the comments, do YouTube comments, we check that area as well.