Malware of the Day – C2 over NTP (goMESA)
What is Malware of the Day?
Lab Setup
Malware: Custom Go-based C2 (goMESA)
MITRE Tactics: TA0011 Command and Control, T1071 Application Layer Protocol, T1573 Encrypted Channel, TA0003 Persistence
TA0005 Defense Evasion, TA0002 Execution
Traffic Type: NTP, TLS over HTTP/1.1
Connection Type: UDP + TCP
C2 Platform: Custom Go-based C2 (goMESA)
Origin of Sample: Active Countermeasures Threat Hunting Lab
Host Payload Delivery Method: *.exe binary, shellcode
Target Host: 192.168.2.115 (Windows 11 x64)
Adversary Host: 64.23.212.29 (Ubuntu)
Beacon Delay: 60s
Jitter: 0s
Malware of the Day Mission
To identify and share examples of post-compromise network activity in order to better detect and respond to potential network threats. Specifically we are looking for Command and Control (C2) communication channels used by attackers to obtain intelligence, issue commands, and exfiltrate data through a compromised host or hosts.
Preface: The Threat Hunting Mindset
As threat hunters, we approach our work with the mindset of a detective, maintaining an open and unbiased perspective to ensure that all potential evidence is considered and interpreted objectively. Our investigation begins when we discover a piece of evidence that suggests something might be amiss. From this point, we employ a variety of supportive tools to either refute or strengthen our case.
The process of threat hunting involves layering evidence, gradually increasing the probability of a compromise as we gather different, independently verified pieces of information. Each new piece of evidence adds to the overall picture, allowing us to build a stronger case. Our ultimate goal is to construct a case that is sufficiently solid to inform the Incident Response team of a probable compromise.
Throughout the investigation, it is crucial to remember the golden rule of threat hunting: never cross the active-passive line. This means that we must use all the tools and techniques at our disposal in a manner that does not alert the intruder to the fact that they are being investigated. By maintaining a passive approach, we can gather the necessary evidence without compromising the investigation.
By adhering to this mindset and methodology, we can effectively investigate potential compromises, providing valuable insights and enabling our organization to respond to threats in an effective manner.
Background
Since organizations routinely keep a close watch on standard protocols like HTTP/S and DNS for signs of Command and Control (C2) traffic, attackers often need to get creative to avoid detection. Instead of relying solely on these monitored channels, they might turn to less common methods.
For instance, they could abuse utility protocols like ICMP, hijack legitimate cloud services to relay commands, or even embed their C2 communications within unassuming data streams like email content or social media activity. Leveraging unconventional channels allows attackers to establish footholds that are more likely to survive initial incident response efforts, leading to recalcitrant footholds that can be challenging to eradicate.
Let’s consider the Network Time Protocol (NTP), a protocol that plays the crucial role of synchronizing time across devices on a network. Its importance and widespread use, however, also creates an opportunity for attackers…
Why NTP Makes an Appealing Covert Channel
The Network Time Protocol (NTP), operating typically over UDP port 123, might seem like a simple utility for keeping computer clocks synchronized, but its fundamental role in network operations creates unique opportunities for covert communication.
Because accurate timekeeping is vital for so many functions—from logging events correctly to ensuring authentication mechanisms work—NTP traffic is often allowed through perimeter firewalls with significantly less inspection than protocols like HTTP/S. This reliance means security postures are frequently more relaxed towards NTP, potentially giving attackers an open lane for C2 traffic that bypasses stricter egress filtering.
Beyond just slipping through defenses, the protocol’s structure lends itself to misuse. While NTP packets have a defined format, the relative simplicity, combined with the possibility of manipulating specific fields (like extension fields or even parts of timestamp fields), allows attackers to embed small custom data payloads within otherwise normal-looking time updates.
To complete the disguise, an attacker’s NTP server used for C2 can often be set up to also respond with valid time information, making the malicious traffic blend seamlessly with legitimate NTP activity and harder to detect by both automated systems and security analysts.
This combination of permitted passage, potential for data hiding, and plausible deniability makes NTP an attractive channel for stealthy C2 operations.
Scenario and Setup
In a recent security incident at Jane Doe Inc. the team successfully identified a compromise using a standard HTTPS channel for Command and Control (C2). The team was ultimately able to contain the damage, and were confident they’d fully removed the attacker from their network. Case closed, or so they thought.
But then, unsettling signs began to reappear. Days after the “all clear,” faint echoes of the intrusion started showing up – strange DNS lookups hitting odd domains, momentary network connections with no clear purpose, and scattered alerts from security tools that didn’t quite match any ongoing legitimate activity.
These weren’t loud alarms, but subtle, persistent whispers suggesting the initial cleanup might not have been enough. The team faced a nagging suspicion: had the attacker left behind a stealthier backdoor, a persistence mechanism designed to survive the usual remediation efforts?
Indeed, the initial compromise allowed the attacker to deploy an implant for a custom Go-based C2 framework known as goMESA onto the target Windows 11 workstation (192.168.2.115).
Once executed, the goMESA agent did not bind directly to UDP port 123, which would conflict with the legitimate Windows Time service. Instead, it utilizes packet sniffing capabilities (leveraging libraries like libpcap/gopacket) to passively monitor all NTP traffic on the host. When it detects NTP packets destined for legitimate external servers, it intercepts them and crafts its own NTP-like packets containing covert C2 data. These packets were sent to the attacker’s C2 server (64.23.212.29).
Critically, this server is configured to also function as a legitimate NTP server. It responds to standard time requests, making casual observation of the traffic appear normal. However, it recognizes the specially crafted packets from the agent, extracts the embedded C2 data, and can embed its own commands within the NTP responses sent back to the agent.
It turned out it was this NTP channel that functioned as the hidden persistence mechanism. After the primary channel over HTTPS was found and eliminated, the attacker utilized the goMESA implant’s NTP channel capabilities, allowing them to download and execute shellcode to resurrect their primary HTTPS channel.
Faced with the puzzling signs of continued compromise, the Jane Doe Inc. security team contacted us to assist with an in-depth investigation. Given the context we decided to broaden our threat hunt beyond the usual suspects. We decided to also turn our attention towards less monitored protocols, since we suspect that a hidden channel might be the root cause of the persistent attacker’s presence.
RITA
Almost immediately upon analyzing the connections in RITA we notice several peculiarities regarding a specific target host – see Image 1 below.
Image 1. RITA results showing a peculiar connection to 64.23.212.29.
First, we notice what appears to be an unexpectedly high volume of NTP connections – 1539 sessions over the analyzed period of 24 hours. This volume seemed excessive for standard time synchronization from a single client.
Adding to the intrigue, RITA also indicates that traffic between the same internal host and the exact same external IP address occurred over TCP port 443 (HTTPS/TLS). The fact that the same target host was involved with both a NTP and HTTPS connection felt inconsistent to our team – NTP services are typically kept separate from general web hosting.
Establishing a Baseline
Intrigued by this connection and the unusual choice of NTP as a potential vector for malicious communication, we decided that establishing a baseline was crucial. We created a trace file on a brand new Windows instance (same version and build) to determine what “normal” NTP traffic should look like on a typical Windows host. This then allowed us to compare this known-good baseline against our target during our investigation, giving us better insight into potentially suspicious deviations – see Image 2 below.
Image 2. RITA results for a known-good NTP connection.
The known-good trace file only contained 47 NTP connections during the same timeframe and exhibited no corresponding HTTPS traffic to its NTP server. This difference significantly raised our suspicion level regarding the communications with 64.23.212.29, prompting us to look deeper into other aspects of the traffic’s characteristics.
AC-Hunter
Transitioning the investigation into AC-Hunter allows us to visualize and analyze the timing and data patterns within the suspicious connections more effectively. When examining the NTP sessions to 64.23.212.29, a clear pattern emerged – see Image 3 below.
Image 3. The AC-Hunter inter-connection duration histogram for 64.23.212.29.
The majority of connections were spaced almost exactly 60 seconds apart, indicating a fixed, programmed beaconing interval. This directly contradicted the behavior observed in the known-good NTP traffic (Image 4 below), which displayed the standard exponential backoff mechanism outlined in RFC 5905.
Image 4. The AC-Hunter inter-connection duration histogram for our known-good NTP connection.
As we can see the legitimate NTP client increased its polling interval over time (roughly 64s, 128s, 256s, 512s, 1024s, before eventually settling around 2048s) to conserve resources, whereas the suspect traffic adhered rigidly to its 60-second schedule.
Further analysis within AC-Hunter focused on connection sizes. All of the legitimate NTP traffic packets were 76 bytes, matching the expected size for standard NTPv4 exchanges – Image 5 below.
Image 5. The AC-Hunter payload size histogram for our known-good NTP connection.
The suspicious NTP traffic largely mimicked this size, perhaps as an intentional effort by the malware authors to blend in. However, AC-Hunter flagged two distinct outliers within the suspicious stream: one connection transferring 1339 bytes and another transferring 1627 bytes – Image 6 below.
Image 6. The AC-Hunter payload size histogram for our suspicious connection contains 2 outliers (see red arrows).
These significantly larger transfers are highly anomalous for NTP, suggesting they might have been involved in the HTTPS connections that were also involved in communication with this host. These larger transfers could potentially have been misused for C2 data transfer, possibly downloading larger commands or even payloads like shellcode, ideally suited to create callbacks to new C2 channels.
DNS and OSINT Investigation: Contextualizing the Server
With behavioral evidence mounting, we then sought external context on the suspicious IP address, 64.23.212.29. A WHOIS query revealed the IP address belonged to infrastructure managed by Digital Ocean, a well-known cloud hosting provider – see Image 7 below.
Image 7. WHOIS information for 64.23.212.29.
While legitimate services operate from cloud platforms, it’s less typical for established public NTP pool servers to reside there compared to infrastructure run by dedicated organizations or institutions. This finding added a subtle layer of suspicion.
Using AlienVault, we found records of a previous automated scan against port 443 (HTTPS) on this IP, which returned a generic HTML page titled “System Update Service” – see Image 8 below. Though this message appears to be a legitimate server message, such placeholder pages are also a common tactic used by C2 operators to make their infrastructure seem innocuous if probed by automated scanners.
Image 8. An archived HTTP scan by AlienVault shows an innocuous-looking placeholder message on the web server, which could very well be part of the attacker’s ruse.
Let’s now contrast these findings with the investigation of the known-good NTP server’s IP address – Image 9 below.
Image 9. WHOIS information for our known-good NTP connection.
A reverse DNS lookup resolved to ‘prod-ntp-5.ntp4.ps5.canonical.com’, indicating infrastructure belonging to Canonical, a provider known for running public services like NTP. The reputable source stood in stark contrast to the murky nature of the suspicious IP, further reinforcing the likelihood that 64.23.212.29 was not a legitimate public NTP server.
Wireshark Deep Dive
To get the most granular view, we opened the original trace file in Wireshark. We first looked at the known-good trace file to create a clear baseline of what we should expect to see – Image 10 below.
Image 10. An overview of the individual packet flow of the known-good NTP server connection.
As expected from our knowledge of how NTP functions, we observed a clean one-to-one exchange of client request and server response packets. The timing intervals between these pairs precisely matched the exponential backoff pattern we saw in AC-Hunter (64s, 128s, 256s, etc.). In the info column on the far-right we can also observe that Wireshark correctly identified and parsed these packets as legitimate NTP.
Now let’s turn our attention to the suspicious traffic to 64.23.212.29, where the picture changes dramatically – see Image 11 below.
Image 11. An overview of the individual packet flow for our suspected malicious connection.
The standard one-to-one client/server exchange is absent. Instead, multiple client packets were often sent before a response was received. More significantly, Wireshark flagged nearly every packet associated with this communication as “Malformed Packet ” – see Image 12 below.
Image 12. The Wireshark “info” column reported nearly every packet involved in this communication as being malformed.
This indicates that the packet structure deviated from the official NTP protocol specification, a strong sign that non-standard data was being embedded within the packets, consistent with a covert C2 channel tunneling over NTP.
Examining the contents of these malformed packets, despite portions appearing garbled or encrypted, revealed specific ASCII keywords not found in legitimate NTP. The string PING was present in the majority of packets, highly suggestive of the C2 agent’s check-in or heartbeat mechanism – see Image 13 below.
Image 13. Numerous packets contained what appears to be the keyword PING.
A handful of server-to-client packets also contained the string COMD, possibly indicating a “command” – see Image 14 below. Though we are squarely in the realm of speculation at this point, it could be that this framework uses 4-letter keywords (PING, COMD etc.) to indicate to the C2 agent the type of packet, allowing it to interpret subsequent data correctly.
Image 14. A number of packets contained what appears to be the keyword COMD.
Finally, we then turn our attention to analyzing both the NTP and HTTPS connections in tandem. While the HTTPS payloads itself were encrypted via TLS, its timing was revealing – see Image 15 below.
Image 15. The transition from NTP to HTTPS connection were all marked with an unusual NTP packet.
Both HTTPS sessions were immediately preceded by a specific, larger-than-usual NTP packet (141 bytes) from the C2 server (purple rectangle). And looking inside of these packets revealed the presence of another 4-letter word – WCON – which, given the context, could possibly be a signal for initializing a “web connection” – see Image 16 below.
Image 16. Both “transition” packets contain the specific string WCON in their payloads.
Though this behaviour seems curious, an overall hypothesis emerged among the team: the attacker likely uses the stealthy, low-bandwidth NTP channel for persistent beaconing and basic commands, but switches to the reliable HTTPS channel when needing to transfer larger or more critical data, such as shellcode, where the unreliability of UDP could be problematic.
Though this is certainly speculative based on what we know thus far, the pieces fit together, suggesting that this NTP traffic is part of a cleverly disguised C2 channel enabling persistent access and the ability to transition to HTTPS when larger payloads are involved.
Endpoint Investigation
Although we usually validate our suspicions by investigating the endpoint process involved, the multiple layers of compounding network evidence was sufficient in this case. This gave us enough confidence to immediately alert the Response team about the compromise and likely source of hidden network persistence.
Conclusion
This Malware of the Day investigation presented a scenario where initial incident response actions failed to completely eradicate an adversary due to a hidden network persistence mechanism. By simulating an attacker leveraging the Network Time Protocol (NTP) for covert Command and Control after their primary HTTPS channel was removed, we uncovered the subtle network behaviors that can signal such elusive threats.
Our analysis demonstrates how a layered investigative approach can successfully unmask C2 channels, even those operating outside commonly monitored protocols, leading us to several key takeaways.
Firstly, attackers continue to exploit the path of least resistance, often hiding within protocols assumed to be benign or essential. NTP, vital for network operations and frequently allowed through firewalls with minimal inspection, proved to be an effective channel for maintaining persistence in this case.
Even after the primary HTTPS C2 channel was discovered and removed, the hidden NTP backdoor allowed the threat actor to retain access and initiate steps to regain full control. This underscores the need for defenders to look beyond the usual suspects (HTTP/S, DNS) and consider how even utility protocols can be abused for covert communication, especially when investigating persistent threats that seem to defy remediation efforts.
Secondly, detecting C2 within uncommon protocols often hinges on behavioral analysis rather than traditional signatures. While standard NTP traffic follows predictable patterns like exponential polling backoff, the malicious traffic exhibited stark deviations.
The rigid 60-second beaconing interval identified by AC-Hunter, the anomalous large packet sizes inconsistent with time synchronization, the high overall connection volume flagged by RITA, and the mixing of NTP and HTTPS services from a single, non-standard source IP were all crucial behavioral indicators.
Tools like RITA and AC-Hunter excel at extricating these deviations from normalcy, enabling us as hunters to spot suspicious activity even when the underlying protocol itself is unconventional.
Thirdly, the power of layered evidence in threat hunting cannot be overstated. No single finding in this investigation definitively proved malicious intent in isolation. The initial RITA findings raised suspicion, AC-Hunter revealed unnatural patterns, DNS info provided critical context about the server’s legitimacy, and finally, Wireshark delivered packet-level confirmation.
It was the cumulative weight of evidence across these different analytical perspectives that transformed initial suspicion into a high-confidence conclusion. This reinforces the core threat hunting principle: building a case piece by piece through methodical investigation using complementary tools.
This investigation serves as a reminder that attacker techniques are varied and continuously evolving. Relying solely on cleaning up the most visible signs of a compromise, like a primary C2 channel, may leave hidden backdoors open. True network resilience requires persistent vigilance and a willingness to hunt for anomalies in unexpected places, scrutinizing traffic patterns beyond the everyday norms.
Capture Files
PCAPs
Because… PCAPs, or it didn’t happen. 😊
The following PCAP files are packet captures taken from the same lab environment over a 24-hour time frame. The files were generated using Wireshark from the target host and include normal Windows OS traffic and normal network broadcast traffic. They have not been edited. The PCAPs are safe, standard PCAP files and do not include any actual malware.
goMESA C2 24 Hour Capture
goMESA24hr.pcapng
File Size: 201.1 MB
SHA-256 Hash: 449DDA61291C568684531259E0E0513205534461B4CB94042515D2E5BD91954F
Zeek Logs
If you are an AC-Hunter user, we are providing the 24-hour Zeek logs for you to import directly into AC-Hunter. The following Zeek Logs have been taken from the same lab environment over a 24-hour time frame and include normal Windows OS traffic and normal network broadcast traffic. They have not been edited. The Zeek logs are safe, standard log files and do not include any actual malware.
Importing Zeek logs into AC-Hunter example:
ssh into your AC-Hunter or RITA server and upload all the Zeek logs contained in the zip file below (all files that have the ‘.log’ extension) into a temporary directory on the server. In this example, we are uploading the Zeek logs into /tmp/zeek_gomesa/
Then run the following command:
For AC-Hunter v6.x and RITA v4.x and earlier:
rita import /tmp/zeek_gomesa/*.log gomesa
For RITA v5.x+:
rita import --logs=/tmp/zeek_gomesa/*.log --database=gomesa
You will now have a new database in the AC-Hunter UI/web interface or RITA CLI titled “gomesa” you can select and view.
goMESA C2 24 Hour Zeek Logs
zeek_gomesa.zip
Size: 800.3 KB
SHA256 Checksum: 346B107FE5AF79CF11D0722FCDDB7B104B49D60DCF733E6EFF67E5E85717E01E
Discussion
Want to talk about this or anything else concerning threat hunting? Want to share how good (or not so good) other detection tools were able to detect this sample?
You are welcome to join our Discord server titled “Threat Hunter Community” to discuss topics surrounding threat hunting. We invite you to join our server here.
A huge thanks to Bill Stearns, Keith Chew and Chris Brenton for allowing me the opportunity to contribute to this awesome initiative, as well as their guidance, helpful feedback, and for being a couple of stand-up gents.
Live long and prosper!
Faan has a profound love for the natural world, technology, design, and retro aesthetics. He is incredibly grateful to have discovered cybersecurity as a path relatively late in his life, and his main interests are threat hunting and post-exploitation custom tooling, in particular C2 frameworks and RATs.