An Attacker’s Guide to Evading Honeypots - Part 2

Introduction

In the first part of this blog series, we described the JA3 hash and reviewed the tools we use to randomize it. In this second part, we’ll show how we use JA3 randomization to enumerate honeypot IP addresses allowing us to avoid them. We’ll also describe how to determine which cloud providers host the honeypots and how to fingerprint a honeypot and then use that fingerprint to search for additional honeypots. In our upcoming part 3 we’ll describe how to analyze a honeynet and identify the specific vulnerabilities it flags, the vulnerability categories it prioritizes, and the services its honeypots mimic.

The information in this blog is a combination of our research and findings from multiple threat intelligence providers and it has been anonymized for security and courtesy. We present the research and findings as if they are for a single hypothetical company named ACME.

Initial Scan

As described in part 1, we began the enumeration process against ACME with aggressive scans against all AWS’ published IPv4 prefixes. We used httpx for our initial scan and tcpdump to capture and save the scan data to a pcap file for later processing.

# tcpdump captures the network traffic on the "eth0" interface with a destination port of 443.
# The traffic is saved to aws_discovered.pcap.
tcpdump port 443 -i eth0 -w aws_discovered.pcap

# httpx runs a scan against all the AWS IP blocks listed in aws_ipv4_prefixes.txt.
# The scan probes the path "/.env/" on ports 80 and 443 and uses random JA3 hashes (-tlsi).
# The output is stored in json to acme.httpx.
httpx -l aws_ipv4_prefixes.txt -path "/.env/" -json -o acme.httpx -p https:443,http:80 -tlsi

Our plan was to run tcpdump and the httpx scan, use ja3.py to extract the JA3 hash and target IP address pairs from the pcap file, find the IP address of our scanning device via ACME’s web UI, and then map the JA3 hashes associated with our device in ACME to the target IP addresses extracted from the pcap file. However, when the httpx scan finished after two days, our scanning device had not been flagged by ACME. We ran the scan a second time with the same result. We then decided to run a very noisy nmap port scan. Although we hoped this scan would be flagged by ACME, it wouldn’t have helped us enumerate ACME’s honeypots because it doesn’t use TLS and therefore wouldn’t have any associated JA3 hashes. Fortunately, after completion of the nmap scan, not only did our device show up in ACME’s web UI, but after 30 minutes our previous httpx scan appeared as well.

# nmap runs a scan against all ports (-p 1-65535) on the target IP addresses (-iL aws_ipv4_prefixes.txt).
# It then probes each open port to determine service/version info (-sVC).
# It then runs the vulners script (--script=vulners) which queries vulners.com to learn whether any known 
# vulnerabilities exist for the CPEs associated with the software identified while probing the open ports.
# The output is saved to loud_scan.json.
nmap -sVC -p 1-65535 -T5 --min-parallelism 2000 --script=vulners -v -oJ loud_scan.json -iL aws_ipv4_prefixes.txt

At this point, we were able to examine ACME’s findings for our scanning device and correlate the associated JA3 hashes with those extracted from the pcap file to confirm ACME’s honeypot IPs. We were disappointed to find ACME had only logged 200 of our 136,274,133 JA3 hashes (one for each AWS IP address). After further investigation, we found that the 200 JA3 hashes were associated with the last four hours of our httpx scan and we concluded that AMCE has a four-hour data cache. With our device now considered malicious by ACME we could have rerun the two-day httpx scan with the hope that ACME would record and retain all 136 million JA3 hashes. Instead, we opted to fingerprint a handful of the confirmed honeypots and search for additional candidates in Censys.

# This is an example mapping of a JA3 hash between the ja3.py output and the ACME data for our scanning device.
# The mapping tells us which destination_ip is the IP of an ACME honeypot.

# Output from ja3.py & pcap file with anonymized JA3 hash.
{
    "destination_ip": "<honeypot_ip>",
    "destination_port": 443,
    "ja3_digest": "**f6ebb4a04cfffffff2f54449b7f4cc32**",
    "source_ip": "<scanning_device_ip>",
    "source_port": 55555
}
# ACME API tool
{
  "message": "<scanning_device_ip>,**f6ebb4a04cfffffff2f54449b7f4cc32**"
}

Fingerprinting

Our goal with fingerprinting is to find a hash, or some other value, derived from a confirmed ACME honeypot that can be used in a Censys search to find other honeypots. Ideally, the fingerprint would be unique to ACME and all the search results would be ACME honeypots. Realistically, however, we were hoping to find a fingerprint that is common to many ACME honeypots but not very common elsewhere. Such a fingerprint would produce a reasonably low number of search results that would be feasible to scan with httpx or nuclei.

The JA3S was the first fingerprint we evaluated on the initial batch of 200 confirmed honeypots. The JA3S is the hash of the cipher configuration of the TLS Server Hello packet rather than the Client Hello packet. We hoped that ACME hadn’t introduced a lot of randomness in the TLS configurations of their honeypots so there would be a small number of common hashes. We also hoped the hashes wouldn’t correlate to standard TLS configurations for a common container or standard VPS configuration etc., otherwise we’d have too many search results. After running ja3s.py on the pcap of our initial scan we found there were only 12 JA3S hashes in use among the 200 confirmed honeypots.

# Extract JA3S hash data from the pcap of our intial httpx scan.
python ja3s.py --json aws_discovered_acme.pcap

# This is a list of anonymized JA3S hashes with distributions across the 200 confirmed honeypots.
92f38a57c239fa4395b345e873459bb3,1
a39d5687e34f95bc83a38756398594fd,1
3ca04875e2934a87d394cc53094f5830,2
aaa459d837f459c487aa49238e475b94,2
5ace5757348d39394dd8475b9a3e4847,3
10983b47a561e03ff98475ca6019f348,3
b59a28b7912eef3759d71300a930c350,4
591875103f985b71034987ee5203c89a,5
000139875610d349d28d6751d049a37f,10
c378543a58907d44309e74358f907b78,22
3254f35a34c53e425435bbb009a78098,46
b312345135a032734d775d7893b472ac,101
[+] Found a total of 12 JA3S hashes on 200 systems.

Unfortunately, the Censys search for the most common JA3S returned more than 12 million results indicating that it’s a very common TLS configuration. The results were similar for the remaining hashes.

We next evaluated the quality of the JARM and Banner hashes as fingerprints. The JARM is very similar to the JA3, but it not only captures the parameters used during the TLS handshake but also includes data like the values of the parameters. The Banner hash is simply the hash of the string returned in the body of an HTTP response.

We can generate both of these fingerprints with httpx.

# httpx runs a scan against the IP addresses for our confirmed honeypots (aws_discovered_acme.lst).
# The scan calculates the JARM (-jarm) and probes the path "/" on port 443 and calculates the sha256 hash 
# of the response (-hash sha256).
# The output is stored in json to acme-fingerprint.httpx.
httpx -l aws_discovered_acme.lst -path "/" -json -o acme-fingerprint.httpx -p https:443 -jarm -hash sha256

# Output of above httpx command for one host (with anonymized hashes)
{
  "hash": {
    "body_sha256": "305a39487f530948b572039487cd5029384a750293847502e834750c92834ff7", -> banner-hash
    "header_sha256": "fe234897d2345a60459f8759d834ea7593b48b7694dc85794ac85da7694e5ff6"
  },
  "port": "443",
  "url": "https://<honeypot_ip>:443",
  "input": "<honeypot_ip>",
  "method": "GET",
  "host": "<scanner_device_ip>",
  "path": "/",
  "jarm": "c349573456ff9873b76c7349a857a239457b62354d6d0495e867f5235cc46abd", -> jarm-hash
  "status_code": 200
}

The Censys search for both of these hashes returned significantly fewer results than the JA3S hash search. The table below shows the format of the three fingerprint searches and the number of results for each one.

Enumeration

Due to the relatively small number of results from the Censys Banner Hash search, we decided to run a scan against that batch of 2,334 devices. Recall earlier that the nmap scan queried vulners.com for the services it discovered and then printed the CVEs associated with any known vulnerabilities for those services. After the nmap scan we ran a handful of nuclei scans for those CVEs against a single ACME honeypot and noted which ones were flagged. At this stage, we performed a nuclei vulnerability scan for one of those CVEs against all 2,334 devices. This has two benefits. First, because we know ACME’s infrastructure flags scans for that specific CVE we improve our chances of being flagged by ACME’s honeypots versus the generic httpx probe we performed earlier. Second, we can replace both the httpx and nmap scans with a single nuclei scan. We couldn’t have done this earlier because we didn’t know which vulnerability scans AMCE would flag until after we’d performed the earlier nuclei scans.

The nuclei scan identified 607 of the 2,334 devices from the Censys search as ACME honeypots.

# nuclei scans the devices listed in censys_query_fingerprint_hosts.txt.
# It uses a CVE template to see if the devices are vulnerable to CVE-2020-15505 (-t .../CVE-2020-15505.yaml).
# JA3 hashes are randomized (-tlsi) and the output is saved to censys_query.txt.
nuclei -t http/cves/2020/CVE-2020-15505.yaml -l censys_query_fingerprint_hosts.txt -tlsi -o censys_query.txt

To enumerate more of ACME’s honeynet we iterated multiple times through the process of:

1. Fingerprint newly identified honeypots (using httpx and ja3s.py)
2. Search Censys for more candidate devices using the new fingerprints
3. Run a nuclei scan with one of the flagged CVEs on the latest batch of candidates
4. Correlate the results with ACME’s findings

The image below shows the locations of the honeypots we’ve enumerated so far.

The final step in the enumeration process was to determine which cloud services host the honeypots, and thus which networks they protect.

We wrote a cloud detection tool that identifies the cloud provider and region for a given IP address. In many cases, the correlation data comes directly from the cloud providers through published json files or APIs. In the cases where that isn’t possible, the data is collated from Project Discovery’s subfinder and bgp.he.net.

The images below show a portion of the output generated when we ran the tool on the 607 “Banner Hash” devices confirmed as ACME honeypots by the targeted nuclei scan. ACME appears to have a strong presence in AWS and COGENT, but little presence in GCP or Azure. It also looks like most of their honeypots are deployed in the northeast regions of GCP and AWS.

Conclusion

In this part of our blog series we’ve outlined our process for enumerating and fingerprinting honeypots. Most of the tools we’ve used are free and easy to use. We’ve shown that we can identify the IPs of ACME’s honeypots and the cloud services that host them (including their regions), and we’ve reviewed our approach to calculating fingerprints and creating Censys searches that use those fingerprints to find additional honeypots. At a high level, our method includes the following steps:

Part A: Confirm first batch of honeypots

1. Run httpx and nmap scans against all AWS IP blocks
2. Find our scanning device in ACME’s data, note the recorded JA3 hashes and correlate them with the httpx target list
3. Run a nuclei scan against one confirmed honeypot to get a list of CVE’s ACME flags

Part B: Enumerate more honeypots

1. Run httpx to get the banner hash for some of the newly identified honeypots
2. Search Censys for more candidates
3. Run a nuclei scan with one CVE (from part A step 3) on the candidates
4. Correlate the scan with ACME’s findings —> new confirmed honeypots
5. REPEAT

Read part 3 where we detail how to determine which CVEs ACME can detect and what services their honeypots mimic. With this information, pentesters and attackers can get a sense of the types of attacks that might go undetected.