How to detect DNS tunneling on a Linux server

Know what a DNS tunnel looks like

Capture the server's DNS traffic with tcpdump

Install tcpdump, and tshark for the analysis further down. On Debian and Ubuntu the tshark installer asks whether non-root users may capture: answer no.

sudo apt install tcpdump tshark

sudo dnf install tcpdump wireshark-cli

Record five minutes of DNS traffic, UDP and TCP, on every interface. With systemd-resolved, programs ask the local stub on 127.0.0.53 and the stub asks upstream: -i any sees both.

sudo timeout 300 tcpdump -i any -nn -Z root -w /root/dns.pcap 'port 53'

Then read it. First the share of each query type: a lot of TXT? or NULL? stands out.

sudo tcpdump -nn -r /root/dns.pcap 'dst port 53' 2>/dev/null \
  | grep -oE '[A-Z0-9]+\? [^ ]+' | awk '{print $1}' | sort | uniq -c | sort -rn

Then the domains with the most unique names under them. A tunnel puts one domain at the top with hundreds of names; normal traffic spreads over many domains with few names each. (The count groups by the last two labels, so example.co.uk shows as co.uk.)

sudo tcpdump -nn -r /root/dns.pcap 'dst port 53' 2>/dev/null \
  | grep -oE '[A-Z0-9]+\? [^ ]+' | awk '{print tolower($2)}' | sort -u \
  | awk -F. '{n = NF - 1; if (n >= 2) print $(n-1) "." $n}' | sort | uniq -c | sort -rn | head -20

And the names with a label of 30 characters or more:

sudo tcpdump -nn -r /root/dns.pcap 'dst port 53' 2>/dev/null \
  | grep -oE '[A-Z0-9]+\? [^ ]+' | awk '{print $2}' \
  | awk -F. '{for (i = 1; i < NF; i++) if (length($i) >= 30) {print; break}}' | sort -u | head -20

The same with tshark

tshark gives statistics by query type and the source and destination of each query, and finds unusually large answers, which tunnels use to carry data back:

sudo tshark -r /root/dns.pcap -q -z dns,tree
sudo tshark -r /root/dns.pcap -Y 'dns.flags.response == 0' -T fields \
  -e ip.src -e ip.dst -e dns.qry.type -e dns.qry.name | head -50
sudo tshark -r /root/dns.pcap -Y 'dns.flags.response == 1 && udp.length > 512' -T fields \
  -e dns.qry.name | sort | uniq -c | sort -rn | head

Check the resolver logs

First see which resolvers the server is configured to use. Queries to any other address are worth a look on their own.

cat /etc/resolv.conf
resolvectl status 2>/dev/null | grep -E 'DNS Servers|Current DNS'

systemd-resolved keeps no query log, but from systemd 252 (Debian 12, Ubuntu 24.04, RHEL 9 and later) it shows every lookup live. Leave it running for a few minutes and watch for long random names:

sudo resolvectl monitor

If the server runs its own resolver, turn on its query log. For unbound, then sudo systemctl restart unbound:

# /etc/unbound/unbound.conf (RHEL) or a file in /etc/unbound/unbound.conf.d/ (Debian, Ubuntu)
server:
    log-queries: yes
    log-replies: yes

sudo rndc querylog on

Both log to the journal. This lists the names with a label of 30 characters or more from the last hour:

sudo journalctl --since "1 hour ago" -u unbound -u named -u bind9 \
  | grep -ioE '([a-z0-9-]{30,}\.)+[a-z0-9.-]+' | sort | uniq -c | sort -rn | head -20

Find the program behind the queries

DNS sockets live for a fraction of a second, so sample them: this looks once a second for a minute and counts the programs that had a DNS socket open. Programs that ask the systemd-resolved stub show up with their own name, since the stub listens on port 53 too.

for i in $(seq 1 60); do sudo ss -unpH '( dport = :53 )'; sleep 1; done \
  | grep -o 'users:(("[^"]*",pid=[0-9]*' | sort | uniq -c | sort -rn

systemd-resolve, unbound or named at the top means the queries come through the resolver, from another program: run the capture again on the loopback interface only (-i lo) and compare the times. For a program you do not know, look at its binary, its folder and the unit that started it:

sudo ls -l /proc/PID/exe /proc/PID/cwd
sudo tr '\0' ' ' < /proc/PID/cmdline; echo
systemctl status PID

A tunnel client is a sign of a compromise, not the compromise itself. Look for the rest as in How to find and remove a crypto miner: persistence in cron, systemd and SSH keys, changed system binaries, and how the attacker got in.

Block the tunnel

Stop the program first. Then make sure DNS can only leave through the resolvers you chose: this nftables table drops DNS to any other address, and DNS over TLS everywhere. Replace 192.0.2.53 with your resolvers from step 3 (several, separated by commas).

sudo nft add table inet dnsguard
sudo nft 'add chain inet dnsguard out { type filter hook output priority 0; policy accept; }'
sudo nft add rule inet dnsguard out ip daddr != { 127.0.0.0/8, 192.0.2.53 } meta l4proto { tcp, udp } th dport 53 drop
sudo nft add rule inet dnsguard out tcp dport 853 drop

A tunnel through your own resolver still works, because the resolver forwards the queries. If you run unbound, answer the tunnel domain with "does not exist":

# In the server: section of unbound.conf
    local-zone: "tunnel.example." always_nxdomain

sudo unbound-checkconf && sudo systemctl restart unbound

With systemd-resolved alone there is no per-domain block: point the server at a resolver that can do it, or rely on removing the program. /etc/hosts does not help, because it cannot match every subdomain.

Questions

My firewall blocks outgoing traffic. Can DNS tunneling still work?

Yes, if the server may query a resolver, and almost every server may. The resolver forwards the queries to the attacker's name server for you. That is why the resolver must also block the domain, or the program must go.

Does DNSSEC or DNS over HTTPS stop it?

No. DNSSEC proves answers are authentic, and the attacker's answers are authentic for their own domain. DNS over HTTPS hides the queries from the network, which makes a tunnel harder to see, not impossible to build.

What legitimate traffic looks like a tunnel?

Antivirus and reputation lookups that encode file hashes in names, the blocklist checks of a mail server, and some CDN, cloud and telemetry names. Look up the domain before acting.