How to find and remove a crypto miner on your server

Spot the miner

Start with what uses the CPU. A miner usually takes one process at 100% per core, running for days.

ps -eo pid,user,%cpu,etime,args --sort=-%cpu | head -n 15

• Names: xmrig and its variants, or names chosen to look harmless: kdevtmpfsi, kinsing, kworkerds, [kthreadd] written by hand, a random string.
• A kernel thread shows in brackets and has no binary: a bracketed name with a path behind it is a fake.
• The user: a miner running as www-data, apache, nginx, redis or postgres came in through that service.

For a suspicious PID, look at its binary, its working directory, its full command line and which systemd unit started it:

sudo ls -l /proc/PID/exe /proc/PID/cwd
sudo tr '\0' ' ' < /proc/PID/cmdline; echo
systemctl status PID

A binary that shows (deleted), or one that runs from /tmp, /var/tmp or /dev/shm, is a strong sign. This lists every such process at once:

sudo ls -l /proc/[0-9]*/exe 2>/dev/null | grep -E '\(deleted\)|/tmp/|/var/tmp/|/dev/shm/'

When top shows nothing: ld.so.preload

If the load is high but no process explains it, the miner may be hiding behind a library loaded into every program through /etc/ld.so.preload. That file normally does not exist. If it does and names a library you do not know, ps and top are lying to you.

ls -l /etc/ld.so.preload 2>/dev/null && cat /etc/ld.so.preload
sudo grep -l LD_PRELOAD /proc/[0-9]*/environ 2>/dev/null

Connections to a mining pool

Miners talk to a pool with the stratum protocol. Ports like 3333, 4444, 5555, 7777 and 14444 are typical, but pools also listen on 443 and 80, so also look at the connections of the process itself:

sudo ss -tnp state established '( dport = :3333 or dport = :4444 or dport = :5555 or dport = :7777 or dport = :14444 )'
sudo ss -tnp state established | grep PID

Miner configuration files usually contain a pool URL. This searches the usual places for them:

sudo grep -rIl -E 'stratum\+(tcp|ssl)://|"pools"|donate-level' /tmp /var/tmp /dev/shm /root /home /var/www /etc 2>/dev/null

What brings it back

Look for persistence before you kill anything, so you know what to remove. Cron first: lines that download with curl or wget, decode base64 or run from /tmp.

sudo cat /etc/crontab
sudo ls -la /etc/cron.d /etc/cron.hourly /etc/cron.daily /var/spool/cron 2>/dev/null
for u in $(cut -d: -f1 /etc/passwd); do sudo crontab -l -u "$u" 2>/dev/null | sed "s/^/$u: /"; done
sudo grep -rE 'curl|wget|base64|/tmp/|/dev/shm' /etc/crontab /etc/cron.* /var/spool/cron 2>/dev/null

Then systemd timers and the unit files changed in the last 30 days:

systemctl list-timers --all
sudo sh -c "find /etc/systemd/system /usr/lib/systemd/system /lib/systemd/system /root/.config/systemd /home/*/.config/systemd \
  -type f \( -name '*.service' -o -name '*.timer' \) -mtime -30 2>/dev/null"

Then SSH keys you did not add, rc.local, shell startup files and accounts you did not create:

sudo sh -c 'cat /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys 2>/dev/null'
ls -l /etc/rc.local 2>/dev/null
sudo sh -c "grep -lE 'curl|wget|/tmp/|/dev/shm' /etc/profile.d/* /etc/bash.bashrc /etc/bashrc /root/.bashrc /root/.profile /home/*/.bashrc 2>/dev/null"
awk -F: '$3 == 0 || $3 >= 1000 {print $1, $3, $7}' /etc/passwd

Finally, check that the system's own binaries were not replaced. The output lists packaged files whose content changed; configuration files are left out.

sudo dpkg --verify | grep -v ' c /'

sudo rpm -Va 2>/dev/null | grep -E '^..5' | grep -v ' c /'

Contain it

If your provider can take a snapshot of the disk, take it now: it keeps the evidence and a way back. Then cut the miner off from its pool. Replace 198.51.100.23 with the pool address from step 1. This nftables table works on Debian, Ubuntu and the RHEL family next to ufw or firewalld, and is gone after a reboot.

sudo nft add table inet quarantine
sudo nft 'add chain inet quarantine out { type filter hook output priority 0; policy accept; }'
sudo nft add rule inet quarantine out ip daddr 198.51.100.23 drop

Keep a copy of the binary and its command line before it is deleted. /proc/PID/exe works even when the file on disk was already removed. The SHA-256 can be looked up on public malware databases.

sudo mkdir -p /root/evidence
sudo cp /proc/PID/exe /root/evidence/miner.bin
sudo tr '\0' ' ' < /proc/PID/cmdline | sudo tee /root/evidence/cmdline.txt
sudo sha256sum /root/evidence/miner.bin

Freeze the process first, so a watchdog does not see it die and restart it at once, look at its parent, then kill it. If the parent is a script or a second miner process, deal with that one too.

sudo kill -STOP PID
ps -o pid,ppid,user,args -p PID
sudo kill -9 PID

Remove it and everything that brings it back

Remove each piece of persistence you found in step 1, then the binaries. For a systemd unit (replace NAME):

sudo systemctl disable --now NAME.service
sudo rm /etc/systemd/system/NAME.service
sudo systemctl daemon-reload

• Cron: delete the lines with sudo crontab -e -u USER, and the files in /etc/cron.d that you did not create.
• SSH: delete the keys you do not recognize from every authorized_keys.
• Accounts: lock or delete users you did not create (sudo usermod -L NAME), and check /etc/sudoers and /etc/sudoers.d/.
• Binaries and configuration files of the miner, wherever step 1 found them.

Miners often make their files immutable so that rm fails with Operation not permitted even as root. Check and clear the attributes first:

sudo lsattr /path/to/file
sudo chattr -i -a /path/to/file
sudo rm /path/to/file

If /etc/ld.so.preload names a library you do not know, move it away (it is unusual for legitimate software to use it) and delete the library it named:

sudo mv /etc/ld.so.preload /root/evidence/ld.so.preload
sudo ldconfig

Reboot, then check the CPU again after an hour, and the next day: a miner that comes back means a piece of persistence or the entry point is still there.

Find how it got in

If you do not close the way in, the miner is back within days. The user the miner ran as and the time its files were created point to it:

ps -o user,pid,lstart,args -p PID
sudo stat /path/to/miner

• Running as the web server user: a vulnerable plugin, theme or application, or an upload form. Look at the web logs around the time the file appeared.
• Running as redis, or files created by it: Redis open to the internet without a password.
• Started by a container or by dockerd: the Docker API exposed on 2375 or 2376.
• Running as root or a normal user with a login just before: a weak password or a stolen SSH key.

last -i -n 50
sudo journalctl --since "14 days ago" | grep -E "Accepted (password|publickey)"

sudo grep -hE 'wget|curl|base64|%2Fbin%2Fsh|/bin/sh' \
  /var/log/nginx/access.log* /var/log/apache2/access.log* /var/log/httpd/access_log* 2>/dev/null | tail -n 50

And the services that are common ways in when they listen on every interface: the Docker API, Redis, web consoles on 8080, Elasticsearch, MongoDB, Memcached.

sudo ss -tlnp | grep -E ':(2375|2376|6379|8080|9200|27017|11211)\b'

Harden the server afterwards

• Close the entry point you found: update or remove the application, bind Redis and the databases to localhost with a password, never expose the Docker API.
• SSH with keys only and no root login: see How to stop SSH brute force attacks.
• Install updates, turn on automatic security updates and the firewall: see Secure a new VPS: the checklist.
• Limit outbound traffic where you can: a web server rarely needs to open connections to arbitrary ports.
• Watch CPU use, outbound connections and changes to cron, systemd and SSH keys from now on, so the next one is caught in minutes and not weeks.

Questions

Can I just kill the process?

It will come back. Miners install a cron job, a systemd unit or an SSH key so they restart, and the hole they came in through is still open. Remove the persistence and close the entry point.

Do I have to reinstall the server?

If the miner ran as root, system binaries changed or /etc/ld.so.preload was used, yes: you cannot prove the server is clean. If it ran as an unprivileged user and nothing else changed, a careful cleanup can be enough.

My provider says the server is mining but I see nothing. Why?

The miner may be hidden with /etc/ld.so.preload, run in a container, or start only at certain hours from cron. Check the preload file, docker ps, and the connections the server opens over time.