How to detect and remove an /etc/ld.so.preload rootkit

Understand how a preload rootkit hides

• Processes: ps and top read the /proc directory; a hooked readdir() skips the attacker's process IDs.
• Files: the same hook hides files by name, often including /etc/ld.so.preload itself and the library, so ls and cat say they do not exist.
• Connections: netstat and ss lose the lines of the attacker's sockets.
• Everything else works normally, which is the point: the server looks clean to every tool you would usually trust.

What it cannot hide from: programs that do not use the system's C library (statically linked binaries such as busybox-static), the kernel, and a system booted from other media. A rootkit loaded as a kernel module is another, harder case: it hides from /proc itself, and the checks below can miss it.

Check with tools the rootkit cannot fool

Get a statically linked busybox

busybox-static is one binary with ls, cat, grep, sh and more, linked without the C library, so a preloaded library never loads into it. ldd must answer not a dynamic executable.

sudo apt install busybox-static
ldd /bin/busybox

The RHEL family has no static busybox in its base repositories. Copy one from a clean Debian or Ubuntu machine (after installing busybox-static there), and call it as /root/busybox instead of busybox in the commands below:

scp /bin/busybox root@SERVER_IP:/root/busybox
ssh root@SERVER_IP 'chmod 700 /root/busybox && ldd /root/busybox'

Read the file directly

Compare what busybox sees with what the system's own tools see. If busybox shows the file and the normal ls and cat say it does not exist, a rootkit is active right now.

sudo busybox ls -la /etc/ld.so.preload
sudo busybox cat /etc/ld.so.preload
ls -la /etc/ld.so.preload
cat /etc/ld.so.preload

For each library the file names, check whether a package owns it. Rare legitimate users exist, such as the Snoopy command logger and some monitoring agents, and they come from a package you installed:

dpkg -S /usr/lib/libhide.so

rpm -qf /usr/lib/libhide.so

A library no package owns is suspect. Note its size and hash, look at the names it contains (hooked functions such as readdir, the word /proc), and see which processes have it loaded:

sudo busybox ls -la /usr/lib/libhide.so
sudo busybox sha256sum /usr/lib/libhide.so
sudo busybox strings /usr/lib/libhide.so | busybox grep -iE 'readdir|/proc|preload|hide' | busybox head

sudo busybox sh -c "busybox grep -l '/usr/lib/libhide.so' /proc/[0-9]*/maps"

Compare /proc with ps

busybox reads /proc without the hook; ps uses it. Every process ID that busybox finds and ps does not is hidden, and the line shows its binary. Processes that ended during the loop can show up once: run it twice, and look at the IDs that appear both times.

sudo busybox sh -c 'for d in /proc/[0-9]*; do p=${d#/proc/}; /bin/ps -p "$p" > /dev/null 2>&1 || echo "$p $(busybox readlink "$d/exe")"; done'

The same for network connections: busybox netstat reads /proc/net directly. A connection in the first list and missing in the second is hidden.

sudo busybox netstat -tan
sudo ss -tanp

LD_PRELOAD in environments and services

The same trick works per process, through the LD_PRELOAD variable in a unit file, a shell profile or /etc/environment. Look for it in the running processes and in the usual files:

sudo busybox sh -c "busybox grep -l LD_PRELOAD /proc/[0-9]*/environ"
sudo busybox grep -rs LD_PRELOAD /etc/environment /etc/profile /etc/profile.d /etc/bash.bashrc /etc/bashrc \
  /etc/systemd /lib/systemd/system /usr/lib/systemd/system /root/.bashrc /root/.profile

Changed system binaries

Some kits also replace ps, ls or sshd. The package database lists packaged files whose content changed (configuration files left out):

sudo dpkg --verify | grep -v ' c /'

sudo rpm -Va 2>/dev/null | grep -E '^..5' | grep -v ' c /'

The certain way: from a rescue system

dpkg and rpm are dynamically linked too. For a result you can rely on, boot the provider's rescue system, mount the server's disk and read the file there, where nothing of the server is running. Find the root partition with lsblk first: /dev/sda1 is an example.

lsblk -f
mount /dev/sda1 /mnt
ls -la /mnt/etc/ld.so.preload && cat /mnt/etc/ld.so.preload

Remove the preload rootkit

If the provider can snapshot the disk, do it first. Then keep a copy of the file and the library as evidence:

sudo busybox mkdir -p /root/evidence
sudo busybox cp -a /etc/ld.so.preload /usr/lib/libhide.so /root/evidence/
sudo busybox sha256sum /root/evidence/*

Rootkits often make their files immutable, so that even root gets Operation not permitted. busybox-static on Debian has no chattr; the system's own lsattr and chattr work here, because preload kits hide files by name and do not usually touch attributes:

sudo lsattr /etc/ld.so.preload /usr/lib/libhide.so
sudo chattr -i -a /etc/ld.so.preload /usr/lib/libhide.so

Remove the preload file with busybox. From the next program started on, nothing is preloaded any more, and the normal ls agrees that the file is gone:

sudo busybox rm /etc/ld.so.preload
ls -la /etc/ld.so.preload

The processes that were already running still have the library loaded, but the new ps and ss you start no longer do, so what was hidden is visible now. Look for it, and freeze what does not belong before you kill it:

ps -ef --forest | less
sudo ss -tanp
sudo kill -STOP PID

Remove what brings it back: cron jobs, systemd units, SSH keys and accounts, as in How to find and remove a crypto miner, which goes through each place. Then delete the library and reboot, so that no running process keeps it:

sudo busybox rm /usr/lib/libhide.so
sudo ldconfig
sudo reboot

Decide when to rebuild the server

Writing /etc/ld.so.preload needs root. Once an attacker has had root, cleaning removes what you found, not necessarily everything that was there. Rebuild from a fresh image when any of these is true:

• System binaries changed (the package check above).
• The kernel is tainted by an unknown module, or lsmod lists a module you cannot explain.
• You did not find how the attacker got in.
• The server holds customer data, payment data or credentials to other systems.

cat /proc/sys/kernel/tainted
lsmod | sort | head -50
sudo dmesg | grep -iE 'taint|module verification|unsigned'

0 in tainted is a clean kernel; other values have reasons, such as proprietary drivers, so look them up before you conclude. To rebuild: set up a new server from the provider's image, update it, restore the data (not programs or configuration you cannot check) from a backup made before the infection, change every secret, then move the traffic over. The new VPS checklist covers the first hour.

Questions

Is /etc/ld.so.preload always malicious?

No, but it is rare. A normal server does not have one. A few legitimate tools use it, such as the Snoopy command logger, and they come from a package you installed: dpkg -S or rpm -qf on the library tells you.

Is deleting the file enough?

It stops the hiding for every new program, which is why it is the first step. It does not remove what the attacker did with root: persistence, other backdoors, stolen secrets. That is what the rest of the cleanup, or a rebuild, is for.

Do rkhunter and chkrootkit find it?

They check for preloaded libraries and known kits, and are worth running. They are scripts that call the system's own tools, so on an infected system they can be shown the same hidden view. Run them from a rescue system for a result you can rely on.