How to stop a server that is sending spam

Check whether the server is sending spam

Start with the mail queue. A healthy server has a handful of messages waiting, often none. Thousands of messages, or one sender that owns most of them, means spam. The second command lists the senders with the most queued messages.

sudo postqueue -p | tail -n 1
sudo postqueue -p | awk '/^[0-9A-Za-z]/ {print $7}' | sort | uniq -c | sort -rn | head

Exim, as on cPanel and with Debian's exim4 (the exim command is there too), counts the queue with exim -bpc. The next lines list the top senders and the top destination domains:

sudo exim -bpc
sudo exim -bp | awk '/^ *[0-9]+[mhd] / {print $4}' | sort | uniq -c | sort -rn | head
sudo exim -bp | exiqsumm | sort -rn | head

A sender of <> or MAILER-DAEMON is a bounce. A queue full of bounces means the server accepted or sent mail to addresses that do not exist: typical of spam sent from here, or of backscatter from a form that sends a copy to whatever address is typed in.

Which account is sending

Spam sent through a mailbox shows up as one authenticated user with far more messages than the others. Postfix logs it as sasl_username, in the log files or, without rsyslog, in the journal:

sudo sh -c "zgrep -h -o 'sasl_username=[^ ,]*' /var/log/mail.log* /var/log/maillog* 2>/dev/null" \
  | sort | uniq -c | sort -rn | head
sudo journalctl --since "24 hours ago" -t postfix/smtpd | grep -o 'sasl_username=[^ ,]*' \
  | sort | uniq -c | sort -rn | head

Exim logs the login as A= on every message it accepts, and the folder of the script as cwd= for mail submitted locally (cPanel logs it by default). A cwd= inside a website, such as /home/user/public_html/wp-content/…, points at the script:

sudo grep -h ' <= ' /var/log/exim_mainlog /var/log/exim4/mainlog 2>/dev/null \
  | grep -o ' A=[^ ]*' | sort | uniq -c | sort -rn | head
sudo grep -h -o 'cwd=[^ ]*' /var/log/exim_mainlog /var/log/exim4/mainlog 2>/dev/null \
  | sort | uniq -c | sort -rn | head

Which PHP script is sending

Mail sent by PHP's mail() goes through the local sendmail and does not log in, so it has no user. Read the headers of one queued message: with mail.add_x_header on, PHP adds X-PHP-Originating-Script, the user ID and file name of the script.

sudo postcat -hq QUEUE_ID | grep -iE '^(From|To|Subject|X-PHP-Originating-Script):'
sudo exim -Mvh QUEUE_ID | grep -iE '(From|To|Subject|X-PHP-Originating-Script):'

If the header is not there, turn it on, and have PHP log every mail() call with the full path and line of the script. Find the php.ini that your web server uses (PHP-FPM has its own on Debian and Ubuntu):

php --ini | grep 'Loaded Configuration'
ls /etc/php/*/fpm/php.ini /etc/php.ini 2>/dev/null

mail.add_x_header = On
mail.log = syslog

Restart PHP-FPM or the web server (sudo systemctl restart php8.3-fpm, php-fpm or httpd, depending on the system), wait a few minutes and read the log:

sudo journalctl --since "1 hour ago" | grep 'mail() on' | tail -n 20
sudo grep -h 'mail() on' /var/log/syslog /var/log/messages 2>/dev/null | tail -n 20

Programs that talk SMTP directly

Some spam tools skip the mail server and connect to port 25 of the recipients' servers themselves, so they leave nothing in the queue. Look at the outgoing SMTP connections and the programs that own them. Your mail server (master/smtp for Postfix, exim) is normal; php, perl, python or a random name is not.

sudo ss -tnp state established '( dport = :25 or dport = :465 or dport = :587 )'
sudo ss -tnp state established '( dport = :25 )' | grep -o 'users:(("[^"]*"' | sort | uniq -c | sort -rn

Stop the spam now

Stop the source first, then the queue. For a mailbox, change its password in your panel or mail system and end its sessions. For a script, keep a copy as evidence and make it unreadable, so the web server can no longer run it:

sudo mkdir -p /root/evidence
sudo cp -p /path/to/script.php /root/evidence/
sudo chmod 000 /path/to/script.php

For a process that talks SMTP directly, find its PID in the ss output above, check where it comes from and stop it as in the crypto miner guide: it is malware, and it has persistence of its own.

While you clean up, keep the queue from going out. With Postfix, defer_transports holds everything for remote delivery and keeps accepting new mail, so nothing is lost:

sudo postconf -e 'defer_transports = smtp'
sudo postfix reload

When the queue is clean (next step), release it:

sudo postconf -e 'defer_transports ='
sudo postfix reload
sudo postqueue -f

With Exim, freeze the messages of the sender you found. Frozen messages stay in the queue and are not delivered:

sudo exiqgrep -i -f 'spammer@example\.com' | xargs -r sudo exim -Mf

Clean the mail queue

Delete the spam by sender, and the bounces with it, so that the legitimate messages in the queue stay. Run step 1 again first, to be sure of the sender address.

sudo postqueue -p | awk '/^[0-9A-Za-z]/ && $7 == "spammer@example.com" {sub(/[*!]$/, "", $1); print $1}' \
  | sudo postsuper -d -
sudo postqueue -p | awk '/^[0-9A-Za-z]/ && $7 == "MAILER-DAEMON" {sub(/[*!]$/, "", $1); print $1}' \
  | sudo postsuper -d -

With Exim, the first line deletes the messages of the sender, the second the frozen ones, which after spam are mostly bounces:

sudo exiqgrep -i -f 'spammer@example\.com' | xargs -r sudo exim -Mrm
sudo exiqgrep -z -i | xargs -r sudo exim -Mrm

Count the queue again. If it grows back, the source is still active: go back to step 1.

Find how the spam got out

Changing a password or deleting a script stops this round. The next one starts unless you know how it began.

A compromised mailbox

A mailbox used by a spammer logs in from many addresses, often in many countries, sometimes thousands in a day. Count the addresses that used it:

sudo sh -c "zgrep -h 'user@example.com' /var/log/mail.log* /var/log/maillog* /var/log/exim_mainlog* /var/log/exim4/mainlog* 2>/dev/null" \
  | grep -oE '\[[0-9]{1,3}(\.[0-9]{1,3}){3}\]' | sort | uniq -c | sort -rn | head -20

The password was guessed (look for many failed logins on that mailbox before the first successful one) or stolen from the user's computer or another site. Choose a long new password, and check every other mailbox with a weak one.

A PHP script

A script that sends whatever a request asks for is either malware uploaded through a hole, or a legitimate form without protection. Look for mailers and recently changed PHP files in the sites:

sudo grep -rlE 'mail\s*\(\s*\$_(POST|GET|REQUEST)|LeafMailer|eval\s*\(\s*base64_decode' \
  --include='*.php' /var/www /home 2>/dev/null | head -50
sudo find /var/www /home -name '*.php' -type f -mtime -7 2>/dev/null | head -50

Then see who called the script, and how often. A contact form posted thousands of times by a few addresses is being abused; a file nobody should know about, called from many addresses, is a backdoor.

sudo sh -c "grep -hI 'POST /path/to/script.php' /var/log/nginx/*access*.log /var/log/apache2/*access*.log /var/log/httpd/*access_log 2>/dev/null" \
  | awk '{print $1}' | sort | uniq -c | sort -rn | head

• A malicious script: remove it, then find how it was uploaded (an outdated plugin or theme, a weak admin password, an upload form) and close that. The crypto miner guide shows how to look for the rest of what an attacker leaves.
• An abused form: add a CAPTCHA or a hidden field that bots fill in, never let the visitor choose the recipient, and do not send a copy to the address typed into the form.

An open relay

Rarer today, but check it: a mail server that delivers mail for anyone to anyone. In Postfix, mynetworks must list only the server itself and your own networks, and the relay restrictions must end with reject_unauth_destination:

postconf mynetworks smtpd_relay_restrictions smtpd_recipient_restrictions

Prevent it from happening again

SPF, DKIM and DMARC

These three DNS records tell other mail servers which servers may send for your domain and whether a message really comes from you. They do not stop a hacked mailbox from sending spam through your server, but they stop others from forging your domain, and big providers now expect them before they deliver your mail. An example for a domain whose mail leaves from 203.0.113.10:

example.com.         TXT  "v=spf1 mx ip4:203.0.113.10 -all"
_dmarc.example.com.  TXT  "v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com"

DKIM needs a key on the mail server (OpenDKIM for Postfix, built into Exim; panels like cPanel and Plesk do it for you) and its public part in DNS. Check what is published:

dig +short TXT example.com
dig +short TXT _dmarc.example.com
dig +short TXT default._domainkey.example.com

Rate limits

A limit turns a compromised account from a hundred thousand messages into a few hundred. Postfix limits the messages per client address; mail that PHP sends through sendmail does not go through it, and per-user limits need a policy service such as postfwd.

sudo postconf -e 'smtpd_client_message_rate_limit = 100' 'anvil_rate_time_unit = 3600s'
sudo postfix reload

Exim can limit each authenticated user in its RCPT ACL. On cPanel, use **Max hourly emails per domain** in WHM's Tweak Settings instead.

# In the RCPT ACL (acl_check_rcpt), before the accept for authenticated users
deny  authenticated = *
      ratelimit     = 200 / 1h / strict / per_rcpt / $authenticated_id
      message       = Sending rate limit exceeded

Only the mail server may use port 25

Nothing on a web server except the mail server needs to connect to port 25 of other servers. This nftables rule rejects outgoing connections to port 25 from every other user, which stops scripts and malware that talk SMTP directly. Replace postfix with your mail server's user: Debian-exim for Exim on Debian and Ubuntu, exim on the RHEL family, mailnull on cPanel.

sudo nft add table inet smtpguard
sudo nft 'add chain inet smtpguard out { type filter hook output priority 0; policy accept; }'
sudo nft add rule inet smtpguard out tcp dport 25 meta skuid != { root, postfix } reject

Many cloud providers block outgoing port 25 by default and open it on request. If you do not send mail from the server at all, leave it blocked and send through a relay service on 587.

And the usual hardening

• Keep WordPress, plugins and themes up to date, and remove the ones you do not use.
• Protect the mail logins against brute force like SSH: see How to stop SSH brute force attacks.
• Keep mail.add_x_header and mail.log on, so the next time the script is known in seconds.

Get the server off blocklists

Ask for removal only after the spam has stopped and the cause is closed: a server that lists again right after delisting is treated more harshly the second time. Check the main lists by DNS, with the server's public address in IP:

IP=203.0.113.10
REV=$(echo "$IP" | awk -F. '{print $4"."$3"."$2"."$1}')
for bl in zen.spamhaus.org bl.spamcop.net; do echo "$bl: $(dig +short "$REV.$bl" A)"; done
dig +short -x "$IP"

An empty answer means not listed; an address like 127.0.0.2 means listed. Spamhaus refuses queries sent through public resolvers and answers 127.255.255.254 to them: in that case use its website. The last line shows the reverse DNS (PTR) of the address, which should be a host name that resolves back to it.

• Spamhaus: look the address up on check.spamhaus.org, which explains each listing and how to request removal.
• SpamCop: listings expire by themselves once reports stop (spamcop.net).
• Microsoft (Outlook, Hotmail): request delisting at sender.office.com.
• Gmail: no delisting form; Postmaster Tools shows your domain's reputation as it recovers.

Questions

The server has no mail server. How can it send spam?

PHP's mail() uses the sendmail that almost every distribution installs, and malware can connect to port 25 of other servers by itself. Check the outgoing connections in step 1 and the rule that keeps port 25 for the mail server.

Will SPF, DKIM and DMARC stop the spam?

No. They prove which servers may send for your domain, and spam sent through a hacked mailbox on your server passes them. They protect your domain from forgery and help your legitimate mail get delivered.

How long does it take to get off a blocklist?

It depends on the list. SpamCop listings expire by themselves once the reports stop; Spamhaus and Microsoft act on a removal request once the problem is fixed. Reputation at Gmail and Outlook recovers over days of clean sending.