How to block AI crawlers on your server

Know which crawlers you are dealing with

The main ones, as of October 2026, from each operator's own documentation. Some are crawlers; some are only robots.txt tokens that control how another crawler's data is used.

Decide per purpose. Blocking training crawlers keeps your content out of future training sets; blocking AI search and user fetchers also keeps your site out of those answers, and the visits they send. To see which of them reach your server, count the user agents in the access log:

sudo awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

Ask them to stay away with robots.txt

Put a robots.txt at the root of every host you want to cover (each subdomain has its own). Several User-agent lines can share one rule:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /

This list blocks training and leaves search alone. Add OAI-SearchBot, Claude-SearchBot and PerplexityBot to keep out of AI search too. Keep your existing rules for other crawlers below it, and never write User-agent: * with Disallow: / unless you also want to leave Google and Bing.

What robots.txt cannot do

• It is a request. Crawlers that follow it stop; the others do not, and nothing tells you which is which.
• User-triggered fetchers may not apply it, by their operators' own account.
• Changes take time: operators cache the file, often for up to a day.
• A scraper that hides its name or claims to be a browser is not covered at all.

Block by user agent in nginx or Apache

The web server can refuse the crawlers that announce themselves. This stops them even if they ignore robots.txt, as long as they keep their name.

nginx

The map goes in the http context: files in /etc/nginx/conf.d/ are included there by the Debian, Ubuntu and RHEL-family packages.

sudo tee /etc/nginx/conf.d/ai-crawlers.conf > /dev/null <<'EOF'
map $http_user_agent $ai_crawler {
    default 0;
    ~*(GPTBot|ClaudeBot|CCBot|meta-externalagent|PerplexityBot) 1;
}
EOF

Then add this inside each server block that should refuse them, in /etc/nginx/sites-available/ on Debian and Ubuntu or /etc/nginx/conf.d/ on the RHEL family:

if ($ai_crawler) {
    return 403;
}

Test the configuration, reload, and check: the first curl must print 403, the second 200 (replace example.com with your site).

sudo nginx -t && sudo systemctl reload nginx
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.2" https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/

Apache

With mod_rewrite, inside each <VirtualHost> (both the port 80 and the port 443 one) or in the site's .htaccess. Rewrite rules in the main configuration do not apply to virtual hosts.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|meta-externalagent|PerplexityBot) [NC]
RewriteRule ^ - [F]

sudo a2enmod rewrite
sudo apachectl configtest && sudo systemctl reload apache2

sudo apachectl configtest && sudo systemctl reload httpd

Verify real bots by reverse DNS

Anyone can send User-Agent: Googlebot. Scrapers do it to get past rules that allow search engines. A real crawler is confirmed by forward-confirmed reverse DNS: the address resolves to a name in the operator's domain, and that name resolves back to the same address.

sudo apt install -y bind9-host

sudo dnf install -y bind-utils

ip=66.249.66.1
name=$(host "$ip" | awk '/pointer/ {print $NF; exit}' | sed 's/\.$//')
echo "$name"
host "$name" | awk '{print $NF}' | grep -qxF "$ip" && echo "confirmed" || echo "NOT confirmed"

• Googlebot: googlebot.com or google.com. Bingbot: search.msn.com. Applebot: applebot.apple.com. CCBot: crawl.commoncrawl.org.
• A name in another domain, or one that does not resolve back to the address, is a fake.

OpenAI, Anthropic, Perplexity and Common Crawl publish their crawlers' address ranges as JSON instead (the links are in each operator's documentation). This checks an address against such a list:

python3 - 20.171.207.10 https://openai.com/gptbot.json <<'EOF'
import ipaddress, json, sys, urllib.request
ip = ipaddress.ip_address(sys.argv[1])
req = urllib.request.Request(sys.argv[2], headers={"User-Agent": "Mozilla/5.0"})
prefixes = json.load(urllib.request.urlopen(req))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in prefixes]
print("in the list" if any(ip in n for n in nets) else "NOT in the list")
EOF

The same JSON lists can be loaded into an nginx geo block or the firewall, if you want to allow only the real crawlers that use a given name.

Rate limit crawlers

A rate limit works whatever a crawler calls itself, and protects the server from the ones that crawl too fast. In nginx, this limits each address whose user agent looks like a bot to 30 requests a minute, with a short burst; other visitors are not counted, because an empty key is not limited.

sudo tee /etc/nginx/conf.d/bot-limit.conf > /dev/null <<'EOF'
map $http_user_agent $bot_key {
    default "";
    ~*(bot|crawler|spider|scraper) $binary_remote_addr;
}
limit_req_zone $bot_key zone=bots:10m rate=30r/m;
limit_req_status 429;
EOF

Then, in the server or location blocks of the dynamic pages, and reload as above:

limit_req zone=bots burst=10 nodelay;

Apache has no per-address request limit of its own (mod_ratelimit limits bandwidth). The usual options are mod_evasive, a fail2ban jail on the access log, or nginx or a CDN in front.

When a crawler ignores everything

If a crawler ignores robots.txt and changes its user agent, block the addresses it uses at the firewall (replace the example range):

sudo ufw insert 1 deny from 198.51.100.0/24

sudo firewall-cmd --permanent --zone=drop --add-source=198.51.100.0/24
sudo firewall-cmd --reload

Questions

Does blocking AI crawlers hurt my Google ranking?

Not if you block the right names. Google-Extended in robots.txt does not affect Google Search, according to Google. Blocking Googlebot, or a pattern that matches it, does.

Is robots.txt enough?

For the crawlers that follow it, yes. It does nothing against the ones that ignore it or hide their name, which is why the user-agent rules and rate limits exist.

Should I block AI search crawlers too?

It depends on whether you want to appear in AI answers. Blocking them keeps your content out of those answers and loses the visits they send. Many sites block training crawlers and allow search.