InfraFenceInfraFence

Live Security Operations

Get Started

Bots

How InfraFence recognizes about 1,500 bots from public lists, unmasks fake search engine bots with reverse DNS, and lets you allow, log or block each category or bot.

Last updated:

The lists

Each server downloads two public bot lists once a day, straight from their publishers:

Together they recognize about 1,500 bots. The Bots page groups them in categories: search engines, SEO tools, AI crawlers, scanners, monitoring, advertising, link previews, feed readers, HTTP libraries, browser automation, archivers and research.

Allow, log only, block

For each category, and for each single bot, choose Allow, Log only or Block, or let a bot Follow the category and a category Follow the list. The page explains each category: what it is, the risk, and what happens if you block it. It also shows the bots Seen on your servers (last 7 days).

  • Block bans the bot's address at the firewall, for the ban duration set under Settings › Agent. If Allow web server configuration changes is on, the agent also refuses blocked bots inside nginx or Apache by their user agent.
  • Log only reports the bot, at most once an hour for each address and bot.
  • Allow lets them in, after checking that a bot claiming to be a search engine really is one.

Blocking search engines removes your sites from search results; the page warns you before you do.

Fake search engine bots

Attackers often pretend to be Googlebot. For 18 well-known bots (Googlebot and the other Google crawlers, Bingbot, Yahoo, Yandex, Baidu, Applebot, DuckDuckBot, Facebook, LinkedIn, Twitter, Pinterest, Ahrefs and Semrush) the agent checks the address with forward-confirmed reverse DNS: the address must resolve to the bot's domain, and that name back to the same address. A bot that fails is reported as spoofed. Results are cached for 24 hours.

Unknown bots

A user agent that looks automated (bot, crawler, spider, curl, wget, python-requests and similar) but is in no list is reported as an unknown bot, so you can decide what to do with it.

Under a hosting panel

When a hosting panel owns the web server configuration, blocked bots are still banned at the firewall; only the user-agent blocking inside the web server is left out.