What is bot traffic?

By Nik Andreev, Founder & SEO specialist · Last reviewed · Editorial policy

Bot traffic is every request to your site made by automated software instead of a person. It includes search crawlers like Googlebot, AI crawlers and fetchers, monitoring tools, scrapers and malicious bots. Some of it is useful, some is not, and because bot names are easy to fake, sorting it reliably needs verification.

Full guide How to detect bot traffic in your logs A log-first method: the signals that identify a bot, separating good crawlers from scrapers, and why verification matters. Read the guide →

At a glance

What counts
Any automated request: crawlers, fetchers, monitors, scrapers, attack tools
Good bots
Search engine crawlers, AI crawlers you allow, uptime monitors
Unwanted bots
Unauthorized scrapers, credential stuffing, spoofed crawlers
Where you see it
Server, CDN and WAF logs. Page-tag analytics usually misses it
How to identify it
User agent first, then verify: published IPs, reverse DNS or signatures
Main pitfall
Trusting the user agent. It is self-reported and easy to fake

How to spot it

  • User agent: most bots name themselves with a token such as Googlebot or GPTBot. Treat it as a claim, not proof.
  • Behavior: no JavaScript or asset requests, very regular timing, many URLs per minute, or requests for robots.txt and sitemaps.
  • Verification: confirm the claim with published IP ranges, FCrDNS or an HTTP message signature. A bot name that fails these checks is user-agent spoofing.
From Lume's data

How much of the traffic is bots

15 to 75%
of all requests came from bots, depending on the site
Under 3%
of requests came from AI agents browsing on a person's behalf, on every opted-in site

Method: Requests to sites that opted in to this anonymous analysis, August, September, October and November 2026 (to 4 November). Each site is summarized separately, then figures are rounded. Counts are requests claiming each bot name.

Types of bot traffic

AI crawler Bots that collect pages for AI training or AI search. Web scraper Software that extracts content or data from pages, often without permission. Agentic browser A browser driven by an AI agent on a person's behalf. It can look almost human.

Quick answers

Why doesn't Google Analytics show bot traffic?

Most bots do not run JavaScript, so the tag never fires, and GA4 also filters known bots. Server logs see every request.

Is all bot traffic bad?

No. Search engines and many AI systems need to crawl your pages to show them. The goal is to tell the useful bots from the rest.

How do I know a bot is who it claims to be?

Check the IP against the operator's published ranges, run forward-confirmed reverse DNS, or validate a Web Bot Auth signature where one is sent.

Sources