How to verify Googlebot (and catch spoofed bots)
On this page
The fastest way to verify Googlebot is a forward-confirmed reverse DNS lookup: take the IP address from your server logs, run a reverse DNS lookup, confirm the hostname ends in googlebot.com or google.com, then run a forward DNS lookup on that hostname and confirm it resolves back to the same IP. If both directions match, the request is genuinely from Google. If the user-agent says “Googlebot” but the IP fails this check, you are looking at an impostor. The user-agent header alone is never proof, because any client can set it to any string in a single line of code.
That gap between what a request claims and what it can prove is where most bot problems live. Fake Googlebot is used to scrape content, bypass rate limits, and evade paywalls, because site owners whitelist “Googlebot” for SEO reasons and attackers know it. It is common enough that verifying Googlebot has become standard practice rather than an optional precaution. The exact same trick now targets AI crawlers: a scraper sets its user-agent to GPTBot and rides the goodwill site owners extend to OpenAI. This guide covers the full verification stack that closes both holes: reverse DNS with a forward-confirm, Google’s published IP ranges, and the newest layer, Web Bot Auth cryptographic signatures. It ends with a worked example you can run against a real log line.
Why the user-agent string proves nothing
A user-agent string is a self-reported label, not an identity. It is a plain-text HTTP header the client chooses, and there is no rule that it be truthful. A request carrying Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) might come from Google, or from a scraper on a residential proxy, or from someone testing with a one-line curl command.
This matters because so much infrastructure keys off the string. Site owners relax rate limits for Googlebot to protect crawl coverage. Some serve fuller markup to it. Analytics tools bucket traffic by user-agent. Every one of those behaviors becomes an attack surface the moment the label is trusted on its own. Distinguishing a genuine crawler from a browser or a scraper is the core problem of telling real bots apart, and it is worth understanding what a web crawler actually is and how a crawler differs from a scraper before you decide what to allow.
Google is blunt about this in its own documentation: the user-agent can be spoofed, so verification has to rest on something the impostor cannot fake. That something is the network path the request came from, and, increasingly, a cryptographic signature.
How common is fake Googlebot?
Fake Googlebot is common, and the rate depends heavily on what kind of site you run. The most detailed recent measurement comes from bot-protection firm Centinel Analytica, which verified every request naming a well-known crawler on the news and media sites it protects between late June and 21 September 2026. On the typical site, 1 in 7 requests calling itself Googlebot did not come from Google. The impostors were not subtle about where they came from, but they were careful about how they looked:
- They copy the current version. 94% of fake Googlebot requests claimed the real Googlebot’s current version string, so checking the version number catches almost nothing.
- They run from data centres. 88% of fake Googlebot requests came from data-centre addresses, about a third of them from a single hosting provider.
- They want your content. 95% of fake Googlebot requests asked for pages rather than images or other assets, compared with 85% for the real Googlebot. Impostors skip the rendering resources and go straight for the articles.
Other sites see far less. In a peer-reviewed 2025 study of 36 Duke University websites presented at ACM IMC, only 33 of 9,103 Googlebot requests came from suspicious networks, on a site that already filtered bots through IP allowlists. The gap between those two figures is the point: how much fake Googlebot you see depends on your content and your existing defences, so the only number that matters is the one in your own logs.
The trend is up. Sysadmin Chris Siebenmann described a surge in June 2026 of hundreds of fake Googlebot requests a day, on some days thousands, where he had previously seen only occasional attempts: “This June, the floodgates opened.” Google’s own Gary Illyes is blunt about why. Asked on the Odys Podcast in October 2026 how common Googlebot impersonation is, he answered “Unfortunately very” (as reported by PPC Land), and said that setting up a script to crawl as Googlebot “would take me probably 30 seconds.” He attributes the rise to AI: companies harvesting content for training have made impersonation worth the effort.
The three layers of real verification
Verifying a bot means proving the request originated from infrastructure the named operator actually controls. No single signal does this cleanly on its own, so production verification stacks three layers, each stronger than the last.
| Layer | What it proves | Strength | Main limitation |
|---|---|---|---|
| Reverse DNS + forward-confirm (FCrDNS) | The IP resolves to and from an operator-owned domain | Strong for Google | Needs a live DNS lookup per IP; only works for operators who set PTR records |
| Published IP ranges | The IP sits inside a range the operator publishes | Strong, fast | Ranges change; you must refresh the list; not every operator publishes one |
| Web Bot Auth signatures | The request carries a valid cryptographic signature from the agent | Strongest | New standard; only signing agents are covered |
The layers are complementary, not redundant. Reverse DNS and IP ranges answer “did this come from Google’s network?” Web Bot Auth answers “is this request cryptographically signed by the agent it claims to be?” - a question that holds even when an agent runs from cloud IPs it does not publish. Use them together and the naive spoof has nowhere to hide.
Layer 1: forward-confirmed reverse DNS (FCrDNS)
Forward-confirmed reverse DNS is the method Google has recommended since 2006, and it remains the gold standard for Googlebot. It works in two directions so a spoofer cannot fake it with a single crafted DNS record.
The steps:
- Take the source IP from your log line, for example
66.249.66.1. - Run a reverse DNS (PTR) lookup on it. A genuine Googlebot IP returns a hostname like
crawl-66-249-66-1.googlebot.com. - Confirm the hostname ends in
googlebot.comorgoogle.com. Do not acceptgoogleusercontent.comas proof of Googlebot (see below). - Run a forward DNS (A/AAAA) lookup on that hostname and confirm it resolves back to the original IP.
Both directions must agree. An attacker might control the reverse record for their own IP, but they cannot make crawl-66-249-66-1.googlebot.com forward-resolve to their server, because they do not control the googlebot.com zone. As Centinel Analytica puts it, only Google can make a googlebot.com name resolve forward to an address it controls.
Two traps catch otherwise careful implementations:
- The
googleusercontent.comhole. Google Cloud virtual machines get reverse DNS names undergoogleusercontent.com, and those names forward-confirm correctly. A check that accepts that suffix lets anyone who rents a Google Cloud VM pass as Google: “Both lookups pass, and the request came from whoever rented the machine.” Google usesgoogleusercontent.comfor some user-triggered fetchers, so if you need to recognise those, verify them against Google’s published fetcher IP ranges instead of trusting the suffix. - Rejecting real special crawlers. Crawlers such as AdsBot use
google.comhostnames and appear inspecial-crawlers.json, not the Googlebot file. A check that only acceptsgooglebot.com, or only loadscommon-crawlers.json, will reject real AdsBot traffic. This two-way handshake is what makes reverse DNS trustworthy, and the mechanics are worth reading in full in our guide to reverse DNS lookup for bot verification.
The command-line version, using host on Linux or macOS:
$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1The forward lookup returns the same IP you started with, so this request is genuinely Googlebot. If step 2 had returned a hostname on some unrelated domain, or step 4 had resolved to a different IP, you would reject it.
Layer 2: Google’s published IP ranges
Google publishes the exact IP ranges its crawlers use as machine-readable JSON, so you can verify without a DNS round-trip on every request. This is faster than reverse DNS at scale and useful when you want to pre-filter traffic in a firewall or edge worker.
Published ranges are also the most decisive signal in practice. Gary Illyes called the IP address “the most valuable right now” for verification (as reported by PPC Land), because Google, Bing, Apple, and OpenAI all publish their crawler ranges. Centinel Analytica’s data backs that up: when it cross-checked its verdicts against operators’ own IP lists, 99.99% of verified requests came from the published ranges, and 98% of unverified requests came from outside them.
The relevant files:
| File | Covers |
|---|---|
developers.google.com/static/crawling/ipranges/common-crawlers.json |
Googlebot |
developers.google.com/static/crawling/ipranges/special-crawlers.json |
AdsBot, other special crawlers |
developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json |
Google-triggered fetchers (Google infrastructure) |
developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json |
User-triggered fetchers (may run from user devices) |
Each entry is a CIDR block, for example 66.249.64.0/27. To verify, check whether the source IP falls inside any range in the matching file. Google notes that these ranges change, so you cannot cache the list forever. Refresh it on a schedule (daily is reasonable) and diff it, or you will start rejecting real Googlebot from a newly added block.
Two cautions. First, membership in an IP range proves the request came from Google’s network, not that the specific product is Googlebot rather than, say, a user-triggered fetch; match against the right file for the crawler you expect. Second, IP ranges verify origin but say nothing about intent, which is why range-matching pairs best with the behavioral view of what the crawler actually requested once inside.
Layer 3: Web Bot Auth signatures
Web Bot Auth is an emerging standard that lets a bot cryptographically sign its requests, so verification no longer depends on DNS or IP lists at all. It is built on HTTP Message Signatures (RFC 9421) and public-key cryptography. The agent holds a private key, publishes the corresponding public key in a directory, and signs each request; your server verifies the signature against the published key.
Mechanically, a signing agent sends a Signature header and a Signature-Agent header pointing to its key directory. You fetch the directory, find the key identified in the signature, and verify. A valid signature proves the request was produced by the holder of that private key, which no user-agent spoof and no IP forgery can replicate. It works even when an agent operates from generic cloud IP space it never published, which is exactly the case that defeats IP-range matching.
This is the direction the agent web is moving, because IP ranges and reverse DNS were designed for a handful of large search crawlers, not for the thousands of AI agents now fetching pages. Google, Cloudflare, and others are backing the approach. Note one limit for this specific job, though: Googlebot itself does not sign its requests yet. Google’s experimental signing covers only a subset of Google-Agent requests, so for Googlebot, reverse DNS and IP ranges remain the checks that decide. For the full mechanics of keys, directories, and the signing flow, see our explainer on what Web Bot Auth is. For now, treat it as the strongest layer where it is available, and fall back to reverse DNS and IP ranges for agents that do not yet sign.
A worked example: verifying one Googlebot hit
Here is the full process against a single log line, start to finish. Suppose your access log contains this entry:
66.249.66.1 - - [25/Jul/2026:09:14:22 +0000] "GET /pricing HTTP/1.1" 200 5120
"-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"The user-agent claims Googlebot. Do not trust it yet. Verify:
Step 1 - reverse DNS. Look up the IP:
$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.The hostname ends in googlebot.com. So far so good, but a reverse record alone is not enough.
Step 2 - forward-confirm. Resolve that hostname back:
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1It resolves to 66.249.66.1, the same IP. Forward and reverse agree. This is genuine Googlebot.
Step 3 - cross-check the IP range (optional belt-and-suspenders). Confirm 66.249.66.1 falls inside a CIDR block in common-crawlers.json, for example 66.249.64.0/27 covers .64.0 through .64.31… in practice you check the file for the matching block. If it is listed, you have two independent confirmations.
Now compare that to a spoof. A scraper on a data-center IP such as 45.130.xx.xx sends the identical Googlebot user-agent. Step 1 returns a hostname that does not end in googlebot.com (or returns nothing), so it fails immediately. The user-agent matched; the network path did not. That is the entire point of verification, and it is why a system that only pattern-matches user-agent strings will happily wave the impostor through.
Spoofed GPTBot: the same problem, a new name
Spoofed GPTBot is the AI-era version of fake Googlebot, and it slips past the same naive checks. As site owners started allowing or tracking OpenAI’s crawler, scrapers began setting their user-agent to GPTBot to borrow that trust, exactly as they long borrowed Googlebot’s. The label is just as free to copy.
The numbers are, if anything, worse than for Googlebot:
- In Centinel Analytica’s 2026 news-site data, 84% of GPTBot claims on the typical site were fake, and 99% of Perplexity-User claims. (Pooled across all traffic the GPTBot figure falls to 2%, because real GPTBot volume concentrates on large sites. The typical smaller site sees mostly fakes.)
- Across HUMAN Security’s customer base in 2025, 1 in every 18 requests using an AI crawler user-agent was fake, over 2 million spoofed requests a day. OpenAI’s ChatGPT-User fetcher was the most spoofed, at a ratio HUMAN describes as “one in every six requests.”
HUMAN’s data also shows why “it came from the right cloud” is not verification. The more sophisticated spoofers sent their fake ChatGPT-User traffic from the same Microsoft Azure network (ASN 8075) that real ChatGPT-User uses, and HUMAN observed over 14,000 requests from one address that sat close to, but outside, OpenAI’s published ranges. A rule that allows the whole network lets them straight through. Match the exact published blocks, not the provider.
The good news is the verification pattern transfers directly. OpenAI publishes IP ranges for its crawlers as JSON, mirroring Google’s approach:
GPTBot, the training crawler, publishes its ranges (see what GPTBot is for the full profile and how it differs from OpenAI’s other agents).OAI-SearchBot, which powers ChatGPT search results, publishes a separate range.ChatGPT-User, the user-triggered fetcher, publishes its own.
To verify a claimed GPTBot hit, match the source IP against OpenAI’s published GPTBot ranges, the same way you match Googlebot against common-crawlers.json. A request with the GPTBot user-agent from an IP outside those ranges is spoofed. And because OpenAI, Google, and others are adopting Web Bot Auth, signature verification is becoming the single method that covers every signing agent at once, instead of a per-operator scramble for IP lists. OpenAI’s ChatGPT agent already signs its requests, and as HUMAN notes, because those requests are signed and validated against OpenAI’s keys, they cannot be spoofed the way crawler user-agents are. If you want to see which AI systems are actually reaching your pages and sending referral traffic, that is a question of tracking AI referral traffic on top of verifying the crawlers themselves.
This is the layer of the web Lume is built to measure. Lume captures the non-human half of your traffic server-side and cryptographically verifies which agents are genuine, combining Web Bot Auth, reverse DNS, and published IP ranges so a spoofed GPTBot is flagged rather than counted as the real thing. The same engine confirms real Googlebot, so your analytics reflect verified agents rather than whatever a user-agent header claimed.
Putting it into practice at scale
Verifying one log line by hand is easy; verifying millions is an engineering task. A few principles keep it reliable.
- Cache verification results by IP. Reverse DNS is a network round-trip. Verify an IP once, cache the verdict with a sensible TTL (an hour is common), and reuse it. Do not look up the same Googlebot IP on every hit.
- Refresh IP-range files on a schedule. Google and OpenAI both change their published ranges. Pull the JSON daily, and alert if a file fails to load rather than silently falling back to a stale copy.
- Verify at the point of decision. If you serve or block based on bot identity, verify before you act, not in a nightly batch. A batch job tells you yesterday’s scrapers; inline verification stops today’s.
- Log the verdict, not just the user-agent. Store whether each request was verified, spoofed, or unverifiable. That turns your logs into an audit trail and makes spoofing patterns visible. Learning to detect bot traffic in your logs starts with recording an honest verdict on every hit.
Done well, verification changes what your data means. A “Googlebot: 40,000 requests” line in an unverified report is a guess. The same line after verification is a fact, split into real Googlebot and the impostors that were riding its name.
FAQ
Is checking the user-agent enough to verify Googlebot?
No. The user-agent is a self-reported header that any client can set to any value, so it identifies nothing on its own. Fakes even copy the version number: in one 2026 study, 94% of fake Googlebot requests claimed the real Googlebot’s current version. Verification requires confirming the request came from Google’s network via forward-confirmed reverse DNS or a published IP range, and ideally a Web Bot Auth signature. Treat the user-agent as a claim to be checked, never as proof.
What is FCrDNS?
FCrDNS stands for forward-confirmed reverse DNS. You run a reverse DNS lookup on the source IP to get a hostname, confirm the hostname belongs to the operator’s domain (such as googlebot.com), then run a forward DNS lookup on that hostname and confirm it resolves back to the original IP. Both directions must agree, which is what stops a spoofer who controls only their own reverse record.
Which domains should a real Googlebot IP resolve to?
A genuine Googlebot IP resolves to a hostname ending in googlebot.com or google.com. Googlebot itself typically returns a crawl-*.googlebot.com hostname. Do not treat googleusercontent.com as proof: Google Cloud virtual machines use that suffix too, so anyone renting one can pass a check that accepts it. If the reverse DNS lands on any other domain, or the forward lookup does not return the original IP, the request is not authentic Googlebot.
How common is fake Googlebot?
Common, and rising. On the typical news and media site in a 2026 study by Centinel Analytica, 1 in 7 requests calling itself Googlebot did not come from Google. Rates vary widely by site: a university site in a 2025 academic study, which already filtered bots by IP, saw 33 suspicious Googlebot requests out of 9,103. Google’s Gary Illyes, asked how common impersonation is, answered “Unfortunately very.”
Can I verify Googlebot without doing a DNS lookup on every request?
Yes. Google publishes its crawler IP ranges as JSON (Googlebot lives in common-crawlers.json), so you can check whether a source IP falls inside a published CIDR block without a DNS round-trip. Refresh the file regularly because the ranges change. Many teams combine both: IP-range matching for speed, reverse DNS as a fallback and cross-check.
How do I catch a spoofed GPTBot?
Match the source IP against OpenAI’s published crawler IP ranges, exactly as you match Googlebot against Google’s ranges. A request carrying the GPTBot user-agent from an IP outside those ranges is spoofed. Because OpenAI and others are adopting Web Bot Auth, verifying a cryptographic signature is becoming the most durable way to confirm any AI agent, GPTBot included.
Does Web Bot Auth replace reverse DNS and IP ranges?
Not yet, but it is the strongest layer where it is available. Web Bot Auth proves a request is cryptographically signed by the agent it claims to be, which holds even when the agent runs from cloud IPs it never published. Until every agent signs, use it alongside reverse DNS and published IP ranges rather than instead of them.
