What is Googlebot?

By Nik Andreev, Founder & SEO specialist · Last reviewed · Editorial policy

Googlebot is Google's main web crawler. It fetches pages so they can be indexed and shown in Google Search, follows robots.txt, and identifies itself with the Googlebot token in its user agent. Because that name is easy to fake, a request only counts as Googlebot once it passes reverse DNS or matches Google's published IP ranges.

Full guide How to verify Googlebot (and catch spoofed bots) Step-by-step reverse DNS and IP checks, commands, and the mistakes that reject real Google crawlers. Read the guide →

At a glance

Operator
Google
Type
Search engine crawler
User-agent token
Googlebot
Follows robots.txt
Yes
How to verify
Forward-confirmed reverse DNS, or Google's common-crawlers.json IP ranges
Blocking it means
Pages drop out of Google Search over time

How to spot it

  • In your logs: the user agent contains Googlebot. Match the token, not the full string, because Google uses several variants.
  • To verify it: run a reverse DNS lookup on the IP. The hostname must end in googlebot.com or google.com, and a forward lookup of that hostname must return the same IP. Anything else is user-agent spoofing.
  • At scale: check IPs against the ranges Google publishes in common-crawlers.json instead of running DNS lookups one by one.
host 66.249.66.1
# crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com
# must return 66.249.66.1

Related Google crawlers and tokens

Google-Extended A robots.txt token for Gemini training, not a crawler. Blocking it does not affect Search. FCrDNS The two-step DNS check Google documents for verifying its crawlers.

Quick answers

Does blocking Googlebot remove me from Google?

Over time, yes. Google cannot crawl what it is blocked from, so those pages drop out of Search.

Is a googleusercontent.com hostname proof of Googlebot?

No. Google uses that domain for other fetchers, so it does not prove a request came from Googlebot.

Sources