What is robots.txt?

By Nik Andreev, Founder & SEO specialist · Last reviewed · Editorial policy

robots.txt is a plain-text file served at /robots.txt that tells crawlers which paths they may fetch, using User-agent, Allow and Disallow rules. It is standardized as RFC 9309. Well-behaved crawlers follow it, but it is a request, not access control: nothing stops a bot that ignores it.

At a glance

Location
/robots.txt at the top level of the host, lowercase
Standard
RFC 9309 (Robots Exclusion Protocol), IETF Proposed Standard
Main rules
User-agent, Allow, Disallow
Is it access control?
No. RFC 9309 says the rules are not a form of access authorization
Caching
Crawlers should not use a cached copy for more than 24 hours
If it returns 5xx
Crawlers must assume everything is disallowed

How it works

  • Groups: each group starts with one or more User-agent lines naming a product token, followed by Allow and Disallow path rules. Token matching is case-insensitive.
  • Most specific wins: a crawler uses the group that matches its token, and falls back to User-agent: * only if none matches. Within a group, the longest matching path rule applies.
  • What it cannot do: it does not hide a URL, protect private content or stop a bot that chooses to ignore it. For that you need authentication or a WAF rule.
User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/

Other ways to control bot access

Web application firewall (WAF) Actually blocks requests. Use it when a bot ignores robots.txt. llms.txt A proposed file that points AI models to your best content. It grants or blocks nothing. Google-Extended A robots.txt token for Gemini training. It is not a separate crawler.

Quick answers

Does robots.txt keep a page out of Google?

No. Google says a blocked URL can still appear in results without a description. Use noindex, and let Google crawl the page to see it.

Do AI bots obey robots.txt?

Crawlers like GPTBot and OAI-SearchBot do. OpenAI says rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person triggered the fetch.

Can I use robots.txt to stop bad bots?

No. A bot that ignores it is not stopped. Blocking needs a firewall or server rule.

Sources