What is GPTBot? A field guide to AI crawlers
On this page
GPTBot is OpenAI’s official web crawler for collecting public webpage text to train its generative AI models. It requests pages like any browser, but it announces itself with the user-agent token GPTBot and exists for one purpose: gathering training data. It does not index your site for search, and it does not fetch pages on behalf of a person typing into ChatGPT. Those are separate bots with separate names, and treating them as one crawler is the most common and most expensive mistake site owners make.
This guide defines GPTBot precisely, then maps it against the other major AI crawlers so you can decide what to allow, what to limit, and what you should never block by accident.
What GPTBot is and what it does
GPTBot is the crawler OpenAI runs to collect publicly accessible content that may be used to train its foundation models. It was introduced in August 2023. In OpenAI’s words, “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” Its user-agent string looks like this:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
The version number changes over time, so match on the GPTBot token rather than the whole string. OpenAI also says that when GPTBot fetches robots.txt it may add a robots.txt marker to that string, which helps when your logs do not record paths, and is one more reason a full-string match will miss real traffic. OpenAI publishes the IP ranges GPTBot crawls from at https://openai.com/gptbot.json, which matters for verification (more on that below). If you are new to how automated agents move through a site, our primer on how a web crawler actually works covers the mechanics.
The key point about GPTBot: it feeds model training only. Blocking it keeps your content out of future training data. It has no effect on whether ChatGPT can cite you in a live answer, because live answers come from different bots. One wrinkle from OpenAI’s docs: if you allow both GPTBot and OAI-SearchBot, “we may use the results from just one crawl for both use cases to avoid duplicative crawling,” so you may see fewer GPTBot hits than you expect.
How much does GPTBot actually crawl?
GPTBot is one of the busiest bots on the web, and it reaches more sites than its raw volume suggests. Three network operators, each measuring its own traffic, put numbers on it:
- Second only to Googlebot on Cloudflare. In Cloudflare’s 2025 Radar Year in Review, GPTBot originated about 7.5% of all Verified Bot traffic, behind Googlebot. Across categories, search engine crawlers made 40% of Verified Bot traffic and AI crawlers 20%.
- The top AI-only crawler. Looking only at AI crawlers, Cloudflare found GPTBot’s share rose from 5% to 30% between May 2024 and May 2025, while ByteDance’s Bytespider fell from 42% to 7%.
- Wide reach, modest volume. Fastly’s Q2 2025 Threat Insights Report found GPTBot crawled content from 95% of all websites crawled by AI bots, despite ranking third in crawl volume at 20%, behind Meta and Google.
Training still dominates. Cloudflare says training crawls reached as much as 7-8x search crawling and 32x user-action crawling at peak in 2025, a figure “heavily influenced by OpenAI’s GPTBot.” But the fastest growth is in the other two OpenAI bots. Cloudflare saw ChatGPT-User’s peak request volumes reach as much as 16x the start of the year, with a weekly pattern Cloudflare reads as growing use at schools and in the workplace, and OAI-SearchBot’s late-October spike was about 5x its January level. Fastly reported that ChatGPT-User and OAI-SearchBot made nearly 98% of all AI fetcher requests between mid-April and mid-July 2025, and in its January 2026 data the pair still made 97% of fetches, with 57% of fetcher requests going to content that could not be or had not yet been cached. In other words, the user-fetch bot tends to hit your origin, not your CDN cache.
GPTBot also gives back far less than it takes. Cloudflare’s crawl-to-refer ratio compares how often a platform crawls a site with how often it sends a visitor. In June 2025, OpenAI’s ratio was 1,700:1, against roughly 14:1 for Google, and over 2025 it reached as much as 3,700:1 in March. Anthropic’s was higher still, peaking at 500,000:1. Those ratios are what most blocking decisions are really about.
Training vs search-index vs user-fetch crawlers
AI crawlers fall into three jobs, and the job determines whether blocking one helps or hurts you.
A training crawler collects text to teach a model; Common Crawl’s CCBot, an open web-archive crawler whose dataset is widely reused to train models, is one example. Blocking it is a data-licensing decision, not a visibility decision.
A search-index crawler builds the index an AI assistant reads from when it answers a live query. Block this one and you disappear from that assistant’s answers, the same way blocking Googlebot removes you from Google.
A user-fetch crawler retrieves a single page in real time because a person asked the assistant to read that specific URL. It is closer to a human click than to bulk crawling.
Blocking the wrong category is silent. Nothing errors. You simply stop showing up in AI answers while believing you only opted out of training. Understanding the split is the foundation of any sensible policy, and it pairs directly with work on optimizing your site for AI search.
The major AI crawlers compared
The table below lists the crawlers you will see most in server logs, what each does, its user-agent token, and how to control it.
| Crawler | Operator | Job | User-agent token | How to allow or limit |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | GPTBot |
robots.txt honored |
| OAI-SearchBot | OpenAI | Search index for ChatGPT search | OAI-SearchBot |
Allow to stay in ChatGPT search |
| ChatGPT-User | OpenAI | User-triggered page fetch | ChatGPT-User |
Fires only on a user action; robots.txt may not apply |
| OAI-AdsBot | OpenAI | Checks landing pages submitted as ChatGPT ads | OAI-AdsBot |
Visits only pages submitted as ads |
| ClaudeBot | Anthropic | Training | ClaudeBot |
robots.txt honored |
| Claude-User | Anthropic | User-triggered page fetch | Claude-User |
Fires on a Claude user request |
| PerplexityBot | Perplexity | Search index | PerplexityBot |
Allow to appear in Perplexity |
| Google-Extended | Training control token | Google-Extended |
robots.txt token, not a crawler | |
| Applebot-Extended | Apple | Training control token | Applebot-Extended |
robots.txt token, not a crawler |
| CCBot | Common Crawl | Open crawl reused for training | CCBot |
robots.txt honored |
Two rows deserve a note. Google-Extended and Applebot-Extended are not crawlers you will see hitting your server. They are robots.txt tokens that tell Google and Apple whether they may use content they already fetched (with Googlebot and Applebot) for AI training. Disallowing Google-Extended stops Gemini training use without touching your Google Search ranking. That separation is deliberate, and it is why a blanket block is rarely what you want.
How to allow and limit each crawler
You control most of these through robots.txt, using the exact user-agent token. To keep your content out of OpenAI training while staying fully visible in ChatGPT’s search answers:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Apply the same logic per operator: disallow the training token, allow the search token. Do not confuse this with an llms.txt file: llms.txt is a curated Markdown map of your content, not an access control, and it does not allow or block anything. Note that user-fetch bots like ChatGPT-User and Claude-User act on a direct human request, so robots.txt crawl rules may not apply to them the way they do to bulk crawlers.
OpenAI makes that explicit for its own bot. Its crawler documentation describes robots.txt tags for OAI-SearchBot and GPTBot only, and says of ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.” If you need to control ChatGPT-User, robots.txt is not a reliable lever; OpenAI publishes its IP ranges at openai.com/chatgpt-user.json for firewall or edge rules instead.
Changes are not instant. OpenAI notes it can take about 24 hours from a robots.txt update for its search results to adjust, so verify with logs rather than assuming an immediate effect.
How many sites block GPTBot?
Fewer than the headlines suggest, but the share rises sharply with popularity and with news content:
- Most top sites have no robots.txt at all. In mid-2025 Cloudflare found only about 37% of the top 10,000 domains had a robots.txt file, and GPTBot was disallowed in just 7.8% of the files it found. Google-Extended appeared in only 5.6%, which suggests many sites that block GPTBot forget the other training tokens. Its 2025 Year in Review found AI crawlers, including GPTBot, ClaudeBot, and CCBot, had the most fully disallowed directives of any user agents.
- Popular sites block more. A September 2025 study of the top million websites by researchers Paul Bouchaud and Pedro Ramaciotti found 13.4% of the top 100,000 sites block GPTBot, nearly doubling to 25.2% among the top thousand.
- News sites block most. The same study found 34.2% of news outlets disallow GPTBot, rising to 55% for outlets with high factual reporting. A separate study of 3,369 reputable news sites and 710 misinformation sites found over 50% of the reputable sites disallow GPTBot.
Blocking is not free, either. An economics working paper by Hangcheng Zhao and Ron Berman (revised April 2026) found that large news publishers that blocked LLM crawlers saw “a 7% post-blocking decline in weekly visits” within six weeks, measured with SimilarWeb and Semrush data. That study covers blocking of LLM crawlers in general, not GPTBot alone, and large publishers specifically, so treat it as a caution rather than a forecast for your site.
Why user-agent strings cannot be trusted on their own
A user-agent string is just a text header the client chooses to send, and anything can send any string. A scraper can set its user-agent to GPTBot and impersonate OpenAI to slip past rules meant to allow legitimate crawlers, or to make its traffic look benign in your analytics. Nothing in the request itself proves the sender is who it claims to be. Fastly’s 2026 bot report describes the cost of getting this wrong: a bot “falsely declaring itself to be ChatGPT” can lead a team to build policy around a presumed “ChatGPT problem” that is in fact a malicious bot problem.
Real verification confirms identity independent of the header. The two standard methods are checking the request’s source IP against the operator’s published ranges (like OpenAI’s gptbot.json), and running a reverse-then-forward DNS lookup so the IP resolves back to the operator’s domain. Our walkthrough on verifying Googlebot shows the same reverse-DNS technique that applies to AI crawlers. Lume runs these checks continuously: it tracks every operator and user-agent token in a live agent catalog and verifies which requests are genuine, so a spoofed GPTBot is flagged rather than counted as the real thing.
Verification also turns raw logs into a decision you can act on. Once you can trust the label, you can see real AI crawler and referral activity in agent analytics and set robots.txt policy on facts instead of guesses.
Frequently asked questions
Is GPTBot the same as ChatGPT?
No. GPTBot is the crawler that gathers training data. ChatGPT is the product. When a ChatGPT user asks the assistant to open a specific link, that page is fetched by ChatGPT-User, not GPTBot.
Does blocking GPTBot remove me from ChatGPT search?
No. GPTBot only affects training data. ChatGPT’s search answers are built from OAI-SearchBot. Block GPTBot and you stay in ChatGPT search. Block OAI-SearchBot and OpenAI says you “will not be shown in ChatGPT search answers, though can still appear as navigational links.”
How much traffic does GPTBot send back?
Very little, directly. GPTBot is a training crawler, so it does not refer visitors at all; referrals come from people clicking links in ChatGPT answers. Measured across OpenAI as a whole, Cloudflare put the crawl-to-referral ratio at 1,700:1 in June 2025, compared with about 14:1 for Google.
Is GPTBot the same as OAI-SearchBot?
No. They are separate crawlers with separate tokens and separate jobs. GPTBot trains models; OAI-SearchBot indexes pages for ChatGPT’s live search feature.
How do I verify that a request is really GPTBot?
Check the request’s IP against OpenAI’s published ranges at openai.com/gptbot.json, or run a reverse DNS lookup that resolves back to an OpenAI domain and confirm it with a forward lookup. The user-agent string alone is not proof.
Should I block GPTBot?
That depends on your goal. Block it if you do not want your content used in future model training. Keep it allowed if you want your material represented in those models. For scale, a September 2025 study found 13.4% of the top 100,000 websites block GPTBot, and 25.2% of the top thousand. Either way, do not block OAI-SearchBot unless you also want out of ChatGPT’s search answers.
