How to spot server throttling and inconsistent status codes blocking your crawlers

Server throttling and inconsistent status codes are among the most damaging problems in technical SEO precisely because they are invisible from a browser. Your homepage loads fine, your team sees 200s all day, and yet Googlebot, GPTBot, and ClaudeBot are quietly collecting 403s, 429s, and 503s on the same URLs.

The traffic decline shows up weeks later as pages slipping out of the index or vanishing from AI answers, long after the root cause has scrolled off the top of your logs. The gap between “works in my browser” and “works for the crawler” is where rankings go to die.

This is a diagnosable, fixable class of problem. The key is to look at your traffic the way a crawler experiences it, not the way a person does.

What server throttling and inconsistent status codes actually are

Server throttling is the deliberate or accidental limiting of how many requests a client can make in a given window, usually enforced by returning a 429 or 503 once a threshold is crossed. Inconsistent status codes are the related symptom: the same URL returning different HTTP responses depending on who is asking, when they ask, or how many times they have asked recently.

Both problems share a root trait. The response a client receives is a function of more than the URL. It depends on the requester’s user agent, IP address and reputation, request rate, geography, and whether an edge security layer decided to issue a challenge. A person on a residential connection sails through. A crawler hitting a hundred URLs a minute from a datacenter IP range trips a rule the person never touches.

Throttling is not inherently wrong. Rate limiting protects origin servers from being overwhelmed, and returning a 429 with a Retry-After header is the correct way to tell an aggressive client to slow down. The damage comes from two failure modes: limits set so low that legitimate crawlers cannot make progress, and blanket security rules that treat every non-browser client as hostile. You may have never intended to block Googlebot, but a web application firewall rule written to stop scrapers will happily return 403 to any client that does not send browser-like headers, and honest crawlers do not. The next sections show how to see this happening and trace it to its source.

Which status codes crawlers care about and how they react

Crawlers interpret status codes as instructions, and the difference between two adjacent codes can be the difference between a temporary slowdown and permanent removal from the index. Google’s documented behavior draws a hard line between codes that mean “come back later” and codes that mean “this content is gone.”

The critical detail most site owners miss: a 403 and a 429 look similar in a log file but are treated in completely opposite ways. Google treats 403 and 401 the same as a 404, a signal that the content does not exist, and previously indexed pages get removed. A 429 is read as an overload signal and only slows crawling. Using 403 to fend off aggressive bots therefore risks deindexing the very pages you want ranked.

Status code	What it means to a crawler	How Googlebot / AI crawlers react	Correct use / fix
200 OK	Content served successfully	Normal crawl and indexing; a 200 that returns an error or empty page becomes a “soft 404” and is dropped	Return 200 only for real content
403 Forbidden	Access denied, treated as “content does not exist”	Google treats it like a 404 and removes the URL from the index; a hard stop, not a slowdown	Never use to throttle; reserve for genuinely private content
429 Too Many Requests	Server overload, slow down	Google treats it as a server error and reduces crawl rate; indexed URLs are kept at first but eventually dropped if it persists	Pair with a Retry-After header; raise the limit for verified crawlers
503 Service Unavailable	Temporary outage, retry later	Crawl rate reduced; Google suggests a couple of hours or 1-2 days at most, and the same URL erroring for multiple days may be dropped	Use for planned maintenance only, with Retry-After
500 / 502 / 5xx	Server error	Crawl rate cut proportionally to affected URLs; sustained errors shrink crawl budget and deindex	Fix the origin; do not let errors persist for days

Google documents its behavior in detail. It honors a Retry-After header on 503 and 429 responses, as either a delay in seconds or an absolute date, and when it sees a significant number of 500, 503, or 429 responses it reduces crawling “across the whole hostname,” including the URLs that still return content. AI crawler operators publish far less about how they handle errors and retries, and their revisit schedules are opaque, so a stretch of 429s or 503s can mean a page simply never makes it into the model’s training set or the retrieval index behind a live answer.

Two operational facts anchor everything that follows. First, 403 causes removal while 429 causes slowdown, so the code you choose has strategic consequences, the same way choosing the right redirect code does. Second, duration is decisive: a few hours of errors is survivable, but Google’s guidance on throttling is to serve 500, 503, or 429 only for “a couple of hours, or 1-2 days,” and it warns that if Googlebot sees these codes “on the same URL for multiple days, the URL may be dropped from Google’s index.” Its crawler documentation puts the same rule plainly: already indexed URLs are preserved at first, “but eventually dropped.” The goal of detection is to catch these patterns inside that window, not after.

The robots.txt trap: when the block hits robots.txt itself

The most damaging place for an intermittent block is the robots.txt file, because Google reads its status code as an instruction about your whole site. Google’s robots.txt documentation spells out the consequences:

• A 5xx or a timeout stops crawling. If robots.txt returns a server error, “For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file.” After that, Google uses the last good version for the next 30 days. DNS failures, timeouts, and reset connections all count as server errors.
• A 403 removes your rules. “Google’s crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn’t exist.” A WAF rule that serves 403 to bots on /robots.txt does not block Googlebot; it tells Google your site has no crawl restrictions at all, so disallowed sections become crawlable.
• Fixes take time to land. Google generally caches robots.txt for up to 24 hours, and longer when refreshing fails because of timeouts or 5xx errors. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt update.

This is not a rare edge case at web scale. In the HTTP Archive’s 2025 Web Almanac, 85% of robots.txt requests returned a 200, while “Timeouts are ~1.0%, 403 responses are ~0.5%, and 5xx are ~0.1%.” Monitor the status code your robots.txt returns to verified crawlers as closely as you monitor your homepage.

How to detect intermittent crawler blocking

You detect intermittent crawler blocking by segmenting your status codes by client, because the problem is invisible in any aggregate that mixes human and bot traffic. A 99.4% success rate across all requests can hide a 40% failure rate for Googlebot, since humans vastly outnumber crawlers and drown out the signal.

Start with the data sources that show what crawlers actually received, not what a browser gets on demand.

• Server-side access logs. The ground truth. Every request, its user agent, source IP, timestamp, and the exact status code returned. This is the only place that captures what the crawler saw, because a browser retest happens later and under different conditions.
• Google Search Console. The Crawl Stats report (Settings, then Crawl stats) breaks Googlebot fetches down by response code and flags spikes in 403, 429, and 5xx. The Pages report surfaces “Blocked due to access forbidden (403)” and “Server error (5xx)” as indexing reasons.
• Edge and CDN analytics. If a WAF or CDN sits in front of your origin, its logs show challenges and blocks that never reach your application logs at all, which is exactly where challenge-based blocking hides.
• Dedicated bot analytics. A platform like Lume captures every request server-side at the edge and breaks traffic down by status class, so you can see the non-human half of your traffic on its own terms instead of buried in a blended total.

Once you have the data, look for these specific patterns.

1. Status codes that vary by user agent for the same URL. If /pricing returns 200 to Chrome and 403 to Googlebot, that is a user-agent or bot rule, not a broken page. This is the single most diagnostic signal, and it is why Lume includes an inconsistent-status detector that flags exactly this: the same URL returning different status codes to different agents.
2. Clusters of 429 or 503 that track request rate. Plot bot status codes against requests per minute. If failures begin above a threshold, such as 200 requests a minute during a traffic spike, you have found your rate limit.
3. An “agents-blocked” signal on important crawlers. Watch specifically for verified Googlebot, GPTBot, or ClaudeBot receiving 403, 429, or 503. Lume surfaces this as an explicit agents-blocked signal, so a throttled crawler does not stay buried in the noise.
4. Bursts that correlate with deploys or rule changes. A block that starts the hour a WAF rule shipped is a strong causal lead.

To reproduce a suspected block on demand, request the URL while spoofing the crawler’s user agent and compare it to a normal browser request:

curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -I https://example.com/pricing
curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" -I https://example.com/pricing

If the first returns 403 and the second 200, the server is discriminating by user agent. One caveat: a sophisticated WAF also checks IP reputation and JavaScript execution, so a curl test from your office IP may pass even when the real Googlebot from a datacenter range is blocked. Absence of a block in a curl test is not proof of innocence. With reliable detection in place, the next task is tracing each pattern to its root cause.

Root causes: why the same URL returns 200 to a browser but 403, 429, or 503 to a crawler

The same URL returns different codes to different clients because the server’s response depends on request attributes that differ between a human and a crawler. Understanding which attribute triggered the block tells you exactly which system to fix.

There are five recurring root causes, roughly in order of how often they bite.

• User-agent rules. A WAF or application rule blocks or challenges clients whose user agent is not a known browser, or matches a “bad bot” list too broadly. Honest crawlers announce themselves as GPTBot or Googlebot, which is precisely what a naive block rule catches. The browser, sending a normal Chrome user agent, is waved through.
• IP reputation and datacenter ranges. Crawlers operate from cloud and datacenter IP ranges that carry lower trust scores than residential IPs. A rule that challenges or blocks low-reputation IPs hits crawlers and spares humans on home broadband.
• Rate limiting and throttling. A per-IP or per-user-agent limit returns 429 or 503 once a crawler exceeds it. A person clicking through a handful of pages never approaches the threshold; a crawler fetching hundreds of URLs a minute crosses it in seconds. During traffic spikes, an origin under load may start shedding crawler requests first.
• JavaScript challenges. Managed challenges present a JavaScript or interactive test instead of an outright block. A real browser solves it silently, but a crawler that cannot execute the challenge receives what is functionally a block, often a 403 with a cf-mitigated: challenge header. For a non-rendering crawler, a challenge and a block are the same event.
• Vendor defaults and managed bot rules. Security and CDN products ship with bot rules you did not write. In July 2025, Cloudflare changed its default to block AI crawlers unless they pay creators for their content. This kind of blocking is invisible in robots.txt: a 2025 academic study inferred that 1,433 (14%) of the top 10,000 sites in October 2024 actively blocked two of Anthropic’s AI crawlers at the server or CDN level, and only 35 (2%) of those also restricted them in robots.txt. To trace which edge rule is responsible, follow how to tell if Cloudflare or your firewall is blocking AI crawlers. Google noticed the same pattern from its side in 2023: “an uptick in website owners and some content delivery networks (CDNs) attempting to use 404 and other 4xx client errors (but not 429) to attempt to reduce Googlebot’s crawl rate.”
• Geographic and edge rules. Rules that restrict or challenge traffic by region catch crawlers fetching from a data center in a blocked geography while local human visitors pass.

A concrete scenario ties these together. A news site enables an aggressive bot-mitigation rule ahead of a product launch. During the resulting traffic spike, the WAF starts returning 429 to any client exceeding 200 requests a minute. Googlebot, crawling fresh articles at exactly that moment, hits the ceiling and backs off. The new articles are never fetched, never indexed, and the site quietly loses the news cycle it built the content for, while every human visitor saw a fast, healthy site the entire time.

The pressure that leads teams to write these rules is real. SourceHut founder Drew DeVault wrote in March 2025 that he had spent “anywhere from 20-100% of my time in any given week mitigating hyper-aggressive LLM crawlers,” which arrived with random user agents from tens of thousands of mostly residential IP addresses. He also described the cost of blunt defenses: “our mitigations can’t always reliably distinguish users from bots.” Rules written under that pressure are exactly the ones that catch legitimate crawlers too.

The spoofing complication makes root-cause analysis harder. A meaningful share of traffic claiming to be from well-known AI crawlers is fake, and for some crawler identities the spoof rate is higher still. If you loosen a rule to let “Googlebot” through by user agent alone, you also let every impersonator through. This is why verification matters: you need to know whether a throttled request came from the real crawler or a spoofed one before you decide how to respond. Because Lume verifies agents cryptographically rather than trusting the user-agent string, it can tell a genuinely throttled Googlebot from a spoofed one, which changes whether the right fix is to raise a limit or tighten a rule. With causes identified, remediation becomes a matter of matching the fix to the mechanism.

How to fix throttling and inconsistent status codes

You fix crawler blocking by allowing verified legitimate crawlers ahead of your security rules and by choosing status codes that match your actual intent. Most of the damage comes from good rules applied in the wrong order or the wrong code chosen for the situation.

Work through these fixes in priority order.

1. Allowlist verified crawlers first in the rule chain. Put an explicit allow for verified search and AI crawlers before any block or challenge rule. If the allow does not come first, your later block and challenge rules catch Googlebot too. Crucially, allow by verification, not by user-agent string alone, or you open the door to spoofers.
2. Verify crawler identity properly. For Googlebot and other IP-publishing crawlers, use reverse DNS plus forward DNS: run a reverse lookup on the source IP, confirm the hostname belongs to the crawler, then a forward lookup to confirm it resolves back to the same IP. For automation, match the source IP against the crawler’s published IP range file. Never trust the user-agent header by itself.
3. Replace 403 throttling with 429. If you are using 403 to fend off aggressive bots, switch to 429. A 403 deindexes; a 429 only slows crawling. Always attach a Retry-After header so the crawler knows exactly how long to wait.
4. Raise or exempt rate limits for verified crawlers. Set crawler limits high enough for legitimate indexing to complete. Different AI crawlers self-pace differently, so a single tight global limit starves the ones that crawl in bursts. Exempt verified crawlers from the general per-IP limit entirely where you can.
5. Reserve 503 for genuine, short maintenance. Use 503 with Retry-After only for planned downtime, and keep it under one to two days. Beyond that, Google begins dropping the URLs.
6. Convert blocking challenges to non-blocking checks for good bots. A JavaScript challenge is a block for a non-rendering crawler. Route verified crawlers around managed challenges rather than serving them a test they cannot pass.
7. Provision for traffic spikes. Since throttling often triggers under load, make sure origin capacity and rate-limit ceilings account for peak crawler-plus-human traffic, so the crawler is not the first request shed.

After any change, confirm the fix from the crawler’s perspective, not just the browser’s. Re-run the user-agent curl comparison, watch Search Console Crawl Stats for the 403 or 429 rate to fall, and confirm in your bot analytics that verified crawlers are back to receiving 200s. A fix you cannot see in crawler-segmented data is a fix you have not verified. Once resolved, the remaining job is making sure it never silently recurs.

How to monitor so it does not silently return

You monitor for crawler blocking by continuously watching crawler-segmented status codes and alerting on any rise in 403, 429, or 5xx to verified important crawlers. Because this problem is invisible to human-focused monitoring, it will recur the moment a new WAF rule or deploy reintroduces it unless something is watching the bot slice specifically.

Build monitoring around a few durable practices.

• Track status class by verified agent, continuously. Aggregate uptime is not enough. You need the failure rate for Googlebot, GPTBot, and ClaudeBot on their own, verified so spoofed traffic does not distort the numbers. Lume’s Status Codes dashboard is built for exactly this cut: traffic by status class, an inconsistent-status detector, and an agents-blocked signal that fires when important verified crawlers start getting 403, 429, or 503.
• Alert on the inconsistency, not just the outage. The dangerous pattern is a single URL returning 200 to humans and 403 to crawlers, which never trips an uptime monitor. Alert when the same URL diverges by agent.
• Gate deploys and rule changes. Treat WAF and CDN rule changes as risky. After any security or infrastructure change, check crawler status codes before considering it done, since these changes are the most common cause of a fresh block.
• Watch Search Console as a lagging confirmation. Crawl Stats and the Pages report confirm indexing impact, but they lag by days. Use them to validate that a fix worked, not as your first line of detection.

Effective monitoring closes the loop that makes this problem so costly: the long, silent gap between the block starting and the traffic loss becoming visible. Catch the pattern in crawler-segmented data within hours, and it is a minor config fix. Catch it after deindexing, and it is a recovery project.

Frequently asked questions

Why does my site return 200 in the browser but 403 to Googlebot?

Because the server’s response depends on more than the URL. A browser sends a normal user agent from a trusted residential IP and, if challenged, silently executes JavaScript. Googlebot sends a bot user agent from a datacenter IP range and cannot solve interactive challenges. Any WAF, CDN, or application rule keyed on user agent, IP reputation, or JavaScript execution can therefore serve 200 to the human and 403 to the crawler for the identical URL.

Does returning 429 to crawlers hurt my SEO?

A short burst of 429s does not hurt and is the correct way to ask a crawler to slow down, especially with a Retry-After header. The harm comes from persistence. If important pages return 429 for more than roughly two days, Google starts dropping them from the index. Use 429 to smooth out load, not as a permanent wall, and exempt verified crawlers from tight limits so indexing can complete.

How long can my server return 503 before pages get deindexed?

Treat one to two days as the safe ceiling for 503 with a Retry-After header; that is the window Google itself suggests for emergency throttling. A few hours of downtime generally has no lasting effect on indexing. Beyond that, Google warns that a URL returning these codes for multiple days may be dropped from the index, and the longer it lasts, the more URLs are at risk. For anything longer than planned maintenance, fix the underlying availability rather than serving sustained 503s.

What happens if my robots.txt returns an error to Googlebot?

It depends on the error. A 5xx, timeout, or DNS failure makes Google stop crawling the site for the first 12 hours while it retries, then use the last good copy for the next 30 days. A 4xx other than 429, including a 403 from a firewall, makes Google treat the site as if it had no robots.txt, so everything becomes crawlable. Either way, the fix can take up to a day to take effect, because Google generally caches robots.txt for up to 24 hours.

How do I tell a real Googlebot from a spoofed one?

Do not trust the user-agent string, since a meaningful share of traffic claiming to be a known AI crawler is fake. Verify by reverse DNS plus forward DNS: run a reverse lookup on the source IP, confirm the hostname belongs to the crawler’s domain, then a forward lookup to confirm it resolves back to the same IP. Alternatively, match the source IP against the crawler’s published IP range file. Cryptographic verification, which Lume performs on every request, removes the guesswork entirely.

Do AI crawlers like GPTBot and ClaudeBot react to throttling the same way Googlebot does?

Nobody outside the AI companies knows for sure, which is the problem. Google documents how it handles 429, 503, and Retry-After; the AI crawler operators publish much less about retries and backoff. Their revisit schedules are opaque, so sustained throttling can mean your content never enters the training set or the retrieval index behind a live answer, with no Search Console equivalent to warn you. Monitoring their status codes directly is the only reliable signal.