What is a web application firewall (WAF)?

By Nik Andreev, Founder & SEO specialist · Last reviewed · Editorial policy

A web application firewall (WAF) inspects HTTP requests against a set of rules and blocks, challenges or allows them before they reach your application. It is usually deployed as a reverse proxy or at a CDN. Unlike robots.txt, it actually enforces access, which also means it can block good crawlers by mistake.

Full guide How to tell if Cloudflare or your firewall is blocking AI crawlers A diagnostic flow to find which layer stops GPTBot, ClaudeBot or OAI-SearchBot, and when blocking is the right call. Read the guide →

At a glance

Type
Application-layer (HTTP) firewall, often a reverse proxy or CDN feature
What it checks
Requests against rules: IPs, paths, headers, user agents, rates, attack patterns
Typical targets
Attacks such as SQL injection and cross-site scripting, plus unwanted bots
Actions
Block, challenge, rate-limit, log or allow
Versus robots.txt
Enforced. robots.txt is only a request
Risk for SEO and AI
Rules can block real crawlers, often with a 403 or a challenge page

How to spot it

  • From the bot’s side: a WAF block usually shows up as a 403, a 429 or a challenge page instead of your content.
  • In your logs: blocks at a CDN or edge WAF appear in that provider’s event logs, not in your origin server log. A crawler that seems to have stopped visiting may simply be getting blocked upstream.
  • The safe rule: match bots on verified identity (published IP ranges, FCrDNS or signatures), not on the user-agent string alone.

Other access controls

robots.txt Asks crawlers to stay out of paths. Well-behaved bots follow it, others do not. Verified bot A bot whose identity has been checked. Good WAF rules allow these by identity, not by name. Published IP ranges The IP lists operators publish so you can allow or block their bots reliably.

Quick answers

Should I block bots by user agent in my WAF?

Only as a first pass. User agents are easy to fake, so allow or block well-known bots by verified identity, such as published IP ranges or reverse DNS.

How do I know if my WAF blocks a crawler?

Look at your WAF or CDN event logs for the bot's user agent and IPs, and check for 403 or challenge responses. Requests blocked at the edge never reach your origin logs.

Sources