Which crawlers are on your site — and is AI sending anyone back?
Your logs are full of AI crawlers, search bots, and scrapers, and standard analytics hides all of it. Lume shows you exactly which are hitting you (and whether they’re really who they claim), and measures the human visitors ChatGPT, Perplexity, and other AI answers send back.
Sound familiar?
- GPTBot, ClaudeBot, and Google-Extended are all over my logs. Should I block them, or will I lose AI-search visibility?
- robots.txt says one thing, but the crawlers do another.
- Is ChatGPT or Perplexity actually sending me visitors? It’s all buried in “Direct.”
- Bots are burning my crawl budget hitting old 404s.
What Lume does for you
Every crawler and AI bot, identified
See search crawlers, AI-search crawlers, and AI-training scrapers split by operator, type, and page, so you can reason about Google-Extended vs. Googlebot instead of guessing.
Verified, not guessed
Know a crawler is really Googlebot or GPTBot and not a spoof (verification is live for 7 major operators) so your log analysis and allow/block calls rest on facts.
Measure the AI referral channel
See the humans ChatGPT, Claude, Gemini, and Perplexity send you, detected by referrer host and utm_source, even when the referrer is stripped and it would otherwise land in Direct.
Find where crawl budget leaks
See the status codes crawlers hit (including the pages returning the most 404s to bots) so you can fix what’s wasting their (and your) time.
Fits your workflow
Report traffic straight from WordPress, your CDN, or the edge. No rebuild required.
SEO FAQ
Is GPTBot training on my content or indexing it for search?
Lume classifies each crawler by type, so you can tell AI-training crawlers (e.g. Google-Extended) from AI-search and traditional search crawlers (Googlebot). You see the distinction and decide.
Should I block AI crawlers?
That’s your call. Lume doesn’t block. It gives you the data (who, how much, verified or not) so you can decide, then act via robots.txt or your CDN. Blocking AI-training crawlers won’t remove you from Google Search, but may affect AI-answer inclusion.
Is AI search actually sending me traffic?
Lume measures AI referral visits by referrer host and utm_source, catching the sessions client-side analytics quietly bucket as “Direct.”
How do I tell a real Googlebot from a fake one?
Lume verifies crawlers with forward-confirmed reverse DNS and published IP ranges, so a “Googlebot” coming from a datacenter IP that isn’t Google’s is flagged as unverified.
See which crawlers and which AI answers are working your site.
Free plan, one ingest token. Know your crawlers, and measure the AI traffic you’re already getting.
Start for free →