Blog
Bot traffic
Crawl budget: what it is and how to protect it
Crawl budget is the number of URLs bots crawl on your site in a set window. Learn what wastes it, how logs expose the leaks, and why AI crawlers now spend it too.
How to detect bot traffic in your logs
A server-log-first method to see and classify bot traffic yourself: the signals that identify a bot, how to separate good crawlers from scrapers, and why verification matters.
How to spot server throttling and inconsistent status codes blocking your crawlers
Intermittent 403, 429, 503 responses to Googlebot and AI crawlers quietly drop pages from search and AI answers. Here is how to detect, diagnose, and fix it.
Web crawler vs. web scraper: what's the difference?
A web crawler discovers pages by following links; a web scraper extracts specific data from them. Here is how to tell them apart in your server logs.
What is a web crawler? A modern taxonomy for 2026
A web crawler is an automated program that fetches web pages and follows links. Here is how crawlers work, the five types you now see, and how to tell them apart in your own logs.
Why Google Analytics doesn't show bot or AI crawler traffic (and how to see it)
GA4 only counts visitors that run its JavaScript tag, then silently drops the known bots that do. Here is what that hides, which bots still slip through, and how to measure your real bot share.
See the non-human half of your traffic.
Lume shows you every crawler, scraper and AI agent on your site, and verifies which ones are real. Set up in minutes.
Start for free →