What is a server log?

By Nik Andreev, Founder & SEO specialist · Last reviewed · Editorial policy

A server log, usually the access log, is the file where a web server records every request it processes, one line per request. A typical line includes the client IP, timestamp, request line, status code, response size, referrer and user agent. Because it captures requests that never run JavaScript, it is the most complete record of bot traffic.

Full guide Log file analysis for SEO and AI search optimization: a practical guide How to read your logs to see what search and AI crawlers do, protect crawl budget and spot problems. Read the guide →

At a glance

Also called
Access log, web server log
What it records
Every request the server processes
Common format
Combined Log Format (Apache), also the default style for Nginx
Key fields
IP, time, request line, status, bytes, Referer, User-Agent
What it misses
Requests blocked or served from cache at a CDN or WAF before reaching the server
Typical Nginx path
/var/log/nginx/access.log

How to spot it

  • Read a line: each field maps to one part of the request. The quoted user agent at the end is what bots use to name themselves.
  • Filter bots: search for tokens like Googlebot, GPTBot or ClaudeBot, then verify the IPs, because the names are easy to fake.
  • Mind the gaps: if you use a CDN, cached and blocked requests may only appear in the CDN’s logs, so check both.
66.249.66.1 - - [05/Nov/2026:09:12:44 +0000] "GET /pricing HTTP/1.1" 200 5123 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
From Lume's data

Traffic only server logs show

5 to 55%
of requests came from clients that never identified themselves, depending on the site

Method: Requests to sites that opted in to this anonymous analysis, August, September, October and November 2026 (to 4 November). Each site is summarized separately, then figures are rounded. Counts are requests claiming each bot name.

What you can find in a server log

Bot traffic Every crawler, fetcher and scraper request, including those analytics never sees. Crawl budget Which URLs search crawlers actually spend their requests on. User agent The self-reported client name logged with each request.

Quick answers

Why do server logs show more traffic than Google Analytics?

Logs record every request. Analytics only counts visits where its JavaScript tag runs, and it also filters known bots.

Does a user agent in my logs prove who sent the request?

No. It is self-reported. Verify the logged IP with published ranges or reverse DNS before trusting it.

Sources