What is log file analysis?
Log file analysis is the practice of reading a website's server access logs to see how crawlers actually behave on the site. Every time any client requests a URL, the server writes a line to its access log. Filtering those lines down to search engine and AI crawlers shows, request by request, which pages Googlebot, Bingbot, GPTBot or ClaudeBot fetched, when, and what response they got.
A typical access log line in the widely used "combined" format looks like this (example values):
66.249.66.1 - - [01/Oct/2026:09:14:22 +0000] "GET /glossary/crawl-budget HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"From left to right: the requesting IP address, the time, the method and URL, the status code, the response size, the referrer, and the user agent. Log file analysis is built from those fields.
Logs are the only first-party record of crawler behavior. Search Console reports Google's view, analytics tools miss bots entirely (most bots never run the analytics JavaScript), and third-party tools estimate. The server log records what happened.
Why is log file analysis the only way to prove which AI bots visit?
Log file analysis is the only reliable way to prove which AI crawlers visit a site, because AI companies do not offer a Search Console equivalent for their bots. No dashboard from OpenAI, Anthropic or Perplexity shows a site owner which of their pages GPTBot, ClaudeBot or PerplexityBot fetched. The access log does.
That matters for AI search work. A site cannot be cited by an answer engine whose crawler never reaches its pages. Logs answer practical questions that robots.txt alone cannot:
- Is PerplexityBot reaching the pages you want cited, or being blocked by a firewall or CDN rule?
- Did GPTBot stop visiting after a robots.txt change, as intended?
- Which sections do AI crawlers request most, and do they get 200 responses or errors?
- Are requests claiming to be ClaudeBot actually from Anthropic?
The last question is the one most reports skip, and it changes the answer to all the others.
How do you verify Googlebot in log files?
Verify Googlebot by checking the request's IP address, not its user agent. Google warns that the user agent Googlebot sends is often spoofed by other crawlers [2], so a log line that says "Googlebot" proves nothing on its own.
Google documents two methods [1]:
- DNS lookup. Run a reverse DNS lookup on the IP address from the log, confirm the hostname ends in
googlebot.com,google.comorgoogleusercontent.com, then run a forward DNS lookup on that hostname and confirm it returns the original IP:
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1- IP range matching. Compare the IP address against the IP ranges Google publishes as JSON files, split by crawler type (common crawlers, special-case crawlers and user-triggered fetchers) [1]. For log files with millions of lines, this is the practical method: load the ranges once and match every IP automatically.
Google moved its crawler IP range files to a new location in 2026 [7], so scripts that fetch the old file paths should be checked against the current verification page [1].
How do you verify AI crawlers in log files?
Verify AI crawlers the same way as Googlebot: match the IP address against the ranges the operator publishes, and treat any request that falls outside them as an impostor. OpenAI, Anthropic and Perplexity each publish IP lists for their crawlers; the file locations are listed in the GPTBot, ClaudeBot and PerplexityBot entries.
Three cautions apply to AI crawler logs:
- Each company runs several bots. Training crawlers, search crawlers and user-triggered fetchers have different user agents and different robots.txt behavior. Count them separately. An AI crawler that fetches a page because a user asked a question is not the same signal as a training crawl.
- Spoofing is common. Scrapers copy well-known bot names. Unverified "GPTBot" traffic can make a site look heavily crawled by OpenAI when it is not.
- Some crawlers publish no IP list. For those, the log can show a user agent claim but cannot prove it. Report that traffic as unverified rather than counting it.
Search Insider recommends reporting AI crawler numbers only after IP verification. Unverified counts can overstate AI crawler activity, and decisions about a crawler allowlist should not rest on them.
What can log file analysis show?
Log file analysis shows how crawlers spend their requests, which is what most technical SEO decisions need. The common findings:
| What to look for | What it means |
|---|---|
| Googlebot requests to parameter, filter or internal search URLs | Crawl budget waste [3] |
| Repeated requests to URLs returning 200 with empty content | Likely soft 404s still being crawled [3] |
| Spikes in 5xx or 429 responses to bots | Google's crawlers slow down when they get these [4] |
| Long chains of 301 responses | Redirect chains that cost extra requests [3] |
| Heavy crawling of CSS, JavaScript and image files | Resources count toward crawl budget too [5] |
| Important pages never requested | Discovery problems, such as orphan pages or deep pages |
| URLs crawled that no internal link points to | Orphan pages Google still remembers |
| Mobile versus desktop Googlebot share | Most sites should see mostly the smartphone crawler [2] |
The most useful single measure is the share of verified Googlebot requests that land on URLs you want indexed. If most requests go elsewhere, the site has a crawl efficiency problem regardless of its size.
What is the difference between log file analysis and the Crawl Stats report?
Log file analysis covers every request from every bot, while the Search Console Crawl Stats report covers only Google's crawlers and shows summaries and samples. Google launched the expanded Crawl Stats report in November 2020 [6], and for many sites it is enough to spot Googlebot response time problems and error spikes.
| Log file analysis | Crawl Stats report | |
|---|---|---|
| Bots covered | All, including Bing and AI crawlers | Google only |
| URL detail | Every request | Example URLs |
| Setup | Access to raw logs, a tool or script | None |
| Verification | You do it | Already Google's data |
| AI crawler visibility | Yes | No |
Use the Crawl Stats report for a quick health check and log files for anything that needs complete URL-level detail or covers non-Google bots.
How has log file analysis changed?
Log file analysis has shifted from a Google-only crawl audit to a way of measuring every crawler that matters for search, including AI. The dated changes below come from Google's own announcements; AI crawler dates are in each bot's entry.
2020: Search Console adds detailed crawl stats
In November 2020, Google announced new and improved crawl stats for sites in Search Console [6], which covered part of what SEOs had used logs for.
2023 onward: AI crawlers arrive
The arrival of named AI crawlers such as GPTBot gave site owners a new reason to read logs: robots.txt decisions about AI training and AI search need evidence of what those bots actually request. See the GPTBot entry for its introduction date.
2026: Google's IP range files move
In March 2026, Google announced a new location for its crawlers' IP range files [7], the files log analysis relies on for automatic verification.
How do you do log file analysis?
Do log file analysis in five steps:
- Get the logs. Ask the host or developer for raw access logs covering at least several weeks. On sites behind a CDN, the CDN's logs may hold requests that never reached the origin server, so check both.
- Filter to bots. Keep lines whose user agent names a crawler you care about.
- Verify. Match IPs to published ranges or run DNS checks [1]. Drop or separately label unverified lines.
- Join with a site crawl and the sitemap. Comparing crawled URLs against linked and listed URLs exposes waste, orphans and neglected sections.
- Segment and trend. Group by site section, status code and bot, and compare week over week, especially after releases, migrations or robots.txt changes.
Dedicated log analysis tools do this at scale, but a spreadsheet or short script is enough for a small site.
What are the limits of log file analysis?
Log file analysis shows what crawlers requested, not what they did with it. A Googlebot hit does not mean the page was indexed; Google crawls pages it may never index [8]. An AI crawler hit does not mean the page was used in training or cited in an answer. Logs also miss anything that never reaches the server, such as requests answered from a CDN cache when only origin logs are collected, and they are often kept for a short time by default. Treat logs as proof of access, and combine them with indexing and citation data for the rest of the picture.