Log file analysis for SEO: what Googlebot and AI crawlers really crawl

Why log files matter for SEO

Every other crawl data source is a model or a summary. A site crawler such as OurSEO Tools, Screaming Frog or Sitebulb shows how a bot could move through your links. Search Console's Crawl Stats report shows Google's own totals and a sample of example URLs. The access log is different: it is the server's record of each request it actually answered, one line per request, with the exact URL, the status code it returned and the bot that asked.

That is why log file analysis answers questions nothing else can:

• Which sections does Googlebot spend its visits on, and which does it barely touch?
• Is Googlebot still fetching old URLs, parameter variations or redirects you thought were gone?
• When did Google last fetch your most important pages?
• Does Googlebot get the same status codes and response times as your users?
• Which AI crawlers visit, how often, and what do they fetch?

Log analysis matters most on large sites, sites with faceted navigation or many parameters, and sites after a migration. On a small site with a clean structure it is still a useful health check, but you will rarely find crawl budget problems there.

What Google officially says

What a log line contains

Most Apache and nginx servers write the combined log format by default. Each request becomes one line with the same fields in the same order. Here is a line that claims to be Googlebot smartphone requesting a sorted category page:

203.0.113.42 - - [28/Oct/2026:06:14:07 +0000] "GET /shoes/running/?sort=price HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Fields worth adding

Two extra fields make the log far more useful for SEO: the host (to separate subdomains served by the same server, since Google's crawl budget is per hostname) and the response time. In nginx, define a format and use it in your access_log line; $request_time is in seconds with millisecond resolution:

log_format seo '$remote_addr - $remote_user [$time_local] "$request" '
               '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
               '$host $request_time';
access_log /var/log/nginx/access.log seo;

In Apache, %D adds the time taken to serve the request in microseconds and %{Host}i adds the host:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %{Host}i %D" seo
CustomLog /var/log/apache2/access.log seo

Appending the new fields at the end keeps the standard field numbers ($1, $7, $9) unchanged, so existing scripts keep working and the last field ($NF) is the response time.

Getting your logs

Hosting and servers

• nginx: usually /var/log/nginx/access.log.
• Apache: usually /var/log/apache2/access.log (Debian, Ubuntu) or /var/log/httpd/access_log (RHEL, CentOS).
• IIS: usually under C:\inetpub\logs\LogFiles, in the W3C format with a header line that names the fields.
• Shared hosting: look for “Raw Access” or “Raw Access Logs” in cPanel or a log download area in your host's panel. If there is nothing, ask support.

Logs are rotated: older files get names such as access.log.1 and access.log.2.gz. Read compressed files with zcat or zgrep instead of unpacking them.

CDN and load balancer logs

If a CDN such as Cloudflare, Fastly, Amazon CloudFront or Akamai sits in front of your site, the origin server never sees requests answered from the CDN cache, and the IP in the origin log may be the CDN's, not the bot's. Two consequences:

• Use the CDN's own request logs (most CDNs offer log export, often on paid plans) or log the real client IP that the CDN forwards in a header such as X-Forwarded-For or Cloudflare's CF-Connecting-IP.
• Never verify bots by the IP in an origin log behind a CDN without checking which IP it is. You would be looking up the CDN's address.

The same applies to load balancers and reverse proxies: the last hop's log has the real client IP; the application servers behind it may not.

How much data you need

Many hosts keep logs for only a few days. One day of data shows you a snapshot; crawl frequency only becomes visible over weeks. Collect at least 30 days for a first analysis, longer for large sites, and set up an archive if your host deletes old files. Always save logs before and after a migration, so you can see how Googlebot moves from the old URLs to the new ones.

Privacy and GDPR

Access logs contain IP addresses, and under the GDPR IP addresses are personal data. Treat logs like any other personal data:

• Filter to bot traffic as early as possible; for SEO you need almost nothing about human visitors.
• Remove or anonymise visitor IPs before sending logs to an agency, a freelancer or a third-party tool.
• Limit who can access raw logs and set a retention period that matches your privacy policy.
• Bot IPs that resolve to a crawler operator's hostname are not visitor data, so you can keep them for verification.

Verifying real Googlebot

Scrapers and bad bots often send a Googlebot user agent, hoping you will not block them. If you count their hits as Google's, your analysis is wrong; if you block “Googlebot” by user agent to stop them, you may block the real one. So verify before you analyse.

Google offers two methods. “Manually: For one-off lookups, use command line tools. This method is sufficient for most use cases.” And: “Automatically: For large scale lookups, use an automatic solution to match a crawler's IP address against the list of published Google IP addresses.”

Manual check: reverse, then forward DNS

1. “Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.”
2. “Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.”
3. “Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name.”
4. “Verify that it's the same as the original accessing IP address from your logs.”

This is Google's own example from its documentation:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The forward step matters. Anyone who controls the reverse DNS of their own IP range can make it return a name ending in googlebot.com, but they cannot make Google's DNS point that name back to their IP.

Automatic check: Google's IP range files

For millions of log lines, DNS lookups are slow. Instead, download Google's IP range files and check whether each IP falls inside one of the ranges. “Note that the IP addresses in the JSON files are represented in CIDR format”, so use a library that understands CIDR, such as Python's ipaddress module. The files change over time, so refresh them regularly rather than hard-coding ranges.

Three kinds of Google requesters

Google groups its crawlers and fetchers into three categories, each with its own hostnames and IP file:

Do not be surprised by many different IPs. Google's crawlers run from many datacenters, and Google says: “Therefore, your logs may show visits from several IP addresses.” Also, “Google egresses primarily from IP addresses in the United States.” Seeing US IPs crawling a site aimed at another country is normal.

Which bots you will see

Google crawlers

Google publishes the list of crawlers that, in its words, are “more likely to show up in log files and that we've received questions about.” The ones that matter most for SEO analysis:

Two crawlers you will not find by name. Googlebot-News “doesn't have a separate HTTP request user agent string”; it crawls with the normal Googlebot strings. Google-Extended, the robots.txt token that controls use of your content for Google's Gemini models, also “doesn't have a separate HTTP request user agent string”. Blocking it in robots.txt does not change what you see in the log.

AI crawlers

AI companies run their own bots, and many sites now see them every day. The main tokens at the time of writing:

• OpenAI: GPTBot (collects content for model training), OAI-SearchBot (for ChatGPT search results), ChatGPT-User (fetches pages when a ChatGPT user asks for them).
• Anthropic: ClaudeBot.
• Perplexity: PerplexityBot.

In practice, AI crawlers often fetch the HTML of a page without its scripts and styles, and many do not run JavaScript. If your logs show AI bots fetching only HTML on a site that builds its content in the browser, they may be seeing little text. Separate training crawlers from search and user-triggered fetchers: blocking a training bot and blocking a search bot have very different effects on whether you appear in AI answers.

Key analyses

Work only with verified bot hits. Then look at the data from these angles.

Crawl share by section

Group Googlebot hits by the first folder of the path (/shoes/, /blog/, /search/) and compare each section's share of the crawl with its importance to your business. If your product categories get a small slice while internal search results or tag pages get a large one, Google is spending its time where you do not need it.

Status codes served to bots

Count the status codes returned to verified Googlebot. On a healthy site the large majority are 200 (and 304 if you support conditional requests). Watch for:

• 5xx and 429: errors and rate limiting make Google lower the crawl rate. Find the URLs and times they occurred.
• 3xx: a large share means internal links, sitemaps or canonicals still point at redirected URLs.
• 404 and 410: fine for pages that are truly gone. Google says a 404 “is a strong signal not to crawl that URL again”. A 404 on a page you want indexed is a bug.
• Different codes for bots and users: if bots get errors or blocks that users do not, check firewall, bot protection and rate-limit rules.

Crawl frequency and recency

For each important URL, find the date of the last verified Googlebot hit and how many hits it had in the period. Pages that change often and still go weeks without a visit need stronger internal links and accurate sitemap <lastmod> dates. Also chart hits per day: Google says “your Google crawl rate should be relatively stable over the time span of a week or two”, so a sudden drop or spike is worth explaining.

Important URLs never crawled

Take the URLs from your sitemap and your own crawl, and find the ones with no Googlebot hit in the whole period. These are often deep in pagination, linked only from JavaScript, or new pages with few internal links. They usually line up with “Discovered – currently not indexed” in Search Console.

Orphan URLs: crawled but not linked

The reverse set is just as useful: URLs Googlebot requests that are not in your crawl or your sitemap. They can be old URLs from before a redesign, orphan pages nobody links to anymore, parameter variations or URLs linked from other sites. Decide for each group: link it, redirect it, or return 404 or 410.

Wasted crawl on parameters and duplicates

Count Googlebot hits on URLs with a query string and group them by parameter name: ?sort=, ?color=, ?sessionid=, ?utm_source=. Many hits on many combinations of the same listing are the classic crawl trap. Fix it at the source: stop linking to the variations, keep tracking parameters out of internal links, and block crawling of unneeded combinations in robots.txt. A canonical tag helps Google consolidate duplicates for indexing, but it does not stop Google from requesting the URLs.

Redirects and chains

List the redirected URLs Googlebot keeps requesting and where they lead. Google advises: “Avoid long redirect chains, which have a negative effect on crawling.” Update internal links and sitemaps to point straight at the final URL, and make each old URL redirect to its destination in one hop.

Response time to bots

With response time in the log, find the slowest templates for Googlebot and the times of day they slow down. Slow responses reduce how much Google crawls. A template that is fast for cached visitors can still be slow for Googlebot if bots mostly request uncached pages.

Resources: JavaScript, CSS and images

Googlebot fetches the scripts, styles and images it needs to render pages. Check that these requests get 200 responses, not 403, 404 or robots.txt blocks. A blocked or failing script can mean Google renders a page without its content.

Pages that look like soft 404s

Logs cannot see page content, but the size field gives a hint. Many 200 responses of almost the same small size on different URLs (empty search results, “no products found” pages) are candidates for soft 404s. Google warns that “soft 404 pages will continue to be crawled, and waste your budget.” Confirm by opening a few and checking the Page indexing report.

AI crawler activity

Count hits per AI bot, which sections they fetch, and which status codes they get. Large volumes from one bot can cost you server resources; zero visits from a search-oriented AI bot may mean a firewall or robots.txt rule is blocking it. Verify the bot before you draw conclusions.

Command-line recipes

You do not need a special tool for a first analysis. These commands work on a combined-format log on Linux or macOS (and in Git Bash or WSL on Windows). Field numbers assume the standard format shown above.

1. Save the lines that claim to be Googlebot, from current and compressed logs:

cat access.log access.log.1 > all.log
zcat access.log.*.gz >> all.log
grep 'Googlebot' all.log > gbot-claimed.log

2. List the unique IPs and verify them with reverse DNS:

awk '{print $1}' gbot-claimed.log | sort -u > gbot-ips.txt
while read ip; do host "$ip"; done < gbot-ips.txt

Keep only IPs whose hostname ends in googlebot.com or google.com and resolves back to the same IP, then filter the log to those IPs (for example with grep -Ff verified-ips.txt, which matches each IP as a fixed string anywhere in the line). For large logs, match against the JSON IP ranges with a short script instead.

3. Status codes served to Googlebot:

awk '{print $9}' gbot.log | sort | uniq -c | sort -rn

4. Crawl share by top-level section:

awk '{split($7, p, "/"); print "/" p[2]}' gbot.log | sort | uniq -c | sort -rn | head -20

5. Most-crawled parameter URLs:

awk '$7 ~ /\?/ {print $7}' gbot.log | sort | uniq -c | sort -rn | head -20

6. Unique URLs crawled versus total hits:

wc -l < gbot.log
awk '{print $7}' gbot.log | sort -u | wc -l

7. Googlebot hits per day:

awk '{print substr($4, 2, 11)}' gbot.log | sort | uniq -c

The dates sort as text, which is fine within one month; across months, sort the output by date in a spreadsheet.

8. Last Googlebot visit to one URL:

grep '"GET /shoes/running/ ' gbot.log | tail -n 1

9. Slowest responses, if response time is the last field:

awk '{print $NF, $9, $7}' gbot.log | sort -rn | head -20

10. Requests per AI crawler:

grep -oE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot' all.log | sort | uniq -c | sort -rn

Here gbot.log is the verified file from step 2. For the never-crawled and orphan analyses, export the unique crawled paths to a file, export the URL list from your site crawl and sitemap, and compare the two lists with comm or a spreadsheet. For logs in the millions of lines, a log analysis tool or a database is easier than shell commands, but the questions stay the same.

Logs vs the Crawl Stats report

Search Console's Crawl Stats report (Settings → Crawl stats) is Google's own view of its crawling. It is a good first check and a good cross-check for your logs. Google notes: “This report is aimed at advanced users. If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail.”

Expect the numbers to differ. Google says: “Therefore you might see slight differences between your site's request logs and the numbers reported here.” Three common reasons:

• Requests that never reached you. For the “Page could not be reached” category, Google explains: “Because these requests never reached the server, these requests will not appear in your logs.”
• robots.txt outages. If the report shows far more crawls than your logs, Google may have been unable to fetch your robots.txt for too long. In that case it counts crawls it might have made but did not make.
• Missing log sources. A CDN answering from cache, another server behind the load balancer, or a subdomain logged elsewhere will make your logs show fewer hits than Google.

One more difference helps you: the report's purpose breakdown separates “Discovery: The URL requested was never crawled by Google before.” from “Refresh: A recrawl of a known page.” Your logs cannot tell you that directly, but over a long enough period you can see which URLs are new to Googlebot.

Turning findings into fixes

Avoid one tempting shortcut: adding noindex does not save crawling. Google explains that it “will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time.” After the fixes go live, repeat the analysis on fresh logs to confirm Googlebot has changed its behaviour.

Common mistakes

• ✕Trusting the user agentCounting every “Googlebot” line as Google. Verify by DNS or IP ranges first.
• ✕Analysing a single dayOne day cannot show crawl frequency or trends. Use weeks of data.
• ✕Ignoring the CDNOrigin logs behind a CDN miss cached hits and may show CDN IPs instead of bot IPs.
• ✕Counting hits instead of URLsA thousand hits on ten URLs is a very different story from a thousand hits on a thousand URLs.
• ✕Mixing hostsCrawl budget is per hostname. Separate www, subdomains and staging hosts.
• ✕Including test tool hitsGoogle-InspectionTool hits come from your own tests in URL Inspection and the Rich Results Test, not from Search crawling.
• ✕Blocking “Googlebot” by user agentTo stop a fake bot you block the real one too. Block by verified IP instead.
• ✕Sharing raw logsSending logs with visitor IPs to a third party without filtering or anonymising them.

FAQ

Sources

• Google — Verify requests from Google crawlers and fetchers
• Google — Overview of Google crawlers and fetchers (user agents)
• Google — List of Google's common crawlers
• Google — List of Google's special-case crawlers
• Google — Optimize your crawl budget
• Google Search Console Help — Crawl Stats report

Quotes checked against the official pages on 30 October 2026.