What your logs reveal that other tools can't
Crawl reports and site audits guess at how search engines move through your site. Your raw server access logs show it directly: every request Googlebot made, the exact URL, the status code it received, and when. That's ground truth, not a simulation.
This matters most on large sites where crawl budget is finite. If Googlebot spends its visits on parameter URLs, expired products, or redirect chains, your important pages get crawled less often and updates take longer to show in search.
- Which URLs Googlebot requests most and least often
- Status codes bots actually receive (200, 301, 404, 410, 5xx) versus what you assume
- Crawl frequency by directory, template, or page type
- Redirect chains and loops that waste crawl requests
- Orphan URLs bots find that aren't in your sitemap or internal links
- Spikes in 5xx errors that quietly suppress crawling
How we run a log analysis
We start by getting a representative window of access logs from your host, CDN, or server (Apache, Nginx, Cloudflare, and similar). We verify real Googlebot by reverse-DNS so spoofed bots don't skew the picture, then segment traffic by user agent, path, and status.
From there we map crawl behavior against your actual priorities and hand you a specific action list: URLs to block, redirects to collapse, error patterns to fix, and templates that need better internal linking.
- Verify Googlebot and Bingbot by reverse DNS, not user-agent string alone
- Segment crawl by desktop vs. mobile bot and by page template
- Compare crawled URLs against sitemap and analytics for gaps
- Flag high-crawl, low-value URLs eating budget
- Identify important pages that are rarely or never crawled
More on technical seo
Frequently asked questions
How much log data do you need?
Enough to see a real pattern, usually a few weeks to a couple of months of access logs depending on your traffic and how often the site changes. More history helps us spot trends rather than a single odd day. We'll tell you exactly what to export from your host or CDN.
Is this useful for a small site?
It's most valuable on large or complex sites where crawl budget is a real constraint. For a small brochure site, a standard technical audit usually surfaces the same issues faster. On a consultation we'll tell you honestly whether log analysis is worth it for your case.
Can't I just use Search Console's crawl stats?
Search Console's crawl stats report is a helpful summary, but it's aggregated and sampled. Raw logs give you the individual requests so you can trace exactly which URLs and templates are being crawled or skipped.