Crawl budget is real and your clients are probably wasting it

Here is a number most agency clients have never seen: the percentage of Googlebot's daily crawl visits that go to pages their site will never rank for. Across the audits we have run at LazyMetrics, that number routinely falls between 30% and 60% on small-to-mid-size sites. Session IDs, filter URL permutations, internal search pages, 404s with live inbound links, redirect chains — all of it quietly drains the crawl budget that should be going to product pages, service pages, and blog content that can actually move rankings.

Crawl budget optimization is one of the highest-ROI technical SEO fixes an agency can deliver, and it is almost never reported clearly to clients. This post walks through what crawl budget actually is, the six most common ways sites waste it, how to find the problem, how to fix it, and — critically — how to frame the impact in a client report.

What crawl budget actually means

Googlebot does not crawl every URL on every site every day. Google allocates what it calls a crawl rate limit — the maximum crawl speed Googlebot is willing to sustain on a domain without overloading the server — and a crawl demand score, which reflects how popular and fresh a site appears to be. Together these two factors define a site's practical crawl budget: the finite number of URLs Googlebot will fetch in a given period.

For large authoritative sites, crawl budget is rarely a concern. For the typical 200–2,000 page agency client — especially e-commerce, local multi-location, or CMS-heavy sites — it is very real. Every crawl request Googlebot spends on a junk URL is one it does not spend on a money page. When money pages are crawled less frequently, they are re-indexed more slowly after content updates, and new pages take longer to appear in search results.

The goal of crawl budget optimization is simple: make every crawl request count. That means directing Googlebot's attention toward pages that are indexable, unique, and have ranking potential, and blocking or de-prioritizing everything else.

The six most common crawl budget killers

These are the patterns we see draining Googlebot crawl budget on agency client sites at a disproportionate rate. Any one of them can be the culprit; multiple together can make an already-crawl-constrained site nearly invisible to Google.

Faceted navigation and filter parameters

This is the single most common Googlebot crawl budget drain on e-commerce and directory sites. A site with 500 products and five filter dimensions can generate hundreds of thousands of unique URLs that are mostly duplicate or near-duplicate content. Without proper parameter handling, Googlebot will attempt to crawl them all.

Session IDs in URLs

Older e-commerce platforms — and some CMS-based sites — append session identifiers to URLs for authenticated states. Each session ID creates a technically unique URL, multiplying the crawlable URL space by every active user session. Googlebot sees thousands of "new" pages that are identical in content.

Internal search result pages

If a site's internal search function generates crawlable URLs and those URLs are linked to anywhere on the site — or appear in any sitemap — Googlebot will crawl them. These pages have no SEO value and consume crawl budget that should go elsewhere.

Thin, duplicate, and uncanonicalised pages

Pagination sequences without proper canonicals, tag/category archives on blog platforms, printer-friendly URL variants, and author archives all add pages to the crawlable URL space that duplicate existing content. Without a canonical pointing back to a definitive version, Googlebot treats each as a separate indexation candidate.

Broken pages with live inbound links

A 404 is a crawl budget dead end. Every time Googlebot follows an internal link to a URL that returns a 404, it spends a request and gets nothing indexable. Sites that have gone through multiple rounds of URL restructuring often have dozens or hundreds of these.

Redirect chains

A three-hop redirect chain — /old-page → /interim-page → /newer-page → /current-page — forces Googlebot to spend three crawl requests to reach one final URL. Google's own guidance notes that it may stop following redirect chains after a certain number of hops, meaning the final destination may not get crawled at all.

How to find the crawl budget waste

You do not need a full log file analysis to get started. Three sources will cover 90% of what you need.

1. Google Search Console — Coverage and Crawl Stats

In GSC, open the Coverage report and look at the "Excluded" tab. A large number of pages excluded as "Crawled — currently not indexed" or "Discovered — currently not indexed" are a signal that Googlebot is visiting pages that are not earning a spot in the index. Under Settings → Crawl Stats, check for a high ratio of crawl requests to indexed pages, and watch for disproportionate crawl of file types you would not expect (e.g. HTML pages that should be thin parameter URLs).

2. Screaming Frog crawl with parameter configuration

Run a full crawl of the site. In the URL filter, look for patterns like ?sessionid=, ?color=, and /search?q= in the URL column. Sort by number of inlinks to find which junk URLs are most deeply embedded in the site's internal link structure. Export all 4xx response codes and cross- reference against inlinks to find broken pages with live pointing URLs.

3. Log file analysis (if available)

Access logs filtered by the Googlebot user agent show exactly what was crawled and when. Import into a tool like Screaming Frog Log Analyser or a simple spreadsheet. Segment URLs by type (parameter URLs, pagination, money pages) and calculate what percentage of total Googlebot requests went to each segment. This is the most direct proof of crawl budget waste — and the most compelling data to show a client.

How to fix it

Fixes are straightforward once the waste patterns are identified. Priority order should be based on volume — fix the issue responsible for the most wasted crawls first.

Faceted navigation and filter parameters

Use Google Search Console's URL Parameters tool (where still available) to indicate that certain parameters do not change page content meaningfully. More reliably: add a noindex meta tag on filtered pages that have no search volume potential, and block the parameter patterns in robots.txt using the Disallow directive. Noindex prevents indexation; Disallow prevents crawl. Use both together for the largest crawl budget gains.

Near-duplicate and thin pages

Add rel="canonical" pointing to the definitive version of the content. For pagination, use canonical on page 2+ pointing to page 1 where consolidation is appropriate. For print or variant URLs, canonical back to the primary URL.

Broken internal links

Update every internal link pointing to a 404 to point to the correct live URL. Where no equivalent page exists, update the link to the nearest relevant live page. Do not rely on redirects as a substitute — fix the links at the source so Googlebot and users reach the destination in one hop.

Redirect chains

Flatten every redirect chain so each URL redirects directly to its final destination in a single 301. Any internal link still pointing to an intermediate URL in the chain should be updated to point to the final URL directly. Check and update sitemaps for the same — sitemaps should never list redirecting URLs.

Sitemap hygiene

Audit the XML sitemap and remove any URLs that: return a non-200 status code, are marked noindex, are parameter or filter variants, or are redirect URLs rather than canonical destinations. A clean sitemap is a direct signal to Googlebot about which URLs are worthy of crawl priority.

Measuring the impact

The primary measurement dashboard is GSC Crawl Stats under Settings. After fixes are deployed, watch for a shift in where Googlebot is spending its requests. You want to see: total crawl requests stable or slightly reduced (fewer junk URLs to visit), while the ratio of crawled-to-indexed pages improves.

On the Coverage report, track the "Excluded — Crawled, currently not indexed" count over time. A declining number of excluded pages alongside a stable or growing "Valid" indexed page count is the signature of improving crawl efficiency.

For ranking impact, segment the client's money pages in GSC Performance and watch for improved average position and increased impressions over the 6–10 weeks following the fix. When Googlebot is visiting money pages more frequently, content updates index faster and ranking positions become less volatile.

Also monitor average crawl response time in GSC Crawl Stats. A site that was serving thousands of dynamic filter pages will often show a drop in average server response time after those URLs are blocked — because the server is no longer generating them for Googlebot.

How to report this to clients

Most clients have no frame of reference for "crawl budget." The phrase means nothing to a business owner. The way to make this land is to translate it into the language of wasted spend and missed opportunity.

Frame it like this: "Googlebot was visiting your site X times per day. We found that 40% of those visits were going to pages that have no chance of ranking — filter URLs, old broken pages, redirect dead-ends. We fixed that. Now the same number of Googlebot visits are concentrated on your service pages, product pages, and blog content — the pages that drive leads and revenue."

If you have before/after log file data, show the visual: a pie chart of crawl requests by URL type before and after. The shift from "junk URLs" to "money URLs" in that chart is immediately understandable to any client, regardless of their SEO knowledge.

Follow it with the indexing freshness data — how quickly money pages were re-indexed after a content update before versus after the crawl efficiency fix. Tie that to the ranking movement on those pages over the following weeks. That chain of evidence — crawl efficiency → faster indexing → ranking improvement — is what makes the work defensible at renewal time.

The agencies that retain clients longest are the ones that connect technical work to business outcomes in language that sticks. Crawl budget optimization is one of the cleanest stories to tell — you fixed something invisible that was silently costing them rankings, and you can prove it. That is the kind of deliverable that earns renewals. See how LazyMetrics Execution helps your team ship these fixes at scale — or start a free trial and run your first crawl audit today.

Frequently asked questions