Crawl Efficiency: Why Pages Stay Out of the Index

Every fetch of a page costs the engine computation, so crawling is budgeted like any other expense. The HTML crawl rate — the share of crawl requests that return real HTML — is the direct measure of how effectively a site converts that budget into understanding. The documented reading is candid: "the higher the crawl rate, the more Google cares about the site." Crawl attention is allocated by a URL scheduler that weighs how often a URL changes and how important it is, populating different crawl layers from daily to near real-time.

That makes the crawl rate a feedback signal rather than a setting. A site that is cheap to fetch, consistent across mobile and desktop, and quick to respond is revisited more; a site that wastes fetches trains the scheduler to come back less. Server response times belong to the same account: fast responses "increase crawl efficiency," and a 499 error — the client closing the connection before the server answers — means the crawler received nothing and there is no retry.

Search Console states Crawled – currently not indexed and Discovered – currently not indexed as statuses; the doctrine reads them as verdicts. They mark pages the engine fetched (or found) and declined to index because the value did not justify the cost: thin or duplicated content, unnecessary pages, low-quality subdomains, mobile versions missing what desktop shows. The governing rule is the one that runs the whole cost account — the cost of ranking a site cannot exceed the cost of not ranking it.

"If ranking you is costlier than not ranking you, deindexing begins."

— Cost-of-retrieval doctrine

Two consequences follow. First, the status is not a penalty to appeal; it is an accounting entry to correct, by making the page worth processing or removing it. Second, the verdict is portfolio-level: a crawl profile full of declined pages shades the engine's judgment of the whole site, so pruning dead weight is not just tidiness — it changes how the next page is evaluated.

The documented targets are unambiguous, and every one of them is checkable in server logs:

One measurement habit outranks the rest: raw log files. Search Console is documented as exposing only around 30% of actual crawl data, so the log file is the honest record of what the crawler asked for, what it received, and where the budget went.

The waste list is short and specific:

• Query-parameterized filter URLs: faceted URLs that "force Google into a huge level of computational consumption for canonicalization and URL consolidation" — filtering systems should be non-crawlable.
• Temporary pages: publishing "too many pages that will be deleted or removed within two weeks" makes crawling the site "less worthwhile"; ephemeral content should declare itself with an unavailable_after robots meta.
• Low-quality subdomains: the engine keeps crawling them, "whose content quality also impacts Google's judgment of your website."
• Near-duplicates: duplicate detection exists to skip redundant fetching — it "speeds up the crawling and saves bandwidth" when the site does not fight it.
• Pages with no query behind them: documents with no search demand to trigger an index entry are processed and then shelved — pure cost, no return.

The discipline that keeps a site on the investment side of the ledger:

1. 1Prune the declined pages: review Crawled – currently not indexed and Discovered – currently not indexed reports, then remove or consolidate what does not earn its URL — documented cases report significant click gains after pruning.
2. 2Consolidate duplicates: one strong page beats three weak reflections; overlap beyond the threshold turns documents into near-duplicates the engine must choose between.
3. 3Close the parameter holes: make filtered views non-crawlable and keep URLs short, since short URLs are "easier to parse, resolve and request."
4. 4Shorten the crawl paths: consistent internal links from strong pages let the crawler reach every important document in few hops.
5. 5Watch the real logs: 499 errors, redirect chains and dead parameters show up in raw logs long before any dashboard.
6. 6Keep the content worth fetching: natural, original, semantically precise pages are the only yield that justifies the spend — gibberish and scaled output are the fastest way to lose the budget.