Crawl Budget
The finite attention search engines spend crawling a site. Matters mainly for large sites; wasted on junk URLs, it starves real pages.
Crawl budget is the amount of crawling a search engine is willing and able to do on your site in a given period — a combination of how fast your server can respond without strain (crawl capacity) and how much the engine wants to crawl based on your site's demand and freshness (crawl demand). It is finite.
For small sites it rarely matters — Google will happily crawl a few hundred pages. It becomes a real constraint on large sites and ones that generate huge numbers of low-value URLs: faceted navigation, endless filter combinations, session parameters, and duplicate query-string variants. Every crawler request spent on junk is one not spent discovering or refreshing a page that matters.
You protect crawl budget by not creating worthless URL spaces in the first place, blocking the ones you can't avoid in robots.txt, keeping your sitemap to clean canonical URLs, and serving fast responses. A tidy URL space isn't just a crawl-efficiency issue — the same parameter sprawl that wastes budget is what spam systems read as near-duplicate bloat.
Related terms
- Indexing — Storing a crawled page in a search engine's index so it can appear in results. Crawled is not the same as indexed.
- Thin Content — Pages with little unique value — auto-generated, scraped, doorway or near-duplicate pages that spam systems demote.
- XML Sitemap — A machine-readable list of the canonical, indexable URLs on a site, submitted to search engines to aid discovery.
- robots.txt — A file at the site root that tells crawlers which paths they may or may not fetch. It controls crawling, not indexing.