robots.txt
A file at the site root that tells crawlers which paths they may or may not fetch. It controls crawling, not indexing.
robots.txt is a plain-text file at the root of a domain (/robots.txt) that gives crawlers instructions about which paths they are allowed to fetch. Well-behaved bots read it before crawling and respect its Allow and Disallow rules; it's also the conventional place to link your XML sitemap.
The single most misunderstood thing about robots.txt is that it controls crawling, not indexing. Disallowing a URL stops Google from fetching its content, but if other pages link to it, the URL can still be indexed — as a bare link with no snippet. Worse, if a page is blocked in robots.txt, Google never sees the noindex tag on it, so the block can actively prevent the very deindexing you wanted.
Use robots.txt to keep crawlers out of infinite or worthless URL spaces (faceted filters, internal search results, endless parameter combinations) and to protect crawl budget. To keep a page out of the index, let it be crawled and use a noindex meta tag instead.
Related terms
- Crawl Budget — The finite attention search engines spend crawling a site. Matters mainly for large sites; wasted on junk URLs, it starves real pages.
- Indexing — Storing a crawled page in a search engine's index so it can appear in results. Crawled is not the same as indexed.
- Noindex — A directive that tells search engines to keep a page out of their index. The page must stay crawlable for the tag to be seen.
- XML Sitemap — A machine-readable list of the canonical, indexable URLs on a site, submitted to search engines to aid discovery.