🤖 Crawl & indexation

robots.txt & Sitemap Tester

Enter any site URL. We fetch its robots.txt and sitemap.xml from our server, show you the crawl directives and declared sitemaps, and flag the query-string sitemap URLs that cause duplicate-content demotions.

How to test a robots.txt and sitemap.xml

Four steps, no signup. The whole check runs server-side.

1

Paste the site URL

Enter any site's homepage URL. You don't need the full path to robots.txt — we derive it from the host.

2

We fetch from our server

Our server requests robots.txt and every declared sitemap.xml, so the test isn't blocked by your browser or your own cache.

3

Read the directives and sitemaps

Review the parsed user-agent groups, disallow rules and the list of declared sitemaps, with a URL sample from each.

4

Fix what's flagged

Resolve any query-string sitemap URLs or blocked-but-indexed conflicts before a search engine demotes the duplicates.

What this tester checks

The same guardrails we run internally before publishing a single page.

📜

Crawl directives

Parses each user-agent group and its Allow / Disallow rules so you can see exactly what's blocked.

🗺️

Declared sitemaps

Lists every sitemap referenced in robots.txt, fetches each, and reports its type and URL count.

⚠️

Query-string bloat

Flags query-string sitemap URLs that seed near-duplicate pages and trigger duplicate-content demotions.

🔗

Fetch health

Catches sitemaps declared in robots.txt that error, redirect, or return non-XML — silent crawl killers.

Frequently asked questions

A single query-string URL in a sitemap can seed thousands of near-duplicate pages. Search engines crawl them, find the same content under different parameters, and demote the whole cluster for duplicate content. This tester flags them before that happens.
No. A missing or empty robots.txt simply allows all crawling. The risk is the opposite: an over-broad disallow that blocks pages you want indexed, or a sitemap declared in robots.txt that returns an error.
No. We fetch robots.txt and the sitemaps it declares, then read a sample of URLs from each. We don't recursively crawl every page, so the test is fast and won't load your server.
Yes, the tester is free with no signup. Programmatic access via a REST API is available on the paid managed plans.
⚙️ For developers

Run it in your pipeline

This tester runs entirely server-side — you submit a URL, our backend fetches and parses robots.txt and the declared sitemaps, then re-renders this page with the results. A programmatic REST API for CI/CD and bulk auditing is available on the paid managed plans. See plans →