Crawl & indeksasi

robots.txt & Sitemap Tester

Ngetik URL situs mana wae. Kami nyandak robots.txt sareng sitemap.xml ti server kami, nunjukkeun anjeun kaputusan crawl sareng sitemap anu dinyatakeun, sareng ngabéjaan URL sitemap string query anu nyababkeun demotions konten duplikat.

Carane nyoba robots.txt lan sitemap.xml

Papat léngkah, tanpa ngadaptar. Saben pariksaan dilaksanakeun dina sisi panyedhiya.

1

_Petik URL situs

Enter any site's homepage URL. You don't need the full path to robots.txt, because we derive it from the host.

2

Kita njupuk saka server kita

Server urang requests robots.txt sarta unggal sitemap.xml dinyatakeun, jadi tes teu diblokir ku browser anjeun atawa cache sorangan.

3

Waca pedoman lan sitemaps

Ngawaca grup user-agent anu dianalisis, henteu ngamungkinkeun aturan sareng daftar sitemap anu dinyatakeun, sareng conto URL ti masing-masing.

4

Nyahpasang apa sing ditanda

Ngalereskeun URL sitemap string query atanapi konflik anu diblokir-tapi diindeks sateuacan mesin pencari ngaleupaskeun duplikat.

Apa sing dites dening tester iki

Sacara umum, panulis bisa nulis dina basa anu béda-béda.

📜

Instruksi Crawl

Ngawaskeun unggal grup user-agent sarta aturanna Allow / Disallow supados anjeun tiasa ningali persis naon anu diblokir.

🗺️

Sitemaps dinyatakake

Robots.txt minangka file sing ngandung informasi babagan cara robots.txt digunakaké kanggo ngakses situs web.

⚠️

Текстовая запись

Digunakeun pikeun ngagambarkeun URL anu dipikabutuh pikeun ngajalankeun halaman wéb anu dipikabutuh.

🔗

Njupuk kesehatan

Catches sitemaps declared in robots.txt that error, redirect, or return non-XML. These are silent crawl killers.

Takon-takon kang kerep diajukake

Sahiji URL string panalungtikan dina peta situs bisa ngahasilkeun ratusan halaman anu mirip. Mesin pencari bisa ngacrawl éta halaman, manggihan isi anu sami dina parameter anu béda, sarta ngurangan sadaya cluster pikeun isi anu sarua. Pameriksa ieu bakal ngabélakeunana saméméh éta kajadian.
Teu. A robots.txt kaleungitan atawa kosong ngan ngamungkinkeun sadaya crawling. Resikona sabalikna: a over-wide disallow anu ngablokir halaman anu anjeun hoyong indeks, atanapi sitemap anu dinyatakeun dina robots.txt anu ngabalikeun kasalahan.
No. Kami meunang robots.txt jeung sitemaps éta nganyatakeun, teras maca conto URL ti unggal. Kami teu recursively crawl unggal halaman, jadi tes gancang sarta moal ngamuat server anjeun.
Ya, tester ieu bébas tanpa ngadaptar. Programmatic akses ngaliwatan REST API aya dina rencana managed mayar.
Kanggo pangembangan

Ngjalanake ing pipa sampeyan

This tester runs entirely server-side. You submit a URL, our backend fetches and parses robots.txt and the declared sitemaps, then re-renders this page with the results. A programmatic REST API for CI/CD and bulk auditing is available on the paid managed plans. Lihat rencana →

Pembangun API

Nggunakake kanthi program

Every tool is available through one authenticated REST API on the Pro and API plans. Wire it into your CI, CMS, or dashboards.

curl -X POST https://seo.bike/api/v1/robots-check \ -H "Authorization: Bearer sk_live_yourkey" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com/"}'
200 га.
{ "ok": true, "result": { ... } }
Ngambil kunci API → Metered, per-call pricing. Free tier for the interactive tool above.
Check every page of your site, not one at a time
Pro connects Google Search Console, fetches every page Google skipped, every broken page and every page that ranks low, and writes what to fix on each one.
See how Pro works