🤖 Crawl & indexation

Robots.txt & Sitemap Tester

Tichatora robots.txt uye sitemap.xml kubva kune yedu server, tichakuratidza zviyero zvecrawl uye zvakanyorwa sitemaps, uye tichazivisa iyo query-string sitemap URLs iyo inokonzera kudzoka kweduplicate-content.

Maitiro ekuongorora robots.txt uye sitemap.xml

Nzira mbiri, hapana kushanyira. Iyo yose ongororo inofamba server-side.

Peta URL yesaiti

Iwe haufanire kuziva nzira yakazara ye robots.txt — isu tinoiwana kubva kune mushandisi wewebhusaiti.

We fetch from our server

Isu tinoda kuti mutengi wedu awane robots.txt uye chero sitemap.xml, kuti tikwanise kuongorora kuti hapana chinokanganisa chinogona kuitika kana iwe ukashandisa browser yako kana yako yemahara cache.

Ona mirairo uye sitemaps

Rega kuongorora mapoka evashandisi-agent akapararira, kurambidza mitemo uye kuratidza webhusaiti mapeji, ne URL mifananidzo kubva kune imwe neimwe.

Kugadzirisa zvayakavharwa

Kugadzirisa chero query-string sitemap URLs kana akavharwa-asi-indexed zvikanganiso pamberi pekutsvaga injini inodzikisa ma duplicates.

Chii ichi tester chinoongorora

The same guardrails isu kutamba mukati mekambani pamberi kuburitsa imwe peji.

📜

Crawl directives

Parses yega yega user-agent group uye yayo Allow / Disallow pfungwa kuti iwe unogona kuona zvakajeka chii chakavharwa.

🗺️

Kuzivisa sitemaps

Inonyora mazita ewebsite ese akanyorwa mu robots.txt, anowana ese, uye anozivisa mhando yake uye URL kuverengerwa.

⚠️

Query-string bloat

Flags query-string sitemap URLs iyo inogadzira peji rakapetwa uye rinokonzera kuderera kwezvinyorwa zvakapetwa.

🔗

Kuwana hutano

Inobata sitemaps yakanyorwa mu robots.txt iyo matambudziko, kudzokera, kana kudzokera kune zvisina XML - zvakavanzika zvekuchera vafa.

Zvimwe zvinobvunzwa zvakanyanya

A query-string URL mu sitemap anogona kubereka mamiriyoni enzvimbo dzakapetwa-zvinyorwa. Search injini kutsvaka avo, kuwana zvakafanana zvinhu pasi pezvimwe parameters, uye kuderedza yose cluster yezvinyorwa zvakapetwa.

Hapana. A kushaya kana vakasununguka robots.txt chete anobvumira zvese kutsvaka. Risk ndeyekuzvimira: a over-wide kurambidza kuti blocks mapeji iwe uchida indexed, kana sitemap yakavakirwa mu robots.txt iyo inodzokera matambudziko.

Hapana. Tinowana robots.txt uye maSitemap ayo anozivisa, uye tinoverenga mifananidzo yeURLs kubva pazvimwe. Hatiite kutsvagisa kwepakutanga kwese.

Yeah, tester iri nyore pasina kurodha pasi. Programmatic kuwanikwa kuburikidza REST API iripo pazvirongwa zvakabhadharwa zvakachengetwa.

⚙️ Yevagadziri

Kufamba munzira yako

Iyi tester inofamba zvakananga kubva panzvimbo yeserver - iwe unotumira URL, yedu backend inotora uye inoongorora robots.txt uye yakazivisa sitemaps, uyezve inodzosera iyi peji nemhedzisiro. A programmatic REST API yeCI / CD uye bulk auditing inowanikwa pane zvakabhadharwa zvakachengetwa zvirongwa. Ona mapurojekiti →