Crawlability · Guide
robots.txt and Sitemap.xml Explained
Understand how robots.txt and sitemaps guide crawlers — and how Sourcer checks whether they are reachable for a host.
robots.txt tells crawlers what they may fetch; a sitemap lists URLs you want discovered. Sourcer checks whether both are reachable for the scanned host — useful before a batch SEO scan.
robots.txt basics
- Live at
/robots.txton the host - Use
Disallowcarefully — blocking CSS/JS can hurt rendering - Point to your sitemap with a
Sitemap:line when possible
Sitemaps
Keep sitemaps accurate and under size limits. After publishing new URLs, confirm the sitemap responds, then spot-check pages with the SEO Checker and lock preferred URLs with canonicals.
