Sitemap & robots.txt checker

How to use the Sitemap & Robots.txt Checker

1. Enter your domain, like yoursite.com. To check one file, paste the sitemap URL instead.
2. Fix the red items first: they stop pages from being crawled or indexed. Yellow items are improvements.
3. Use the robots.txt tester below the results to try a fix before you deploy it, then check again.

What this checker tests

• robots.txt: that it loads as plain text, isn't an HTML page from a catch-all route, stays under Google's 500 KiB limit, doesn't block Googlebot from your site or your CSS and JavaScript, and declares your sitemap. Unknown and unsupported directives, like noindex and crawl-delay, are flagged with the reason.
• Sitemap discovery: we read the Sitemap: lines in robots.txt, like Google does. Only if there are none do we try /sitemap.xml and /sitemap_index.xml, and a miss there isn't counted as an error.
• Sitemap files: HTTP status, content type, gzip, valid XML (the unescaped & is the classic bug), the sitemaps.org namespace, the 50,000 URL and 50 MB limits, and sitemap indexes, including nested ones Google won't follow.
• Listed URLs: duplicates, relative or off-site URLs, http vs https mismatches, URLs your own robots.txt blocks, and the format, spread and honesty of lastmod dates.
• Spot check: we fetch a spread of listed pages and flag any that are broken, redirect, are set to noindex, or point their canonical somewhere else.

Sitemap best practices

1. List only canonical, indexable pages that return 200. No redirects, 404s, noindex pages, or parameter and UTM variants. Set canonicals and robots tags with the meta tag generator.
2. Use accurate lastmod dates. Google uses lastmod when it's consistently accurate and ignores it when every URL shows today's date. Set it when the content changes meaningfully, not on every build.
3. Skip priority and changefreq. Google ignores both. They don't hurt, but they don't help either.
4. Split big sites. Each file can hold 50,000 URLs and 50 MB uncompressed. Past that, use several sitemaps and one sitemap index.
5. Use your final URLs. Same protocol and host everywhere: if your site is https://www.example.com, don't list http:// or non-www URLs. Keep them short and readable with the URL slug generator.
6. Declare it and submit it. Add a Sitemap: line to robots.txt so every crawler finds it, then submit it in Google Search Console and Bing Webmaster Tools.

robots.txt mistakes that cost traffic

The expensive ones are rarely subtle. A Disallow: / left over from staging removes a whole site from search. A server error on /robots.txt makes Google pause crawling. A single-page app that answers /robots.txt with its HTML shell leaves crawlers with no rules at all.

Two rules prevent most of the rest. First, don't block a page in robots.txt to keep it out of Google: blocked pages can still be indexed from links, and Google can't see a noindex tag it isn't allowed to fetch. Second, never list a URL in your sitemap that robots.txt blocks. That's a contradiction Search Console reports as an error. Build a clean file with the robots.txt generator, including separate rules for AI training and AI search crawlers.

After the sitemap: check the pages themselves

A clean sitemap gets pages crawled. Whether they rank and get clicked depends on the page. Run your most important URLs through the marketing SEO checker, and see how each title and description looks in results with the SERP preview tool. Launching soon? The launch checklist covers the rest.

Frequently asked questions