WPCheck a site
Menu

Publish a sitemap and robots.txt

A sitemap lists the pages you want found, and robots.txt tells crawlers where they may go. Together they help Google and Bing find your site.

Steps

About an hour

All guides

  1. sitemap.xml
  2. robots.txt
  3. Search Console
  1. List every public page in sitemap.xml, with its final address and the date it really changed.
  2. Leave out pages that redirect, return an error or are marked noindex.
  3. Add a robots.txt at the root with a Sitemap line giving the full address. Don’t block a page that relies on noindex: crawlers must be able to read the tag.
  4. Submit the sitemap once in Google Search Console and once in Bing Webmaster Tools.
Example
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Read the guideMake trackers wait for a choice