WPCheck a site

Free robots.txt tester

Enter a site and a page, and see which crawlers may read it, and which rule decides.

Try it

  1. Tests one page for seven crawlers, search engines and AI alike.
  2. Applies each crawler’s own group of rules when it has one.
  3. Shows the exact rule that decides, using Google’s longest-match logic.
  4. Lists the sitemaps your robots.txt names.

Other free tools