Ignore robots meta tags found on page

This option allows you to control which parts of your site will be scanned.

Normally, robots meta tags found in the header section of individual pages on your site control how different "bots" - automatic crawlers such as search engines - crawl your site. For example, a robots meta tag might tell certain bots not to crawl a certain page on your site.

Selecting this option tells Siteliner to ignore the robots meta tags when crawling the site.

For more information about robots meta tags, please see here. For more information about the Siteliner Bot, please see here.