About the Siteliner Bot

Siteliner is a website analysis tool by Indigo Stream Technologies, the people behind Copyscape.

Since Siteliner provides its results in real time, it performs a controlled crawl of up to 250 pages of a website in a limited amount of time, while ensuring that no excess load is placed on the web server for that site. All Siteliner bot requests are identified by this user agent string:

Mozilla/5.0 (compatible; Siteliner/1.0; +http://www.siteliner.com/bot)

To avoid placing excessive load on your website, Siteliner applies the following protections:

  • Siteliner only retrieves HTML web pages, not embedded images or videos, so it only uses a small amount of bandwidth.
  • A limit of 4 page requests sent at any one time to your server. This load is similar to that of a regular web browser retrieving an HTML page with embedded images, style sheets and scripts.
  • A limit of 250 pages retrieved from a single site during a Siteliner analysis.
  • Each website can only be analyzed by Siteliner users once per 30 days.
  • This rate limit is applied at the level of your server's IP address, rather than its domain name. This ensures that Siteliner cannot be used to generate excessive load on a single server which hosts multiple sites under different domain names.
  • A hidden Javascript captcha that makes it difficult for scripts to automate Siteliner runs.

Blocking Siteliner

If you wish to prevent Siteliner from analyzing your website, you may use a standard robots.txt file in the root directory of your website. Use User-agent: Siteliner to target rules to the Siteliner bot. For example, to block all Siteliner access to your server, simply place the following at the end of your robots.txt file:

User-agent: Siteliner
Disallow: /

If you have any questions or concerns about the Siteliner bot, please feel free to contact us.