|
Exclude Certain Directories: This option tells Siteliner to exclude certain directories from the scan. Simply enter the names of your directories, one on each line, in the text field. Siteliner ignores your robots.txt file and scans your entire site, excluding the directories that you specify. For example, if you are scanning your site (http://www.example.com) and wish to exclude all pages in the 'sports' ('http://www.example.com/sports/') and 'business/banking' ('http://www.example.com/business/banking/') directories, you would enter these two lines into the text field:
sports/
|