Robots

WATCH THE VIDEO

If you want your website to be seen in search results of any search engines, you need to allow your website and various web pages to be crawled and indexed by search engine robots.

There are two files which help search engine robots find what they need: robots.txt and sitemap. These files are located on the Web server.

The contents of the robots.txt file can be managed directly from Sana Admin: Setup > Search engine optimization > Basics.
Moreover, a separate robots.txt file can be generated per webstore in case of a multi-store solution.

The table below provides the description of the indexing settings in Sana Admin.

Field	Description
Indexing	Use this option to allow / deny your webstore to be indexed and crawled by search engine robots. By default, indexing is disabled. Here is the default record in the robots.txt file when everything is disabled: User-agent: * Disallow: / Below you can see the default record in the robots.txt file when indexing is enabled: User-agent: * Disallow: /admin Disallow: /shop/basket Disallow: /profile Disallow: /checkout Disallow: /productcompare When you enable indexing the default record in the robots.txt file states that all search engine robots can crawl almost all webstore URLs except the URLs of Sana Admin and several webstore system pages.
Sitemap in robots.txt	This option allows to add the sitemap URL to the robots.txt file. Below you can see the record in the robots.txt file when Indexing and Sitemap in robots.txt are enabled: User-agent: * Disallow: /admin Sitemap: http://sanastore.com/sitemap.xml To be able to add the sitemap URL to the robots.txt file, at first you should create the sitemap file by running the Sitemap export task from Sana Admin: System > Scheduled tasks. For more information about sitemap, see Sitemap Export.
Custom robots.txt text	In this field, you can add any custom indexing rules. NOTE Indexing rules added to this field will redefine the default behavior no matter whether indexing is disabled or enabled.
Indexing of product attachments	Enable this option if you want product attachments to be indexed by search engines.

For more information about robots.txt and indexing rules, see About robots.txt.

For more information about other settings on the Basics tab, see URL Behavior.