StoreConnect automatically generates a robots.txt file each day for your site. The file tells search engine crawlers which URLs the crawler can access on your site. Learn more about robots.txt.
You can edit the file to control what pages are crawled, and to optimize search requests on your site.
How to find robots.txt
For live stores (not in test mode), the robots.txt is available via your-store.com/robots.txt. If you are running multiple stores within StoreConnect, each store will have it’s own, unique robots.txt file. StoreConnect does not generate the file for test sites or non-production sites.
When downloading the robots file, use your browser’s ‘Private Mode’ to ensure you get the most current version otherwise you could be downloading an older version from your cache.
More information about robots.txt
Visit http://www.robotstxt.org/ for detailed information about what the file is for and how to edit for your needs.
Disable robots.txt for your StoreConnect store
If you want, you can disable the robots file to prevent it from being indexed by search engines.
- Go to your store and add or edit the store's Head Content.
- Add this tag: .
- Save the Content Block.