Edit robots.txt to prevent search engine indexing

StoreConnect automatically generates a robots.txt file each day for your site. The file tells search engine crawlers which URLs the crawler can access on your site. Learn more about robots.txt.

You can edit the file to control what pages are crawled, and to optimize search requests on your site.

How to find robots.txt

For live stores (not in test mode), the robots.txt is available via your-store.com/robots.txt. If you are running multiple stores within StoreConnect, each store will have it’s own, unique robots.txt file. StoreConnect does not generate the file for test sites or non-production sites.

When downloading the robots file, use your browser’s ‘Private Mode’ to ensure you get the most current version otherwise you could be downloading an older version from your cache.

More information about robots.txt

Visit http://www.robotstxt.org/ for detailed information about what the file is for and how to edit for your needs.

Disable robots.txt for your StoreConnect store

If you want, you can disable the robots file to prevent it from being indexed by search engines.

1. Go to your store and add or edit the store’s Head Content.
2. Add this tag: <meta name="robots" content="noindex">
3. Save the Content Block.

Prevent all search engine indexing with stealth mode

If you need to block search engines across all entry points — robots.txt, sitemap, and merchant feeds — use Stealth mode instead. Stealth mode sets Disallow: / in robots.txt, suppresses sitemap and merchant feed responses, and adds an X-Robots-Tag: noindex, nofollow header to every HTTP response.