Skip to content
Log in

Edit robots.txt to prevent search engine indexing

On this page

Use this topic to control which pages search engines crawl on your store, or to stop them indexing the store altogether.

StoreConnect automatically generates a robots.txt file each day for your site. The file tells search engine crawlers which URLs the crawler can access on your site. Learn more about robots.txt.

Choose the right approach

Pick the option that matches how much you need to hide, before you change anything:

Goal Use
Guide crawlers to or away from particular paths The auto-generated robots.txt, described below
Stop search engines indexing the whole store A noindex meta tag in Head Content
Block crawling, sitemap, and merchant feeds across every entry point Stealth mode
Give AI tools structured context about the store An llms.txt file

How to find robots.txt

For live stores (not in test mode), the robots.txt is available via your-store.com/robots.txt. If you are running multiple stores within StoreConnect, each store has its own unique robots.txt file. StoreConnect does not generate the file for test sites or non-production sites.

When downloading the robots file, use your browser’s private browsing mode to ensure you get the most current version. Otherwise you could be downloading an older version from your cache.

More information about robots.txt

Visit http://www.robotstxt.org/ for detailed information about what the file is for and how to edit for your needs.

Stop search engines indexing your store

Add a noindex meta tag to the store’s Head Content to tell search engines not to index any page.

  1. Open the Store record and go to its Head Content.

  2. Add this tag:

    ```html

    ```

  3. Save the Content Block.
  4. Load any page on your store, view the page source, and confirm the noindex tag appears in the <head>.

This tag is a request, not a block. Compliant crawlers honor it; non-compliant scrapers may ignore it. Pages already in a search index stay there until the search engine re-crawls them.

Prevent all search engine indexing with stealth mode

If you need to block search engines across all entry points (robots.txt, sitemap, and merchant feeds), use stealth mode instead. Stealth mode sets Disallow: / in robots.txt, suppresses sitemap and merchant feed responses, and adds an X-Robots-Tag: noindex, nofollow header to every HTTP response.

Related articles

Documentation StoreConnect sync Documentation Configure zones for tax and shipping areas Documentation Location finder Documentation Stealth mode Documentation Generate an llms.txt file for your site

Was this article helpful?

Was this article helpful?