{"title":"Edit robots.txt to prevent search engine indexing","slug":"robots-txt","url":"https://support.storeconnect.com/articles/robots-txt","url_markdown":"https://support.storeconnect.com/articles/robots-txt.md","subtitle":null,"summary":"StoreConnect auto-generates a robots.txt file daily for live stores. Control which pages search engines crawl, or disable indexing entirely by adding a noindex meta tag to your store\u0026#39;s Head Content.","type":"Help_Documentation","video_url":"","keywords":"robots.txt, search engine indexing, SEO, noindex, crawlers, sitemap, crawl control, robots file, search engines","last_modified":"2026-08-21T07:12:35+0000","body_markdown":"Use this topic to control which pages search engines crawl on your store, or to stop them indexing the store altogether.\n\nStoreConnect automatically generates a robots.txt file each day for your site. The file tells search engine crawlers which URLs the crawler can access on your site. [Learn more about robots.txt](https://www.robotstxt.org/robotstxt.html).\n\n## Choose the right approach\n\nPick the option that matches how much you need to hide, before you change anything:\n\n| Goal | Use |\n|------|-----|\n| Guide crawlers to or away from particular paths | The auto-generated `robots.txt`, described below |\n| Stop search engines indexing the whole store | A `noindex` meta tag in **Head Content** |\n| Block crawling, sitemap, and merchant feeds across every entry point | [Stealth mode](stealth-mode) |\n| Give AI tools structured context about the store | An [llms.txt file](llms-txt-for-stores) |\n\n## How to find robots.txt\n\nFor live stores (not in test mode), the `robots.txt` is available via `your-store.com/robots.txt`. If you are running multiple stores within StoreConnect, each store has its own unique `robots.txt` file. StoreConnect does not generate the file for test sites or non-production sites.\n\nWhen downloading the robots file, use your browser's private browsing mode to ensure you get the most current version. Otherwise you could be downloading an older version from your cache.\n\n## More information about robots.txt\n\nVisit [http://www.robotstxt.org/](http://www.robotstxt.org/) for detailed information about what the file is for and how to edit for your needs.\n\n## Stop search engines indexing your store\n\nAdd a `noindex` meta tag to the store's **Head Content** to tell search engines not to index any page.\n\n1.  Open the **Store** record and go to its **Head Content**.\n2.  Add this tag:\n\n    ```html\n    \u003cmeta name=\"robots\" content=\"noindex\"\u003e\n    ```\n\n3.  Save the **Content Block**.\n4.  Load any page on your store, view the page source, and confirm the `noindex` tag appears in the `\u003chead\u003e`.\n\nThis tag is a request, not a block. Compliant crawlers honor it; non-compliant scrapers may ignore it. Pages already in a search index stay there until the search engine re-crawls them.\n\n## Prevent all search engine indexing with stealth mode\n\nIf you need to block search engines across all entry points (robots.txt, sitemap, and merchant feeds), use [stealth mode](stealth-mode) instead. Stealth mode sets `Disallow: /` in robots.txt, suppresses sitemap and merchant feed responses, and adds an `X-Robots-Tag: noindex, nofollow` header to every HTTP response."}