# Edit robots.txt to prevent search engine indexing Source: https://support.storeconnect.com/articles/robots-txt · Last modified 21 August 2026 Use this topic to control which pages search engines crawl on your store, or to stop them indexing the store altogether. StoreConnect automatically generates a robots.txt file each day for your site. The file tells search engine crawlers which URLs the crawler can access on your site. [Learn more about robots.txt](https://www.robotstxt.org/robotstxt.html). ## Choose the right approach Pick the option that matches how much you need to hide, before you change anything: | Goal | Use | |------|-----| | Guide crawlers to or away from particular paths | The auto-generated `robots.txt`, described below | | Stop search engines indexing the whole store | A `noindex` meta tag in **Head Content** | | Block crawling, sitemap, and merchant feeds across every entry point | [Stealth mode](stealth-mode) | | Give AI tools structured context about the store | An [llms.txt file](llms-txt-for-stores) | ## How to find robots.txt For live stores (not in test mode), the `robots.txt` is available via `your-store.com/robots.txt`. If you are running multiple stores within StoreConnect, each store has its own unique `robots.txt` file. StoreConnect does not generate the file for test sites or non-production sites. When downloading the robots file, use your browser's private browsing mode to ensure you get the most current version. Otherwise you could be downloading an older version from your cache. ## More information about robots.txt Visit [http://www.robotstxt.org/](http://www.robotstxt.org/) for detailed information about what the file is for and how to edit for your needs. ## Stop search engines indexing your store Add a `noindex` meta tag to the store's **Head Content** to tell search engines not to index any page. 1. Open the **Store** record and go to its **Head Content**. 2. Add this tag: ```html ``` 3. Save the **Content Block**. 4. Load any page on your store, view the page source, and confirm the `noindex` tag appears in the ``. This tag is a request, not a block. Compliant crawlers honor it; non-compliant scrapers may ignore it. Pages already in a search index stay there until the search engine re-crawls them. ## Prevent all search engine indexing with stealth mode If you need to block search engines across all entry points (robots.txt, sitemap, and merchant feeds), use [stealth mode](stealth-mode) instead. Stealth mode sets `Disallow: /` in robots.txt, suppresses sitemap and merchant feed responses, and adds an `X-Robots-Tag: noindex, nofollow` header to every HTTP response. --- ## Follow StoreConnect - [Email Newsletter](https://getstoreconnect.com/c/lp-newsletter) - [LinkedIn Newsletter](https://www.linkedin.com/build-relation/newsletter-follow?entityUrn=7444956928444862464) - [YouTube](https://www.youtube.com/channel/UCngKdP2x8l1wcbAKW3tvU8g) - [LinkedIn](https://www.linkedin.com/company/storeconnect) - [X / Twitter](https://x.com/storeconnecthq) ## Popular Links - [Partners](https://getstoreconnect.com/partners) - [News](https://getstoreconnect.com/articles/news) - [Events](https://getstoreconnect.com/articles/events) - [Feature Comparison](https://getstoreconnect.com/how-we-compare) - [Download a free trial](https://appexchange.salesforce.com/appxListingDetail?listingId=a0N3A00000FMkeKUAT) - [Book a Demo](https://getstoreconnect.com/contact) ## Documentation - [Help documentation](https://support.storeconnect.com/help-documentation) - [AI agents](https://support.storeconnect.com/ai) - [Videos & tutorials](https://support.storeconnect.com/videos-tutorials) - [Developer reference](https://support.storeconnect.com/developer-reference) - [Release notes](https://support.storeconnect.com/release-notes) - [Troubleshooting](https://support.storeconnect.com/troubleshooting) - [Trust Center](https://trust.getstoreconnect.com/) - [Status Page](https://status.storeconnect.com/) ## Contact - info@getstoreconnect.com - US +1 415 745 3230 - AUS +61 2 8365 2308 100 S Ashley Dr, Suite 600-2461 Tampa FL 33602-600 USA Level 22, Sydney Place 180 George Street Sydney, NSW, 2000, AUS --- StoreConnect Support — https://support.storeconnect.com/articles/robots-txt