robots.txt and AI crawlers, explained for store owners

Updated 2026-10-16 · 10 min read · By the ProductSeen team

Your robots.txt file is a short text file at yourstore.com/robots.txt that tells crawlers which parts of your site they may read. In 2026 it quietly decides whether AI shopping assistants can see your products. One overly broad line can take an entire store out of AI answers.

How robots.txt works in 60 seconds

The file is made of groups. Each group starts with one or more User-agent lines naming a crawler, followed by Allow and Disallow rules for paths.

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /cart
Disallow: /checkout

Key rules most people miss:

A crawler follows only the most specific group that names it. If there's a User-agent: GPTBot group, GPTBot ignores the * group entirely.
The longest matching rule wins. Allow: /products/ beats Disallow: / for product URLs.
* is a wildcard and $ marks the end of a URL (supported by major crawlers).
No robots.txt at all means everything is allowed.

The AI crawlers that matter for stores

User-agent	Operator	What it's used for	Recommendation for stores
`OAI-SearchBot`	OpenAI	Finding and surfacing sites in ChatGPT search results	Allow
`ChatGPT-User`	OpenAI	Visiting pages when a ChatGPT user's request needs it	Allow
`GPTBot`	OpenAI	Collecting content that may be used to train models	Your choice
`PerplexityBot`	Perplexity	Indexing for Perplexity's answers and shopping	Allow
`Google-Extended`	Google	A control token for Gemini / AI use; doesn't affect Google Search	Your choice (allowing helps Gemini)
`ClaudeBot`	Anthropic	Crawling for Anthropic's models	Allow if you want Claude to know your products
`Applebot-Extended`	Apple	A control token for Apple's AI training use	Your choice
`CCBot`	Common Crawl	Open web archive used in many AI training datasets	Optional
`Bytespider`	ByteDance	ByteDance crawling	Optional

Bot names and purposes change. Before making a final decision, check each company's official crawler documentation (OpenAI, Anthropic, Google, Apple and Perplexity all publish one).

Search bots vs. training bots

The most useful distinction: some bots fetch your pages so an assistant can show and link to them (OAI-SearchBot, ChatGPT-User, PerplexityBot). Others collect content for training future models (GPTBot, CCBot). For a store, the first kind is direct shopping visibility. The second is a judgment call. Some brands block training on principle; others allow it so models "know" their brand. There's no proven right answer, but blocking search bots is almost always a mistake for a store that wants sales.

A sensible robots.txt for an online store

# Let AI search and shopping assistants read the store
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
Allow: /
Disallow: /cart
Disallow: /checkout
Disallow: /account

# Everyone else (including Googlebot)
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search

Sitemap: https://yourstore.com/sitemap.xml

Notice the AI group repeats the cart and checkout rules. Because a bot only follows its most specific group, rules in the * group don't carry over.

Common mistakes we see

A blanket "block all AI" snippet copied from a blog, which blocks OAI-SearchBot and PerplexityBot along with training bots.
Disallowing /products or /collections for one bot to "save crawl budget".
A named group that forgets the basics, so the bot is suddenly allowed into cart and account pages (harmless but messy) or, worse, disallowed from everything.
Firewall rules that contradict robots.txt. Some CDN and security settings block AI bots at the network level even when robots.txt allows them. Check your bot-protection settings too.
Staging rules left live: User-agent: * / Disallow: / after a redesign.

Editing robots.txt on Shopify

Shopify generates a default robots.txt for every store, and you can't upload a plain file. To change it, go to Online Store > Themes > ⋯ > Edit code, add a new template called robots.txt.liquid, and edit the rules there. Shopify's template keeps its default rules and lets you add groups and rules. Shopify recommends caution: mistakes can remove pages from search. After saving, open /robots.txt in a private window to confirm the result.

Editing robots.txt elsewhere

WooCommerce / WordPress: use your SEO plugin's robots.txt editor or upload a file to the site root.
BigCommerce: the robots.txt editor is in your store settings (search the control panel for "robots"); BigCommerce's help center has the current location.
Custom / headless: serve a static file at the root of the storefront domain.

Check your store

ProductSeen's free scan reads your robots.txt the way these crawlers do and tells you, bot by bot, whether your product pages are allowed. Next, make sure what they find is readable: see Product schema for AI.

See what AI can read about your products

Free scan: AI crawler access, Product schema, GTINs, reviews, sitemap and llms.txt. Results in under a minute, with the top fixes in plain English.

Scan my store free

Frequently asked questions

If I block GPTBot, will my products disappear from ChatGPT?

Not necessarily. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search results. They're controlled separately, so you can block training while still allowing search. Check OpenAI's current bot documentation, since these details can change.

Does blocking Google-Extended hurt my Google rankings?

Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls whether your content is used for Gemini models and some AI features.

Is robots.txt a security tool?

No. It's a public request that well-behaved crawlers follow. It doesn't stop anyone from visiting a page. Use logins and permissions for anything private.

How do I edit robots.txt on Shopify?

In Shopify admin go to Online Store > Themes > Edit code, add a template named robots.txt.liquid, and modify the default rules. Shopify's developer documentation has the exact template syntax.