AI Crawlers and robots.txt: Are You Blocking the Assistants Who Should Cite You?
· Visora
AI Crawlers and robots.txt: Are You Blocking the Assistants Who Should Cite You?
Here is a quiet failure mode we see over and over in ecommerce GEO audits. A store ranks well in Google, traffic from classic search is healthy, and yet ChatGPT and Perplexity never mention it. The cause is often one line in a file most marketers have not opened in years: robots.txt. AI assistants do not magically know your products exist. Their systems send out crawlers, and those crawlers obey the same rules that Googlebot has followed for two decades. If you have blocked them, you are invisible to the answer engines, no matter how good your content is. If you are new to this, start with [what GEO actually is](/blog/what-is-geo-2026).
Meet the crawlers Several distinct bots matter for AI search. GPTBot is OpenAI's general crawler, while OAI-SearchBot is the one associated with search-style retrieval in ChatGPT. ClaudeBot and anthropic-ai serve Anthropic. PerplexityBot fetches pages for Perplexity. Google-Extended is a separate control for Google's generative products, deliberately split from Googlebot so you can opt out of AI training and surfaces without hurting classic search. Each of these looks for /robots.txt at the root of your domain and reads the User-agent and Disallow rules that apply to it. Some also honor meta tags and HTTP headers, but robots.txt is the common denominator.
The tradeoff nobody explains Blocking an AI crawler is a real choice with real consequences, and it is not simply good or bad. If your business depends on licensed content, you may want to restrict training crawlers. But there is a difference between allowing a bot to train a model and allowing it to retrieve and cite your pages in an answer. Many stores blocked everything in 2023 out of caution and never revisited the decision. Today that blanket block means when a shopper asks an assistant for a recommendation in your category, you cannot be cited. If you are serious about generative engine optimization, you need a deliberate, written position on each bot rather than an accidental default.
Audit in five steps First, open https://yourstore.com/robots.txt in a browser and read it from top to bottom. Second, list every User-agent group that names an AI bot and note the Disallow rules beneath it. Third, check whether a broad group such as User-agent: * with Disallow: / is catching crawlers you actually want. Fourth, verify your important product, guide, and policy URLs are not disallowed by any rule, including wildcard patterns. Fifth, confirm the file itself returns HTTP 200 and is not redirected through a chain that a crawler might give up on.
Misconfigurations that quietly cost citations The most common problem is the blanket block inherited from an old template. The second is a staging rule that made it to production and hides /products or /collections. The third is a CDN or WAF rule that blocks requests with unusual user agents, which stops a crawler even though robots.txt says yes. The fourth is blocking based on a user agent string that changed; bot names evolve, and a rule written for one name may miss the current one. Finally, some platforms inject their own robots.txt, so editing a file in your theme does nothing. Check what is actually served, not what you think you uploaded.
The Shopify and CDN wrinkle Shopify generates a robots.txt that you can customize through the admin, but many stores also sit behind a CDN with its own bot rules. If you use a headless setup, a reverse proxy, or a security plugin, there may be two or three layers deciding who gets in. Trace the request end to end. Use a tool that simulates a specific user agent against a real URL and confirm the response is 200 with real HTML, not a challenge page or a blank shell. An AI crawler that receives only JavaScript with no rendered facts has effectively been blocked anyway.
Verify with real questions Robots.txt is a permission slip, not a guarantee of citation. After you fix it, run the same natural-language questions your customers ask through several assistants and record which of your pages appear. Track that over time with a simple [geo measurement framework](/blog/geo-measurement-framework), and treat any sudden drop as a signal to re-check crawl rules. This is where GEO differs from SEO: a classic ranking can look stable while your answer-engine presence collapses, which is why understanding [geo versus seo](/blog/geo-vs-seo-difference) is more than an academic exercise.
The throughline Your content cannot be cited if the crawler cannot read it. Audit robots.txt today, make a deliberate decision for each AI bot, and remove the accidental blocks that keep good products out of AI answers. It is a ten-minute fix with an outsized effect on your AI search visibility.
Put this into practice
Audit your PDP or category page with Visora, then fix schema and FAQ gaps that block AI citations.
Run a free GEO audit →https://geovisora.com/en/blog/ai-crawlers-robots-txt-2026