Skip to content
Back to writing
Method

AI crawlers: which bots to allow if you want to be cited

The AI crawlers that decide whether ChatGPT, Perplexity, Claude and Google can cite you, the ones that only collect training data, and the robots.txt lines and CDN checks that keep the right ones in.

9 Nov 2026 - 7 min read
On this pageShow 9Hide

"AI crawlers" sounds like one thing, and plenty of robots.txt files treat them that way: one block list that shuts out every bot with AI in its description. The trouble is that the AI companies run several crawlers each, for different jobs, and only some of them decide whether you can be cited.

This guide sorts the crawlers by job, gives the robots.txt lines for a site that wants to be cited, and covers the CDN setting that can quietly undo all of it. It is based on the companies' own crawler documentation, read on October 31, 2026.

Three kinds of AI crawler

Each AI company separates its bots by purpose. The three jobs matter more than the names.

  • Search crawlers fetch and index pages so the assistant can find and link them when it answers. Block these and you drop out of that assistant's search answers.
  • Training crawlers collect content that may be used to train future models. Blocking them is a training opt out. On its own it does not remove you from search answers.
  • User fetchers visit a page because a person asked the assistant something in real time. They are not automatic crawlers, and some companies say robots.txt may not apply to them.

If your goal is to be cited, the search crawlers are the ones that must get through.

Free scan

Check your AI visibility free

See which buying questions ChatGPT, Perplexity, Gemini and Google's AI answers name you for, and who they name instead. No call, no card.

The search crawlers that decide citations

OAI-SearchBot (OpenAI). OpenAI says OAI-SearchBot is used to surface websites in ChatGPT's search features, and that sites opted out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. Its help center goes further: to make a site eligible, allow OAI-SearchBot and confirm that your host or CDN allows traffic from OpenAI's published search bot IP addresses.

PerplexityBot (Perplexity). Perplexity says PerplexityBot is designed to surface and link websites in its search results, is not used to crawl content for AI foundation models, and recommends allowing it along with its published IP ranges.

Claude-SearchBot (Anthropic). Anthropic's crawler documentation, dated April 7, 2026, says Claude-SearchBot navigates the web to improve search result quality, and that disabling it prevents your content being indexed for search, which may reduce your visibility in user search results.

Googlebot (Google). Google says AI is built into Search and that robots.txt rules for Googlebot are the control for how sites are crawled for Search, so AI Overviews and AI Mode depend on Googlebot access plus your snippet settings.

The training crawlers

GPTBot (OpenAI) crawls content that may be used to train OpenAI's foundation models. OpenAI says each of its settings is independent, so a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot.

ClaudeBot (Anthropic) collects web content that could contribute to training Anthropic's models. Anthropic says restricting it signals that your future materials should be excluded from its training datasets.

Google-Extended (Google) is not a crawler at all. Google says it has no separate user agent string and is a robots.txt token used as a control. It covers whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and Google says it does not affect inclusion or ranking in Google Search.

That last point cuts both ways. Blocking Google-Extended does not touch AI Overviews, but because it also covers grounding in Gemini Apps, it can affect whether Gemini uses your pages when it answers. If being cited in Gemini matters to you, think before you block it.

The user fetchers

  • ChatGPT-User. OpenAI says ChatGPT may visit a page with this agent when a user asks a question, that it is not used for automatic crawling, that robots.txt rules may not apply because the action is user initiated, and that it does not decide search inclusion.
  • Perplexity-User. Perplexity's crawler docs say it visits pages to answer a user's question and may link the page in its response, and that because a user requested the fetch, it generally ignores robots.txt rules.
  • Claude-User. Anthropic says Claude may access websites with this agent when people ask Claude questions, and that disabling it prevents Claude retrieving your content in response to a user query, which may reduce your visibility for user directed web search.

You rarely need rules for these. Blocking a user fetcher mostly means the assistant cannot read your page at the moment a buyer asks about you.

The robots.txt lines

A robots.txt file is made of groups: one or more user agent lines followed by rules. Under the robots.txt standard, RFC 9309, a crawler obeys the group that names it and only falls back to the group for all crawlers (the asterisk) when no group matches. If nothing matches at all, nothing is blocked.

So if your file has no rules for these bots and no blanket block, the search crawlers are already allowed. If it has a blanket Disallow under the asterisk, add a named group so the search crawlers are let back in.

Search crawlers you want to be cited by:

  • User-agent: OAI-SearchBot
  • User-agent: PerplexityBot
  • User-agent: Claude-SearchBot
  • Allow: /

Optional training opt out:

  • User-agent: GPTBot
  • User-agent: ClaudeBot
  • Disallow: /

Add Google-Extended to the second group only if you have decided the Gemini grounding trade off above is worth it. Anthropic says to add its rules for every subdomain you want covered, and that blocking its IP addresses instead may not work as an opt out because it stops Anthropic reading your robots.txt at all.

Then wait. OpenAI says ChatGPT search can take about 24 hours to adjust after a robots.txt change, and Perplexity says up to 24 hours.

The CDN and firewall check

robots.txt is a request. Your CDN or firewall decides who actually gets through, and it can block a crawler your robots.txt allows.

Cloudflare is the case most sites need to check. On July 1, 2025 Cloudflare said it was changing its default to block AI crawlers. Its current AI bot documentation splits bots into Search, Agent and Training, and says that from September 15, 2026, new domains get defaults that block Training and Agent bots on pages that display ads while leaving Search allowed. It adds that crawlers which mix search and training are blocked by any setting that blocks AI training. Agent covers chat fetch bots, so on a new domain with ads, a default you never chose can stop an assistant reading those pages when a buyer asks about you.

Perplexity's docs add a second trap: Cloudflare's Bot Fight Mode cannot be skipped with WAF custom rules, so with it switched on, Perplexity's bots can still be challenged even if you wrote an allow rule.

What to check:

  1. Open your CDN's bot or AI crawler settings and read what each category is set to, rather than assuming.
  2. Look for bot protection features that challenge automated traffic across the whole site.
  3. If you allow bots by IP, use the ranges each company publishes: OpenAI and Perplexity both link JSON files of IP ranges from their crawler pages.

How to confirm it worked

Check your server or CDN logs for each search crawler's user agent and look at the response codes on your key pages. A 200 means it got the page. A 403 or a challenge page means something in front of your site is still blocking it. Because user agent strings can be faked, match the request IPs against the published ranges before trusting what you see.

Being crawlable makes you eligible; it does not make an engine name you. The quickest way to see which engines cite you today is our free checker. It runs five buyer prompts on Google AI Overviews, ChatGPT, Gemini and Perplexity and shows every answer and the pages each engine cited, in about two minutes, with no email. For the rest of the on site work, from rendering to page structure, see our guide to LLM SEO.

Quick checklist

  1. Allow OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot.
  2. Block GPTBot and ClaudeBot only if you want a training opt out.
  3. Decide on Google-Extended knowing it also covers Gemini Apps grounding.
  4. Check your CDN's AI bot categories and bot protection settings.
  5. Confirm 200 responses in your logs after a day.

Questions

Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI says its settings are independent: OAI-SearchBot decides whether a site can be shown in ChatGPT search answers, and GPTBot is the training crawler. You can allow the first and disallow the second.
What is ClaudeBot?
ClaudeBot is Anthropic's crawler for collecting web content that could contribute to training its models. Anthropic says restricting it signals that a site's future materials should be excluded from training datasets. Search and user requests use separate bots, Claude-SearchBot and Claude-User.
Does Google-Extended control AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode are governed by Googlebot and the snippet controls. Google-Extended covers Gemini training and grounding in Gemini Apps and Vertex AI.
How long do robots.txt changes take to apply?
OpenAI says it can take about 24 hours for ChatGPT search to adjust, and Perplexity says up to 24 hours. Check your server logs after a day to confirm the bots are getting through.
Free scan

Check your AI visibility free

See which buying questions ChatGPT, Perplexity, Gemini and Google's AI answers name you for, and who they name instead. No call, no card.