AI crawlers: how to control GPTBot, ClaudeBot and more

AI crawlers GPTBot, ClaudeBot and PerplexityBot now make up a large and growing share of automated traffic, and most site owners still manage them with a single blunt rule or none at all. According to Cloudflare network data, training now drives close to 80% of AI bot activity, while live search and user-triggered fetches make up the rest. That split matters, because the bot that scrapes your content for a training dataset is often a different user-agent from the one that cites you inside an answer. Get the distinction wrong and you either leak your whole archive into training sets or you quietly delete yourself from the AI engines. This guide shows the exact robots.txt rules for each major bot, a decision table by goal, and the one thing robots.txt cannot actually do.

What are AI crawlers like GPTBot, and how do you control them?

AI crawlers are automated bots operated by AI companies that request your pages and read the HTML, exactly like a traditional search bot. The difference is intent: some collect text to train large language models, some fetch pages in real time to build a cited answer, and some act on behalf of a single user who asked a question.

You control them the same way you control Googlebot, through the robots.txt file at the root of your domain (for example https://yourdomain.com/robots.txt). Each bot publishes a user-agent token. You write a block naming that token, then add an Allow or Disallow directive. This is the same standard covered in our robots.txt configuration guide, applied to a new generation of bots. The catch is that AI companies now run several bots each, so a single rule is rarely the right answer.

What is the difference between training crawlers and search crawlers?

Training crawlers gather content to teach future model versions. Search crawlers fetch pages so the model can quote a current source in a live answer. Blocking one does nothing to the other, because they use separate user-agents.

Both OpenAI and Anthropic now split their bots into three roles. OpenAI's crawler documentation lists GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for direct user fetches. Anthropic's crawler documentation mirrors this with ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for user-initiated requests.

The practical lesson: treat "block AI" and "stay visible in AI answers" as two separate switches, not one.

How do you allow or block GPTBot, ClaudeBot and PerplexityBot in robots.txt?

Name the user-agent, then add a directive. To block a bot from your whole site, use Disallow: /. To let it in, use Allow: / or simply leave no Disallow rule.

To block the training crawlers only, you would add these blocks: User-agent: GPTBot then Disallow: /, followed by User-agent: ClaudeBot then Disallow: /, and User-agent: Google-Extended then Disallow: /. To keep the search crawlers welcome so you remain citable, add User-agent: OAI-SearchBot then Allow: /, User-agent: Claude-SearchBot then Allow: /, and User-agent: PerplexityBot then Allow: /.

Three rules that trip people up:

• Every user-agent line needs at least one Allow or Disallow line under it, or the block is ignored.
• User-agent tokens are matched by the bot itself, so spelling and case should follow the operator's published token.
• If you block a training bot, you often want to also expose an llms.txt file and keep your XML sitemap clean, so the bots you do allow can find your best pages fast.

Should you block AI crawlers, or let them in?

There is no universal answer, only a goal. Blocking protects content from training use but can cost you citations. Allowing maximizes visibility in the AI engines but hands your archive to model builders. Map the decision to what you actually want.

This goal-based split is the information-gain element most "block AI bots" posts skip: they hand you a copy-paste block-everything file without warning that it also erases you from ChatGPT and Perplexity results. For most businesses that want to be recommended by the AI engines, the middle row is the sensible default. If your priority is being cited, pair this with the on-page work in our technical SEO guide and monitor how much AI traffic you actually earn through AI traffic tracking in GA4.

Does blocking GPTBot remove you from ChatGPT?

No. Blocking GPTBot removes you from OpenAI's training crawl, not from ChatGPT search. Those run on different user-agents. If you disallow GPTBot but keep OAI-SearchBot allowed, ChatGPT can still fetch and cite your pages in its search-grounded answers.

This is the single most common mistake. A site owner reads "block GPTBot to protect your content," ships Disallow: / for every bot they can find, and then wonders why they stopped appearing in AI answers. OpenAI's documentation is explicit that opting out of OAI-SearchBot means a site "will not be shown in ChatGPT search answers," which is a separate choice from opting out of training. Decide each switch on its own.

Do AI crawlers actually respect robots.txt?

The major operators say yes. OpenAI, Anthropic, Google and Perplexity all state in their documentation that their crawlers honor robots.txt directives. Anthropic notes that all three of its bots respect robots.txt, including the non-standard crawl-delay extension, as reported by Search Engine Land.

But robots.txt is a request, not a wall. Compliance is voluntary and not technically enforced, so a bot that ignores the file will still reach content that your server hands out to anyone. If you need genuine enforcement rather than a polite opt-out, you need server-level controls: user-agent and IP blocking at the firewall or CDN, rate limiting, or authentication behind a login. Robots.txt handles the well-behaved majority. Server rules handle the rest, and they also protect your crawl budget from aggressive bots.

How much of your traffic is now AI crawlers?

Enough to matter. Cloudflare's network data shows GPTBot's share of AI crawler traffic grew from 4.7% to 11.7% between July 2024 and July 2025, while ClaudeBot rose from 6% to about 9.9%, per its 2025 crawler report covered by Search Engine Journal.

The bigger story is the imbalance between what these bots take and what they return. Cloudflare's crawl-to-refer analysis found Anthropic crawled roughly 38,000 pages for every visitor it sent back in July 2025 (down sharply from about 286,000 to 1 in January), while OpenAI's ratio sat near 1,091 to 1 and Google's near 5.4 to 1. In plain terms, AI training bots consume far more content than they send back as traffic, which is exactly why the allow-or-block decision is now a business question, not just a technical one. If bots are hammering your server, a crawl budget review plus firewall rules will pay for itself.

How do you verify a bot is really GPTBot or ClaudeBot?

User-agent strings can be faked, so never block or allow based on the name alone if enforcement matters. Verify against the operator's published IP ranges. OpenAI publishes machine-readable IP lists for each bot (gptbot.json, searchbot.json and chatgpt-user.json), and you can confirm a request came from a claimed range before acting on it.

The reliable pattern is a reverse DNS lookup on the requesting IP, followed by a forward lookup to confirm it resolves back to the operator's domain, then a check that the IP falls inside the published range. This is standard log-analysis hygiene and the same discipline you would apply to any bot audit. If you keep server logs, you can measure real AI crawler behavior directly rather than guessing, which is the most honest way to decide whether a given bot earns its access.

Frequently asked questions

Related guides