AI crawlers and robots.txt: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot explained

Most AI companies now use different crawlers for different jobs: one to build a search index that answers can cite, one to collect training data, and one that fetches a page when a user asks. That split matters, because a robots.txt rule meant to block AI training can also make you invisible in AI search if you block the wrong bot.

The main AI crawlers at a glance

Google also offers a Google-Extended token, which limits use of your content for training and grounding in some of Google's other AI systems. It is not a separate crawler and does not change how you appear in Google Search.

Search bots vs training bots

Think of it as two questions. Do you want AI assistants to find and cite your pages? Then allow the search bots. Do you want your content used to train future models? That is a separate choice controlled by the training bots. OpenAI states that GPTBot and OAI-SearchBot are controlled independently, and Anthropic and Perplexity document their bots the same way.

For most publishers and small businesses, being cited in AI answers is valuable traffic and brand exposure, so blocking the search bots is rarely a good idea.

Recommended robots.txt for AI visibility

This version keeps you visible in AI search and opts out of training. Delete the training section if you are happy for your content to be used for training too.

1. User-agent: OAI-SearchBot / Allow: /
2. User-agent: Claude-SearchBot / Allow: /
3. User-agent: PerplexityBot / Allow: /
4. User-agent: GPTBot / Disallow: / (training opt-out)
5. User-agent: ClaudeBot / Disallow: / (training opt-out)
6. User-agent: Google-Extended / Disallow: / (optional)
7. User-agent: * / Allow: /
8. Sitemap: https://yourdomain.com/sitemap.xml

Each "/" pair goes on two lines in the real file: the User-agent line, then the Allow or Disallow line. A plain User-agent: * / Allow: / already lets every well-behaved bot in, which is what Trendifico uses.

How to check what you are blocking

• Open yourdomain.com/robots.txt and search for the user-agent names above.
• Check for a blanket User-agent: * / Disallow: /, often left over from a staging site.
• Look at your CDN or firewall: some "block AI bots" toggles block search bots too.
• Use Search Console URL Inspection to confirm Googlebot can fetch key pages.
• Review server logs for the user-agent strings to see who actually visits.

Common mistakes

• Blocking GPTBot and thinking you are out of ChatGPT. Search uses OAI-SearchBot.
• Blocking OAI-SearchBot to stop training. That removes you from ChatGPT search without stopping training by other bots.
• Relying on robots.txt for private content. It is a request, not security. Use login or noindex for anything sensitive.
• Expecting instant changes. Some providers say robots.txt updates can take about a day to apply.

Want to go further? Read how to check if ChatGPT recommends your brand and what llms.txt is and whether you need one. To see how much of your text fits in a model's context, try the AI token counter.

FAQ

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is for training. ChatGPT search uses OAI-SearchBot, which you control separately.

Do AI crawlers respect robots.txt?

The major search and training bots from OpenAI, Anthropic and Perplexity say they follow robots.txt. Fetchers acting on a direct user request may not apply the same rules.

Does Google-Extended affect my Google rankings?

No. It controls use of content for some of Google's AI training and grounding outside Search. Googlebot still crawls for Search, AI Overviews and AI Mode.

Should a small business block AI bots?

Usually block training bots only if you feel strongly about it, and keep search bots allowed so assistants can cite and link to you.