GPTBot, ClaudeBot, PerplexityBot: which AI crawlers to block in robots.txt (and which not)
Training bots vs. AI search bots: how to keep your content out of model training without disappearing from ChatGPT, Claude, Perplexity and Google AI answers.
Updated
Many websites added “block AI” rules to their robots.txt — and accidentally made themselves invisible in AI search. The key is that AI companies run different crawlers for different purposes: some collect training data, others fetch pages to answer questions and cite sources. You can block one group and allow the other.
AI search crawlers — allow these if you want to be cited
Googlebot— Google Search; AI Overviews are built from Google's search index.Bingbot— Bing, which also powers Microsoft Copilot answers.OAI-SearchBot— ChatGPT search. Blocking it removes your pages from ChatGPT's search answers.Claude-SearchBot— indexes pages for Claude's search results.PerplexityBot— builds Perplexity's search index.
Training crawlers — blocking them is your choice
GPTBot— content that may be used to train OpenAI's models.ClaudeBot— content that may be used to train Anthropic's models.Google-Extended— controls use for Gemini training and grounding. Google says it does not affect inclusion or ranking in Google Search. It isn't a separate crawler; Googlebot fetches the pages.CCBot— Common Crawl, an open dataset used by many AI projects.
User-triggered fetchers
ChatGPT-User, Claude-User and Perplexity-User visit a page when someone asks the assistant to open it. According to the vendors' documentation, OpenAI's and Perplexity's user fetchers may not follow robots.txt because a person requested the page; Anthropic says Claude-User does follow it.
Example: block training, stay visible in AI search
# Keep content out of AI model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Everyone else (incl. Googlebot, Bingbot, OAI-SearchBot,
# Claude-SearchBot, PerplexityBot) may crawl
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlNote how robots.txt works: a crawler follows the most specific User-agent group that matches its name, otherwise the * group. So the rules above don't affect the search crawlers.
Common mistakes
- Blocking
OAI-SearchBotorPerplexityBotwhile trying to block training — you lose AI search citations. - Using outdated names like
anthropic-aiorClaude-Web, which Anthropic has deprecated. - A firewall or bot-protection rule (e.g. in Cloudflare) blocking AI crawlers even though robots.txt allows them. Check your security settings too.
- Expecting robots.txt changes to apply instantly — crawlers re-read it periodically, which can take up to a day.
Check your page in seconds
Free, no signup: see which of 12 AI crawlers your robots.txt blocks, plus the page signals AI search relies on.
Check your site with the free AI visibility checker →