Skip to content
SiteMedic

GPTBot, ClaudeBot, PerplexityBot: which AI crawlers to block in robots.txt (and which not)

Training bots vs. AI search bots: how to keep your content out of model training without disappearing from ChatGPT, Claude, Perplexity and Google AI answers.

Updated

Many websites added “block AI” rules to their robots.txt — and accidentally made themselves invisible in AI search. The key is that AI companies run different crawlers for different purposes: some collect training data, others fetch pages to answer questions and cite sources. You can block one group and allow the other.

AI search crawlers — allow these if you want to be cited

  • Googlebot — Google Search; AI Overviews are built from Google's search index.
  • Bingbot — Bing, which also powers Microsoft Copilot answers.
  • OAI-SearchBot — ChatGPT search. Blocking it removes your pages from ChatGPT's search answers.
  • Claude-SearchBot — indexes pages for Claude's search results.
  • PerplexityBot — builds Perplexity's search index.

Training crawlers — blocking them is your choice

  • GPTBot — content that may be used to train OpenAI's models.
  • ClaudeBot — content that may be used to train Anthropic's models.
  • Google-Extended — controls use for Gemini training and grounding. Google says it does not affect inclusion or ranking in Google Search. It isn't a separate crawler; Googlebot fetches the pages.
  • CCBot — Common Crawl, an open dataset used by many AI projects.

User-triggered fetchers

ChatGPT-User, Claude-User and Perplexity-User visit a page when someone asks the assistant to open it. According to the vendors' documentation, OpenAI's and Perplexity's user fetchers may not follow robots.txt because a person requested the page; Anthropic says Claude-User does follow it.

Example: block training, stay visible in AI search

# Keep content out of AI model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Everyone else (incl. Googlebot, Bingbot, OAI-SearchBot,
# Claude-SearchBot, PerplexityBot) may crawl
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Note how robots.txt works: a crawler follows the most specific User-agent group that matches its name, otherwise the * group. So the rules above don't affect the search crawlers.

Common mistakes

  • Blocking OAI-SearchBot or PerplexityBot while trying to block training — you lose AI search citations.
  • Using outdated names like anthropic-ai or Claude-Web, which Anthropic has deprecated.
  • A firewall or bot-protection rule (e.g. in Cloudflare) blocking AI crawlers even though robots.txt allows them. Check your security settings too.
  • Expecting robots.txt changes to apply instantly — crawlers re-read it periodically, which can take up to a day.
Not sure what your robots.txt currently allows? The AI visibility checker tests all 12 crawlers in seconds.

Check your page in seconds

Free, no signup: see which of 12 AI crawlers your robots.txt blocks, plus the page signals AI search relies on.

Check your site with the free AI visibility checker →