robots.txt for AI Crawlers: Setting Up GPTBot, OAI-SearchBot, ClaudeBot and Co.
The big AI providers now separate crawlers for training from crawlers for search. If you want to appear as a source in ChatGPT, Claude or Perplexity, you should at least allow the search crawlers. Whether you allow training is a separate decision.
The key AI crawlers
| Provider | Crawler | Purpose according to provider |
|---|---|---|
| OpenAI | OAI-SearchBot | Search in ChatGPT |
| OpenAI | GPTBot | Training |
| OpenAI | ChatGPT-User | Fetch when a user asks |
| Anthropic | Claude-SearchBot | Search results in Claude |
| Anthropic | ClaudeBot | Training |
| Anthropic | Claude-User | Fetch when a user asks |
| Perplexity | PerplexityBot | Search in Perplexity |
| Perplexity | Perplexity-User | Fetch when a user asks |
| Googlebot | Google Search, including AI Mode and AI Overviews | |
| Google-Extended | Controls use for Gemini training, no ranking effect |
Note: according to the providers, user fetches (ChatGPT-User, Perplexity-User) may not follow robots.txt in the same way, because they act on behalf of a person.
Example: be visible, allow training
User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / Sitemap: https://www.your-domain.at/sitemap.xml
This is how digtatur.com is set up. If you want to exclude training but stay visible in search, set Disallow: / for GPTBot, ClaudeBot and Google-Extended and keep the search crawlers allowed.
Common mistakes
- Blanket blocking: some security plugins or hosting firewalls block all bots. Check the server logs.
- Blocking Googlebot “to stop AI”: that removes you from all of Google Search.
- Content only via JavaScript: not all AI crawlers run JavaScript. Key texts belong in the HTML.
And llms.txt?
llms.txt is a proposal for an overview file for language models. It is not an official standard, and Google says it does not use it. It does no harm and is quick to create, but does not replace a good robots.txt and clear content.
We are happy to check and set this up for you: GEO Agency Vienna. Platform details: ChatGPT, Claude, Perplexity.
Frequently asked questions
Where do I find my robots.txt?
At your-domain.at/robots.txt. In WordPress it is often generated by an SEO plugin.
Will I rank worse on Google if I block GPTBot?
No. GPTBot belongs to OpenAI and has nothing to do with Google ranking.
