robots.txt and AI crawlers

AI companies crawl the web with named bots. Your robots.txt file is where you tell them what they can touch — and unlike llms.txt, every major crawler actually respects it.

Three different jobs

Separate what these bots do. Some index for AI search, building the indexes behind ChatGPT, Claude, Gemini, and Perplexity answers. Some fetch on demand, pulling a page in real time when a user asks an assistant about it. And some crawl for training, collecting content for future models. You may want different rules for each. Most businesses allow search and on-demand access — that’s how customers find you in AI answers — and decide deliberately about training.

The bots to know in 2026

Names change, so confirm against each vendor’s current docs, but as of 2026:

OpenAI: GPTBot (training), OAI-SearchBot (indexes for ChatGPT search — opt out and you won’t appear in ChatGPT search answers), ChatGPT-User (on-demand fetch).
Anthropic: ClaudeBot (training), Claude-SearchBot (search indexing), Claude-User (on-demand fetch).
Google: Google-Extended governs use of your content for Gemini and AI training, separate from Search. It’s a robots.txt token only — there’s no matching user agent, so you can’t block it with server rules.
Perplexity: PerplexityBot (indexing) and Perplexity-User (on-demand — note it does not reliably honor robots.txt).
Common Crawl: CCBot, an open dataset many models train on.

The mistakes to avoid

In a rush to “block AI,” some businesses disallow everything and erase themselves from the AI results customers now use. Two specific traps: there’s no separate user agent for Google’s AI Overviews, so blocking Googlebot to avoid them also removes you from regular Google Search; and opting out of the search indexers (OAI-SearchBot, Claude-SearchBot) drops you from those AI search surfaces. For a local business trying to be found, allow the search and on-demand crawlers, and be selective about training only.

A reasonable starting point

# Allow AI search + on-demand (how customers find you)
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Decide deliberately about training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow:            # leave open, or use Disallow: / to opt out

User-agent: *
Allow: /

Pair this with an llms.txt summary if you add one. Scan your site to check your robots rules.

See where your site stands

Scan your site free and get your agent-readiness score across all five categories.

Scan your site