Home › GEO & AEO › AI crawlers

robots.txt for AI crawlers: GPTBot, ClaudeBot & co.

If AI systems cannot crawl your site, they cannot mention you. Your robots.txt decides which AI crawlers are allowed in, and an accidental block is one of the most common reasons businesses are invisible in AI answers.

At a glance

AI crawlers like GPTBot, Google-Extended, ClaudeBot, PerplexityBot and CCBot fetch web content for AI systems. Your robots.txt grants or denies them access. To be cited in AI answers, allow the ones you care about; to keep content out of AI, block them. The key mistake to avoid: blocking crawlers you actually want.

What are AI crawlers?

They are bots that AI providers use to read the web, either to train models or to fetch live content for answers. Each identifies itself with a user-agent name you can target in robots.txt.

GPTBot (OpenAI): training data.
OAI-SearchBot (OpenAI): content for ChatGPT search.
Google-Extended (Google): controls use for Gemini/AI, separate from normal Google indexing.
ClaudeBot / anthropic-ai (Anthropic).
PerplexityBot (Perplexity).
CCBot (Common Crawl, used by many AI datasets).

How robots.txt controls them

robots.txt sits at the root of your domain and tells crawlers what they may fetch. You can set rules per user-agent. Reputable AI crawlers honor it.

Example: allow AI crawlers, keep private areas out

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://yourdomain.com/sitemap.xml

A wildcard User-agent: * that allows crawling already lets the AI bots in. If you previously added per-bot Disallow rules for GPTBot & co., remove them to become visible in AI answers.

The trade-off

Blocking AI crawlers keeps your content out of training and answers, which some publishers want. But if your goal is to be found and recommended by AI, blocking them is self-defeating. Decide deliberately, do not block by accident.

How klarava helps

klarava's free check tests whether the relevant AI crawlers are allowed on your site and flags any that are blocked, so you can fix an accidental block in minutes.

Check whether AI crawlers can reach your site, for free.

Run the free check

Frequently asked questions

Should I allow AI crawlers?

If you want to appear in AI answers (ChatGPT, Perplexity, Google AI Overviews), yes. Blocking them protects content from training but also removes you from those answers. Most businesses that want visibility should allow them.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot is OpenAI's crawler used for training data; OAI-SearchBot fetches content to show in ChatGPT search results. You can allow or block them independently in robots.txt.

Does robots.txt guarantee blocking?

robots.txt is a directive that reputable crawlers honor, not a hard technical block. To truly prevent access you need server-side measures. For AI visibility, the relevant point is simply: do not accidentally block the crawlers you want. More on GEO & AEO