The short answer
Let most AI crawlers in. If you want to be named and cited by ChatGPT, Perplexity, Copilot and Google AI Overviews, their crawlers have to read your pages first. Keep the search and fetch agents open, and treat training crawlers like GPTBot as a separate, deliberate choice. Never use a blanket block, because it can also remove you from normal search.
Let most AI crawlers in, block the ones that only take
Let them in, with a few exceptions. If you want AI assistants to name and cite your business, their crawlers have to reach your pages first. Blocking everything is the fastest way to disappear from ChatGPT, Google AI Overviews, Perplexity and Copilot. The smart move is selective. Allow the crawlers that feed live answers, and block the ones that only harvest text to train a model with nothing offered back to you.
The decision is not all or nothing. Each assistant uses named user agents, and each one has a different job. Some fetch a page the moment a user asks a question. Others crawl in the background to build a search index. A third group scrapes text purely for model training. You can say yes to the first two and no to the third, all inside a single robots.txt file.
Get the reasoning right before you touch a setting. Ask what you actually want from AI search. Most businesses want to be recommended, quoted and linked. That requires access. A few businesses hold content they never want reused, such as paid research or member only guides. That requires restriction. Your robots.txt should reflect that choice on purpose, not by accident, because a stray line can remove you from answers overnight.
Which crawlers do what
Know the names before you write a rule. AI crawlers announce themselves with a user agent string in your server logs. GPTBot and CCBot mostly gather training data. OAI-SearchBot and ChatGPT-User fetch pages for live ChatGPT answers. Google-Extended controls training use, while Googlebot still powers AI Overviews. PerplexityBot and Bingbot support Perplexity and Copilot. ClaudeBot serves Anthropic. Treating them as one blob is the mistake.
- GPTBot collects data OpenAI can use for training. Optional for visibility.
- OAI-SearchBot and ChatGPT-User fetch live pages so ChatGPT can cite you. Keep open.
- Google-Extended is a training toggle. Blocking it does not remove you from Google Search.
- PerplexityBot feeds Perplexity answers and citations. Keep open.
- Bingbot powers Microsoft Copilot and Bing. Keep open.
- CCBot is Common Crawl, a dataset many models reuse. Optional.
Notice the pattern. The crawlers that put your name in an answer are separate from the crawlers that only feed training sets. If you want citations, the search and fetch agents matter most, so leave them open. The training agents are a genuine choice, and reasonable owners land on different answers. What no one should do is block the search fetchers, because that quietly deletes you from the results you were trying to win.
How to allow answer engines but stop the scrapers
Start with a plan, then edit one file. Your robots.txt sits at the root of your domain and tells every crawler what it may read. You allow a crawler by naming its user agent and leaving the path open. You block one by naming it and disallowing the paths you want held back. The rules are read top to bottom, so keep them tidy and comment each line so the next person understands it.
List your goals
Decide whether you want citations, training exposure, or both. Write it down before editing anything.
Keep the search fetchers open
Leave OAI-SearchBot, ChatGPT-User, PerplexityBot, Bingbot and Googlebot able to read your pages.
Decide on training agents
Make a deliberate call on GPTBot, Google-Extended, CCBot and ClaudeBot.
Write the rules
Add only the named user agents and paths you mean. Never disallow the whole site by accident.
Test and monitor
Check the live file at yourdomain.com/robots.txt, then watch your logs.
Give the change time and then verify it. Crawlers cache robots.txt, so a new rule can take a few days to take effect. Check your server logs after a week to confirm the agents you allowed are still visiting and the ones you blocked have stopped. Google Search Console and Bing Webmaster Tools both show crawl activity for their bots. If a page you want cited is not being fetched, your rules are the first place to look.
What blocking actually costs you
Block the wrong crawler and you vanish from the answer. AI assistants can only quote text they are allowed to read. If ChatGPT-User cannot fetch your page, ChatGPT has nothing of yours to cite, and it names a competitor instead. The cost is invisible because nothing breaks on your site. You simply stop appearing in a channel that more buyers check every month, and you rarely find out why.
Blanket blocking also hurts ordinary search. Some owners paste a wildcard rule that disallows every user agent, thinking they are only stopping AI. That same rule can stop Googlebot and Bingbot, which removes you from normal search results too. A robots.txt written in a hurry can undo years of work. Always name the specific agents you mean, and never disallow the whole site unless you truly want it hidden from everyone.
Remember what robots.txt cannot do. It is a request, not a lock. Well behaved crawlers from OpenAI, Google, Perplexity and Microsoft honour it, but a rogue scraper can ignore it entirely. If you need to enforce a block, use server rules or a service like Cloudflare that can filter by user agent and IP. For most businesses that want citations, though, the goal is access, so enforcement is rarely the point.
How to check what is already crawling you
Look before you decide. Your server logs already record every AI crawler that has visited, listed by user agent and timestamp. Most hosting panels let you download these logs, and your developer can filter them in minutes. Search the logs for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Bingbot. This tells you who is reading your content today, which is far more useful than guessing, and it shows whether any current rule is already blocking a bot you want.
- Server logs show the raw truth of which crawlers visited and when.
- Google Search Console reports how Googlebot crawls the pages behind AI Overviews.
- Bing Webmaster Tools shows Bingbot activity that feeds Copilot.
- Cloudflare or your CDN dashboard can list bot traffic by user agent.
- A plain robots.txt check at yourdomain.com/robots.txt shows the rules live right now.
Turn the data into a decision. Once you can see who visits, match each crawler to your goal from the first step. Keep every search and fetch agent open. Make a deliberate call on the training agents. Then write the smallest set of rules that gets you there and nothing more. Review the logs again a month later, because crawler names and behaviour change often, and a rule that was correct last year may need a fresh look.