What are AI crawlers, and which ones actually matter?

What an AI crawler actually is

An AI crawler is an automated program that reads your website so an AI assistant can learn what you do. It is software, not a person. It visits your pages, copies the text, and files that content for later use. When someone asks ChatGPT or Perplexity a question, the assistant draws on what its crawlers collected. If a crawler never reads your site, the assistant has little reason to name you.

There are two jobs these programs do, and the difference matters. Training crawlers gather text to teach a model how language and topics fit together. Retrieval crawlers fetch pages in real time to answer a live question, then cite the source. GPTBot trains OpenAI models. OAI-SearchBot and ChatGPT-User fetch pages for answers with links. Knowing which job a crawler does tells you what blocking or allowing it will cost you.

Crawlers identify themselves with a user agent string, a short label in every request they send. Your server records that label in its access logs. This is how you tell a real GPTBot visit from ordinary traffic. Each major assistant runs its own named crawler, and each obeys the rules you set in a file called robots.txt. That file sits at the root of your domain and tells each named crawler what it may read.

Which crawlers matter and what each one does

The crawlers worth your attention belong to the four assistants people actually use to find businesses: ChatGPT, Google, Perplexity and Microsoft Copilot. Each runs one or more named agents. Some collect training data, some fetch pages live, and a few do both. The list below names the agents you will see in your logs and explains what each one is doing when it arrives.

• GPTBot collects text to train OpenAI models, including future versions of ChatGPT.
• OAI-SearchBot and ChatGPT-User fetch pages live when ChatGPT answers a question, and can cite you with a link.
• Google-Extended controls whether your content feeds Gemini and Google AI Overviews.
• PerplexityBot and Perplexity-User read and fetch pages so Perplexity can answer and cite sources.
• Bingbot feeds the Bing index that Microsoft Copilot relies on.
• ClaudeBot and Applebot-Extended gather content for Anthropic's Claude and Apple's AI features.

One name deserves a note. Google-Extended is not a crawler that visits your site. It is a permission flag. Googlebot still crawls your pages for normal search, and Google-Extended decides whether that same content can feed Gemini and AI Overviews. Blocking Google-Extended does not remove you from search results. It only withdraws your pages from Google's AI answers, which is rarely what a business wants.

Bingbot matters more than its name suggests. Microsoft Copilot draws heavily on the Bing index, so a page that Bingbot cannot read is a page Copilot is unlikely to mention. Applebot-Extended and Amazonbot are smaller today, but they follow the same pattern. The safe rule is simple. Allow the retrieval crawlers from every assistant your customers use, and only consider blocking training crawlers if you have a firm reason.

How to check whether these crawlers reach you

Open your server access logs and search for the user agent names listed above. If you see GPTBot, PerplexityBot and Bingbot arriving over the last month, the assistants can read you. If you see nothing, something is stopping them. Most hosting dashboards expose these logs, and tools like Cloudflare, your CDN or a plugin can filter requests by user agent so you do not have to read raw files.

Do this check once a month, not once a year. Crawlers change their names and add new ones, and a site migration or a new firewall rule can silently start blocking them. A single wrong line in robots.txt can hide your whole site from an assistant while normal visitors notice nothing. Diarise a short review, keep a note of which agents you saw, and you will catch a problem in weeks rather than months.

Crawling is not the same as being cited

Being crawled is necessary, but it does not earn a mention on its own. A crawler can read every page you own and the assistant can still recommend a competitor. Access is the first gate, not the finish line. Once the crawlers can reach you, the assistant chooses whom to name based on how clearly your pages answer the question and how well other sources back you up.

Assistants favour pages that state an answer plainly, near the top, in language that matches the question. They prefer sources that other reputable sites mention, that carry a clear author and date, and that are easy to parse. A page buried in marketing language or hidden behind a script the crawler cannot run is a page that gets skipped. Structure and clarity do the work that keywords once did.

This is why access and content have to move together. Open the gates, then give the assistant something worth quoting: a direct answer, real specifics, and a page that reads the way a person asks the question. Do both and you move from invisible to citable. Do only the first and you have simply given a crawler permission to ignore you politely.

What to do next this week

Start with three moves you can finish in an afternoon. Confirm the crawlers can reach you, fix robots.txt if they cannot, then rewrite your most important page so it answers its main question in the first two lines. That order matters. There is no point polishing a page that no assistant is allowed to read, and no point opening the gates to a page not worth citing.

• Confirm access. Check your logs for GPTBot, PerplexityBot and Bingbot in the last month.
• Fix the gate. Remove any robots.txt rule that blocks the retrieval crawlers you want.
• Answer first. Rewrite your key page so the main question is answered in the opening lines.
• Add proof. Give each page a clear author, a date, and real specifics.

Then measure. Ask each assistant the questions your customers ask and note whether you appear. Repeat the same prompts every few weeks so you can see movement. Assistants update their answers on their own schedule, so give changes time to land, often several weeks. Keep a simple record of what you asked and what came back, and you will know whether the work is paying off.

Published 10 September 2026 by Carpe DM Strategies. We run this work on our own business every day, which is where the examples come from.