The short answer
Fetch your own pages while identifying as each AI crawler and confirm you get a 200 response with your real page content in the body. A 403, a 503, a challenge page or an empty shell means you are blocked. Your robots.txt is not proof, because hosting and CDN layers block above it.
Why the robots.txt file is not the answer
Almost everyone checks robots.txt, sees a permissive rule, and concludes they are fine. That check is incomplete, and the gap is where most blocked sites live.
Robots.txt is a request. It asks well-behaved crawlers to respect a rule. The layers above it, your CDN and your host, enforce. Several major providers added AI bot blocking as a default, which means a site can publish a perfectly open robots.txt while its edge quietly returns a refusal to every AI crawler that arrives.
The only reliable test is to behave like the crawler and look at what comes back.
The check, step by step
Fetch as each bot
Request your homepage and two important inner pages while sending each crawler's user agent string. Cover GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended, ClaudeBot, Bingbot and CCBot.
Read the status code
You want 200. A 403 is an explicit block. A 503 or a 429 is usually a bot-management challenge. A 301 or 302 is fine only if the destination also returns 200 for the same agent.
Read the body, not just the code
A 200 can still be a failure. If the body is a challenge page, a cookie wall, or an empty shell waiting for JavaScript to fill it, the crawler got nothing. Search the returned HTML for a distinctive sentence from your page.
Check your sitemap and its URLs
Confirm the sitemap is reachable, declared in robots.txt, and that the URLs inside it match the canonical form your pages actually use. A trailing slash mismatch is enough to split one page into two weak ones.
Look at your server logs
The definitive evidence. Filter your access logs for the AI crawler user agents over the last week. Real, repeated hits returning 200 mean the door is genuinely open. Silence means it is not, whatever your configuration says.
What to do when you find a block
If it is robots.txt, allow the crawlers you want and be deliberate about the ones you do not. Blocking training crawlers is a legitimate choice with a real cost, and it should be a decision rather than a default.
If it is your CDN or host, find the AI bot control in its dashboard and turn the blocking off for the crawlers you want. This is usually a single toggle, and it is usually the largest single improvement available to a site that has never checked.
If the body comes back empty because your pages are rendered by JavaScript in the browser, that is a build problem rather than a permissions problem. Retrieval systems are far more reliable with content present in the initial HTML.
Then re-run the whole check. A fix you have not verified from the crawler's point of view is not a fix.
How often to re-check
Monthly is sensible, and after any change to your host, CDN, security settings or platform. Blocking is frequently reintroduced by an update or a new default, and it fails silently. Nothing breaks, no alert fires, and your traffic from assistants simply stops growing.