Is your website blocking ChatGPT & Claude without you knowing?

Plenty of websites turn AI tools away and their owners have no idea. The cause is usually a rule in a small text file, a Cloudflare setting or a security plugin. Blocking the crawlers that collect pages to train AI models is a fair choice. Blocking the ones that fetch pages to answer questions keeps you out of the answers.

You'd notice fast if your website stopped loading for customers. You wouldn't notice if it stopped loading for ChatGPT. There's no error message and no drop in visitors you can point to. The tool just stops mentioning you, or never starts.

This post explains the two kinds of AI crawler, where blocks usually hide and how to check your own site.

AI companies send two kinds of crawler.

A crawler is a program that visits web pages and reads them. AI companies run several, and they do different jobs.

Training crawlers collect pages to help build future AI models. OpenAI's is GPTBot and Anthropic's is ClaudeBot. CCBot belongs to Common Crawl, a non-profit that keeps a free public archive of the web, which AI companies have long used as training data. Google-Extended and Applebot-Extended work differently. They aren't separate crawlers. They're names you can use in your rules to say whether Google and Apple may use the pages they already crawl for their AI models.

Answer crawlers fetch pages so a tool can find you, quote you and link to you when someone asks a question. Some build a search index ahead of time. Others open a page on the spot when a person asks about it.

Blocking the answer crawlers keeps you out of the answers.

The companies say so themselves. OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT's search answers. Anthropic says blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude's answers. Perplexity says PerplexityBot is how websites get into its search results, and recommends allowing it.

Blocking Googlebot takes you out of Google altogether, AI Overviews included. Blocking Google-Extended has no effect on that: Google says it doesn't change whether you appear in Google Search or how you rank. Apple says the same of Applebot-Extended and its own search features.

The training crawlers are a separate decision. Plenty of businesses would rather their pages didn't train AI models, and blocking GPTBot or ClaudeBot says exactly that. OpenAI and Anthropic use separate crawlers for training and for search, so you can block one and allow the other. Our free check lists the training crawlers for your information and doesn't mark you down for blocking them.

The block usually hides in one of three places.

Your robots.txt file

Every website can have a small text file at yourdomain.com/robots.txt. It names crawlers and says which parts of the site each one may visit. A crawler named in a "User-agent" line, followed by "Disallow: /", is told to stay out of the whole site.

Two patterns cause trouble. One is a blanket rule under "User-agent: *", which means every crawler, followed by "Disallow: /". It can survive from a test version of the site that was never meant to be public. The other is a list of AI crawlers copied from a guide, which blocks the answer crawlers along with the training ones.

One more thing to know. OpenAI and Perplexity say the fetchers that open a page when a person asks (ChatGPT-User and Perplexity-User) may not follow robots.txt, because a person asked for the page. So your robots rules mainly control the search crawlers. A firewall can stop all of them.

Cloudflare

Cloudflare sits in front of many websites and filters their traffic before it arrives. Its AI settings have changed several times:

• In July 2025 it began blocking AI training crawlers by default on new sites, and asking owners up front whether to let AI crawlers in. It said more than a million customers had already switched on its one-click block.
• In July 2026 it split AI traffic into three groups: Search, Agent and Training. Agent covers tools acting for a person in real time, such as a chatbot opening a page because someone asked about it.
• Since 15 September 2026, new sites block Training and Agent on pages that show ads, and allow Search. New sites without ads allow all three. Existing sites keep whatever they had.

Cloudflare also warned that from 15 September a site set to block Training could block crawlers that do search and training at once, and it named Googlebot, Applebot and Bingbot. So if you, or whoever set up your site, ticked a box to block AI at any point, check what that box covers today.

A security plugin or your host's firewall

Some WordPress security plugins come with lists of AI crawlers to block, and they're easy to switch on without reading the list. Some hosting companies block AI crawlers on their own servers, where nothing in your site's settings shows it. A cap on how many pages one visitor may request can have the same effect more quietly, turning a crawler away after a few pages.

A firewall doesn't read your robots.txt file. It makes its own decisions. That's why OpenAI and Perplexity publish the internet addresses their crawlers use and ask site owners to let them through.

How to check your own site.

Start with the rules file. Type your web address followed by /robots.txt into a browser. Look for the crawler names in the table above, and for "Disallow: /" under any of them or under "User-agent: *". If there's no file at all, nothing is blocked there.

Then ask whoever manages your Cloudflare account or your hosting whether any AI or bot blocking is switched on, and what it covers.

Or let our free AI Visibility Check do both. It reads your robots rules for every crawler in the table, and it visits your site as ChatGPT's, Claude's and Perplexity's crawlers to see whether your firewall lets them in. It takes about 30 seconds and changes nothing on your site.

Let the answer crawlers in.

Allow every answer crawler: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot and Bingbot. They're how AI tools find you, read you and link to you.

The training crawlers are your call. Blocking them doesn't stop the answer crawlers reading your site.

Once a change is made, OpenAI says it takes about 24 hours to reach ChatGPT's search, and Perplexity says up to 24 hours for its own systems.

The tune-up removes the block and confirms they get in.

If the check finds a block, the AI Search Tune-Up removes it. We correct the robots rules, and with your login we change the Cloudflare or hosting settings that turn answer crawlers away. Then we run the check again to confirm they get in. Any training crawler blocks you want to keep stay in place.

The tune-up covers the rest of what AI tools need as well: readable pages, clear service pages and structured data. It takes one to two weeks and starts at $1,500 for a site of up to 30 pages. For what happens after the crawlers get in, read how ChatGPT, Claude and Perplexity decide who to recommend, or run the free check to see whether your site needs the work.