Why AI tools can't read some websites, and how to tell if yours is one.
An AI tool can only describe your business from the words it can pull out of your pages. Some websites hide those words in text that only appears after JavaScript runs, inside images, in PDFs or behind forms. The site looks fine to you and close to empty to the tool.
Your website can look perfect in a browser and still give an AI tool almost nothing to work with. What you see is the finished page. What a crawler reads is what arrives before anything runs, and the two aren't always the same.
Here are the usual causes, how to test for each and what fixing them involves.
Most AI crawlers don't run JavaScript.
A web page arrives as code. On many modern sites part of that code is JavaScript, which runs in your browser and fills in the text: the services list, the project descriptions, sometimes the whole page.
Most AI crawlers don't wait for that. In December 2024 Vercel and MERJ published a study of AI crawler traffic on Vercel's network. None of the major AI crawlers they looked at ran JavaScript. That covered OpenAI's crawlers, Anthropic's ClaudeBot, PerplexityBot and those of Meta and ByteDance. They downloaded JavaScript files and never ran them. Google's Gemini was the exception, because it uses Googlebot's systems, which do run JavaScript. Apple says its crawler can too.
That study is nearly two years old. The crawler pages OpenAI, Anthropic and Perplexity publish today say nothing about running JavaScript, so the safe assumption is that their crawlers still don't. If your text only appears after JavaScript runs, those tools see a menu, a footer and very little in between.
Which sites are at risk. Sites built as apps, where the whole page is put together in the browser. Pages that pull in their text from another system after loading, like a project gallery or a jobs list. Tabs and "read more" sections that fetch their content only when clicked.
How to test it. Open a page, right-click and choose "View page source". Search the source for a sentence from the page, like the first line of your services list. If you can find it, crawlers can read it. If the source is mostly code with no sign of your words, they can't.
Words inside pictures get missed.
It's common to see a services list designed as a graphic, a phone number set into a banner image or credentials shown as a scanned certificate. To a crawler reading the page, that's an image with perhaps a short description attached.
Some AI models can read the text in an image when a person uploads one. Don't count on a crawler doing the same with yours. Put your services, service areas and phone number on the page as words, and keep images for photos.
Key facts that only live in PDFs are harder to use.
Google indexes PDFs, so they can turn up in search. They're still a weak home for facts you want AI tools to repeat. A PDF often has no clear title, can't be linked to section by section and may be a scan, which is a picture of text.
Capability statements, brochures and rate sheets are the usual examples. Keep them as downloads if clients like them, and put the essentials on a normal page as well.
Crawlers can't get past forms and logins.
Crawlers don't fill in forms or log in. A case study behind "enter your email to download" is invisible to them. So is anything in a client portal.
That's right for things that should stay private. It's a problem when the pages that prove what you do, like case studies and project lists, sit behind a form.
Thin pages give a tool little to quote.
Sometimes the words are readable and there just aren't many of them. A services page with a heading, three photos and 40 words tells a tool that you offer the service and not much more.
Our check flags key pages that show a crawler fewer than 100 words. No word count guarantees anything. But a page about a service should say what it is, where you do it, who it's for and what you've done, and that usually takes a few hundred words.
Pages crawlers can't find don't get read.
A sitemap is a list of your pages that crawlers can read. Without one, they find pages by following links, and pages with few links pointing to them get missed.
Broken links cost you too. In the same Vercel study, about a third of the visits from ChatGPT's and Claude's crawlers landed on pages that didn't exist. When you rename or remove a page, point the old address at the new one.
You can check your key pages in 30 seconds.
The view-source test above works one page at a time. Our free AI Visibility Check does it for your key pages at once. Under "Can they read your pages?" it shows how many words your home, about, services and contact pages give a crawler before any JavaScript runs, and whether your sitemap lists your pages. Under "Is your business clear?" it checks that your phone number and email are in the page text and not only in an image or a form.
If something else is in the way, like a blocked crawler, the check shows that too. See is your website blocking ChatGPT and Claude.
Most fixes are content changes.
Put the words on the page. Move key facts out of images and PDFs. Bring your case studies out from behind forms. Publish a sitemap. The AI Search Tune-Up does this as part of making sure AI tools can read every important page. It takes one to two weeks and starts at $1,500 for a site of up to 30 pages.
If your whole site is built so its text only appears after JavaScript runs, that's a bigger job. Some platforms have a pre-rendering setting that sends the finished text to crawlers, and switching it on can be enough. If yours doesn't, the check will show the problem, and we'll tell you on a call what fixing it would involve.