Why ChatGPT Can't Read the Parts of Your Site That Sell
Your developer checked robots.txt and the Cloudflare settings and told you ChatGPT can reach the site. They are right about the door, and they answered the wrong question.
The pages ranking for "is my website blocking AI crawlers" grade AI access pass or fail on the whole site. Even the thorough one stops once page source shows body text, and files the content itself under a separate job. For a local service business the block that costs money is further down. Your site passes every whole-site check while the exact passages an AI needs before it recommends you stay invisible or unattributable: your review count, your price, the claim with your name on it. The unit of the audit is the passage, not the site.
A crawler can reach your page and still leave without your best lines
Darren Shaw, founder of Whitespark, set the floor in a Search Engine Journal live session that Loren Baker recapped on September 21, 2026:
"You cannot be surfaced in AI responses if the AI can't even access your website."
Darren Shaw, founder of Whitespark, SEJ Live, September 2026
Nobody argues with that part, and the research backs it. Cyrus Shepard's evidence review of 23 AI citation factors (Zyppy, May 2026) scored URL accessibility 9.5 out of 10, the highest score of all 23 factors.
That score works as a gate. Every dollar you put into content, reviews, and local listings only pays off inside an AI answer if the bot gets through the door first. Owners get misled on what "the door" means. Access is not one switch on the whole domain. It gets decided sentence by sentence, at the moment an AI fetches your page.
That moment matters because of how the newer AI systems work. Russ Jeffery is Director of Platform and Product Strategy at Duda. He said in the same session that "ChatGPT doesn't have their own index, they don't take the time to actually index and save pages within their infrastructure." He also said JavaScript rendering has "gone backwards in the past few years." Google still executes JavaScript and "is still doing a good job at this," in his words. The newer AI systems largely do not.
An AI system that does not run scripts and does not keep an index gets one look at your page: the raw HTML your server sends, minus everything that loads afterward. The review count inside a JavaScript carousel is not in that look. The pricing that only loads when someone clicks a tab is missing too. The "we are experts" claim that never names your business does arrive, but it is attached to nobody. Your site passes the access test while the three passages that win a local recommendation fail it quietly.
I have spent 15 years in SEO across more than 300 businesses, including plenty of local service firms, and this is the failure checklists miss most often. I rebuilt my own site's rendering to take its PageSpeed score from 59 to 97. The whole job came down to what the browser receives in the first response and what it has to wait for. I build and debug JavaScript-heavy tools in Claude Code sessions every day, including the command center I run my content operation from. I have debugged "renders for humans, blank for the bot" failures in my own code. The code keeps teaching the same thing: what a person sees on screen and what arrives in that first response are two different documents.
The standard AI access checklist grades the building, not the rooms you sell from
Every guide ranking for this question walks you through the same checks. Open yoursite.com/robots.txt, the small text file that tells bots what they may crawl, and look for "Disallow" lines naming AI bots. Log into Cloudflare or your CDN, the service in front of your site that speeds it up and filters traffic, and see whether a "block AI bots" setting is on. The thorough version adds a third step: view the page source and confirm your body text is in it.
When you read robots.txt, know which bot does which job. OpenAI, Anthropic, Perplexity, and Google each publish documentation separating the crawlers that fetch pages for live answers from the ones that gather training data. Block a training crawler and your content stays out of future model training. Block a search crawler and you vanish from the answers your customers are reading today.
| Company | Fetches pages for live answers (keep open) | Gathers training data (your call) |
|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot |
| Anthropic | Claude-SearchBot | ClaudeBot |
| Perplexity | PerplexityBot | No separate training crawler listed |
| Googlebot (Search, including AI Overviews) | Google-Extended, a robots.txt token for Gemini training, not a separate crawler | |
| Common Crawl | None | CCBot, an open web archive used in AI training |
The same published documentation lists a third kind of bot: on-demand fetchers such as OpenAI's ChatGPT-User, which grab your page live at the moment someone asks the assistant to look at it. They arrive at answer time, exactly the moment this article cares about. Keep them open too.
The Cloudflare check earns its place. Shaw said "Cloudflare often blocks AI crawlers by default," and that many owners have no idea their developer or host switched it on; he described a Shopify store blocked by Cloudflare. Jeffery called the block-all setting "a bad default by them," a publisher's policy applied to everyone, when a plumber or a dental clinic has every reason to want AI systems reading its pages. He added a warning about robots.txt itself: it is "a guidance policy" that compliant crawlers choose to respect, and "a little bit of a weak link in the chain." Anything truly private belongs behind a login or a server-level restriction, not a line in a text file.
Why do owners who pass all of it still go missing from AI answers? The checks are built to answer "can a bot get in?" A local recommendation depends on "what did the bot find once it was in?" Your Google results make the confusion worse. Google renders JavaScript. Search Console shows your service pages healthy and your review widget seen, and that green dashboard feels like proof the AI side is fine. It proves very little about a system that never runs the script.
Shaw also said, "For the vast majority of business owners, they don't need to worry about this," because sites on Duda, WordPress, or Wix already serve rendered HTML. He is right at the platform level, and that is exactly why the remaining failures hide so well. The platform sends the text. The review widget, the booking plugin, and the pricing calculator somebody bolted on afterward do not. If the whole-site checks do fail for you, start with which AI visibility layer to fix first, then come back to the passages.
Widgets and unnamed claims hide the passages AI needs
Shaw put the review widget problem in one line. Looking at a JavaScript review carousel, he said, "I can't read it because it's JavaScript." His fix needs no developer: write the sentence in plain text on the page, with how many reviews you have, your rating, and the award you won.
He followed with a line that shows how literally these systems read: "You can basically say any BS numbers you want on your webpage, and AI will cite it." Then he closed the loophole himself.
An AI answering "physio near me with good reviews" can only repeat the review count it received. If yours lives inside a widget and a competitor typed theirs into a sentence, the competitor owns that line of the answer by default, even with fewer reviews than you. Accurate numbers in plain text are the entire fix.
Shaw flagged a second source of trouble: small businesses generating React-heavy sites with AI website builders. "If you're just vibe coding a website, they're usually pretty bad, and I would just pay attention to that." Vibe coding means describing a site to an AI tool and letting it write all the code. The result looks polished in a browser and reaches a bot as a near-empty shell, because scripts assemble the whole page after it loads.
The third failure is a wording problem. An audience member, Cody Anderson, asked whether semantic triples matter, meaning sentences built as subject, verb, object, with your business as the subject. Shaw's answer was "a hard yes." His example: "We are experts at hot water tank repair" gives a crawler no business to attach the claim to, while "Johnson Plumbing Denver are experts at hot water tank repair" does. His reasoning fit in five words: "robots are kind of stupid."
Jeffery pushed back on readability, warning that "at that point you are writing for robots," and Shaw agreed the pattern cannot open every paragraph. "It's a sprinkling," he said, saved for the passages you most want tied to your business: your core service, your pricing, your differentiators, your ratings. For shaping those passages so an AI lifts them cleanly, the companion piece on structuring your content so AI quotes it covers the formatting layer. This article stays on whether the passage reaches the bot at all, and whether it arrives with your name on it.
Picture your own business. Say you run a physio clinic in Manchester with a few hundred five-star Google reviews, clear fees for an initial assessment, and a homepage line that reads "we specialize in sports injury rehab." A patient on your site sees all three. An AI fetching that page for "sports injury physio Manchester with good reviews" works with something else entirely.
| Passage | What a person sees | What a no-script AI fetch gets | The fix |
|---|---|---|---|
| Reviews | A carousel of five-star reviews and a rating badge | The empty box the widget fills later: no count, no rating | One plain sentence with the clinic name, real review count and average rating |
| Pricing | Fees under a "Prices" tab | No prices, when the tab loads its content on click | Core prices written into the page itself |
| Specialty | "We specialize in sports injury rehab" | The words arrive, attached to no business | "[Clinic name] specializes in sports injury rehab in Manchester" |
| Location | An embedded map with the address pinned | A map frame, with the address living inside someone else's page | Address, phone and service area in plain text on the page |
None of those gaps would trip a robots.txt check, a Cloudflare toggle, or a quick scroll through page source that confirms "yes, there is text here." Each one decides whether your clinic appears in the answer or the clinic down the road does.
Run the passage audit in view-source, then hand the gaps to whoever built the site
Start by deciding which passages matter, because you cannot audit a page for sentences you never listed. Shaw's own research process is the fastest way in. Ask Gemini, Claude, or Google's Ask Maps what a page for your service should contain. "They're always going to tell you pricing," he said, along with trust symbols, reviews, and case studies. Then use query fan-out, the way an AI quietly breaks one question into many smaller searches before it answers. "It takes your one prompt and turns it into 10 other prompts," Shaw said. "Those are your frequently asked questions."
That research hands you a short list: your price, your review count and rating, your core service claim, your location, and the three or four questions buyers ask before they call. Next, look at the page the way a no-script AI does. On a desktop browser, right-click your service page and choose "View Page Source," or type view-source: in front of the address. That window shows the raw HTML your server sends, before any script runs. Press Ctrl+F (Cmd+F on a Mac) and search for each item.
Mark each item present, missing, or unattached. Missing means it never appears. Unattached means the words are there but your business name is nowhere near them. Run it on your highest-earning service page first, because that is where a hidden number costs the most. Then fix in this order.
Steps 1, 2 and 5 are yours. Steps 3 and 4 are text edits you or an office manager can make in WordPress, Wix, or Duda without touching code, as long as the words live in the page editor. When the missing passage is inside a custom widget or an AI-built site, send your developer the audit list. Ask one question: "Which of these sentences appear in the raw HTML our server sends, before any JavaScript runs?" A good developer answers that in minutes with a yes or no per line. An answer about "the site being fully crawlable" means they checked the building and skipped the rooms.
Count passages found, not bots allowed
Your first number is the simplest one: how many of your listed passages appear in view-source, out of the total. Six out of six on every money page is the target. It is a number you control, it changes the day you edit the page, and it is the only metric in this article that measures the thing that actually broke.
Your second check is the answer itself. Once a month, ask ChatGPT, Perplexity, and Claude with search switched on the same questions your fan-out research produced. Read whether the answer repeats your real review count, your actual price, or your specialty with your name on it. Jeffery's point about ChatGPT keeping no index of its own cuts both ways here. If he is right, there is no reindexing queue to wait in, and a fixed passage can show up the next time a system fetches your page. Whether it fetches your page for a given question depends on signals beyond the page. That is where the local signals AI reads and the order to fix them takes over: consistent name, address, and phone details and the outside sources AI checks before it trusts you.
Answers shift from one run to the next. Judge them over several weeks, never from a single screenshot. The third data point lives in your hosting or CDN logs: visits from OAI-SearchBot, Claude-SearchBot, or PerplexityBot. Those visits confirm the door is open. What the bot carried out is a separate question.
| Looks like progress | What it proves | Measure this instead |
|---|---|---|
| robots.txt allows every AI bot | The door is open | Listed passages present in view-source, out of the total |
| AI bot visits in your logs | A crawler arrived | Whether answers repeat your real number, price or specialty |
| One ChatGPT screenshot naming you | One answer, on one day | The same questions across three AI tools, monthly |
| Review widget installed | A person can see your reviews | Review count and rating in a plain sentence in the page source |
| Search Console says the page is indexed | Google, which runs JavaScript, can read it | What the raw HTML holds for systems that do not run scripts |
Every row on the left feels like a finished job. Agencies love reporting them. The Search Console row is the one that fools the most careful owners, because Google is the one system Jeffery credits with still rendering JavaScript well. A clean Google report tells you the least about the AI tools that skip that step.
Frequently Asked Questions
How do I know if my website is blocking AI crawlers?
Open yoursite.com/robots.txt and look for Disallow lines naming OAI-SearchBot, Claude-SearchBot, or PerplexityBot. Ask whoever manages Cloudflare or your hosting whether an AI bot blocking setting is switched on, since Darren Shaw of Whitespark says Cloudflare often blocks AI crawlers by default. Then open your page source and search for your review count, your prices, and your business name. A site can let every bot in and still hide the sentences that win a recommendation.
Why does Google show my reviews but ChatGPT doesn't mention them?
Google runs JavaScript when it reads your page. Its crawler still sees a review widget that loads by script. Russ Jeffery of Duda said JavaScript rendering has gone backwards in the past few years, and that ChatGPT does not keep its own index of saved pages. ChatGPT works from the raw page your server sends, and a review count that only exists inside a widget is not in it. Add one plain sentence with your business name, review count, and average rating, and keep the numbers accurate.
Do I need to rebuild my website so AI can read it?
For most owners, no. Darren Shaw said sites on Duda, WordPress, or Wix already serve readable HTML. The gap is not in the platform. The fix is a handful of sentences: moving your review count, prices, and main service claim out of widgets and tabs into plain text with your business name attached. A rebuild is worth discussing only if your site was generated with an AI builder and view-source shows almost no readable text, which is the vibe-coded case Shaw warned about.
A clean robots.txt and a switched-off Cloudflare block tell you the door is open. They cannot tell you whether the sentence that should win you the next customer, the one with your name, your rating, and your price in it, is standing inside when the bot walks through.
If you want a second pair of eyes on what your service pages are hiding, book an AI visibility call and we will go through them in view-source together. Before then, open your best page and ask one question: if an AI could only repeat what your server sends, what would it say about you?
