Table of Contents
TL;DR
- A Vercel and MERJ analysis of more than 500 million GPTBot and 370 million ClaudeBot requests in 2024 found that none of the major AI training or retrieval crawlers execute JavaScript. GPTBot fetched JavaScript files in about 11.5 percent of requests and ClaudeBot in about 23.84 percent, but neither ran them.
- The AI client population sorts into three rendering tiers. Most of the agents that visit your site fall into Tier A, which means they see only the HTML your server returns in the very first response.
- A small set of user-triggered fetchers (Google-Agent, ChatGPT-User, Claude for Chrome, Perplexity-User) does render JavaScript because it runs on browser infrastructure when a specific human directs it. Everything else does not.
- The three patterns that break agent readability most often are single-page-app architecture, paywall or cookie walls that block content until interaction, and product details hidden behind a click.
- The four-week pass takes one step per week — confirm what the initial HTML carries, eliminate JavaScript-only paths to key content, simplify navigation, and verify with a no-JavaScript request.
An AI agent requests the homepage at three in the morning.
The server returns two hundred lines of HTML in the first response. None of them contain the business name, the phone number, the price, or the services. The page exists for the human visitor whose browser will later run twelve seconds of JavaScript. For the agent reading the first response, the page does not exist.
That is the gap most small site owners never see, because they never visit their own site without a browser.
OpenAI crawlers do not execute JavaScript, so a site that renders cleanly to humans can be unreadable to the agents now acting on users’ behalf. The Vercel and MERJ analysis of more than 500 million GPTBot and 370 million ClaudeBot requests confirmed this across the largest empirical dataset published so far. The fix on most small sites is not a rebuild. The fix is making sure the content a human reads is in the first HTTP response, before any JavaScript runs.
What does ‘agent-ready’ actually mean for a small website?
Agent-ready means an AI agent acting on a user’s behalf can find your page, read its main content from the initial HTML response, and complete the task the user gave it.
The bar is lower than search-ranked. The bar is higher than visible-to-humans.
A page that renders cleanly to a human visitor through a client-side framework — but only after JavaScript boots — reads as empty to the agent population that visits it today.
A buyer’s AI agent has one job per visit — pick up the answer, or move on. The agent does not wait for an animation, does not click a "load more" button, and does not solve a cookie banner.
The HTML you return in the very first response is the page that exists, as far as the agent is concerned.
That sentence is the most expensive sentence in 2026 web development, and it is also the cheapest one to act on once you understand it.
Why does JavaScript-only rendering break most of the agent population?
Training and retrieval crawlers do not execute JavaScript.
They request your page, receive whatever HTML the server returns in that first response, and walk away. If your headline, your answer paragraphs, your links, and your prices only appear after a client-side framework boots, the agent reads an empty shell and moves to the next candidate.
The empirical case for this is overwhelming.
A 2024 joint analysis from Vercel and MERJ tracked more than 500 million GPTBot requests and 370 million ClaudeBot requests across a large network and two reference sites. None of the major AI crawlers executed JavaScript. GPTBot fetched JavaScript files in about 11.5 percent of fetches and ClaudeBot in about 23.84 percent of fetches, but the bots collected those files as text artifacts and never ran them. Google Search Advocate John Mueller said the same thing in plainer words later that year — the LLMs that read the web for training and retrieval have no trouble with HTML, and the practical problem is heavy JavaScript, not the document type.
For a small site, the implication is calm. Server-side rendering, static HTML generation, or hybrid frameworks that ship populated HTML in the initial response all satisfy the agent.
Pure client-side rendering does not. The choice is not about taste. The choice is about whether the page exists for the agent at all.
Which AI clients still execute JavaScript — and which do not?
A small set of user-triggered fetchers does execute JavaScript — Google-Agent, ChatGPT-User, Claude for Chrome, and Perplexity-User.
These clients run on top of browser infrastructure when a specific human user directs them to a specific page. They read the page the way a Chrome tab does — styles, scripts, async fetches, the whole stack. Google-Agent in particular carries a Chrome user agent string and is documented by Google as Chrome-based.
Everything else in the agent population today sees only the initial HTML response.
That includes the training crawlers that build the foundation knowledge each engine queries. It also includes the retrieval crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — that fetch fresh pages when an engine answers a live query. Googlebot itself does have JavaScript rendering through the Web Rendering Service, and Gemini inherits that capability. AppleBot also renders.
The honest summary is that most of the agent population reads HTML and only HTML.
The user-triggered fetchers are the exception, not the rule. A site that wants to be cited by the retrieval crawlers — the ones that decide whether your page makes the short list at answer time — needs to deliver its answer in the first response.
How do you check whether an AI agent actually sees your page?
The fastest check is curl with no JavaScript.
Request your top page over plain HTTP, save the response, and read it. If your H1, your first paragraph, your prices, your structured data, and your visible navigation are all in that file, the agent sees them. If the file is an empty shell pointing at a JavaScript bundle, the agent does not.
The check takes two minutes per page.
For a small business with five or ten important pages, the whole audit fits in a coffee break. The pattern to look for is what is missing — not what is present. A response that contains the page chrome and the meta tags but not the body content is the most common shape of a failed page, because most modern themes ship that response shape by default.
A second check confirms the first. Open the page in a browser with JavaScript disabled (Chrome’s developer settings expose this) and reload.
The page that loads in that mode is the page the agent reads. If it looks like the real page, you are agent-ready. If it looks like a blank screen with a spinning loader, the agent saw the spinning loader.
These two checks together are the entire diagnostic. Everything else is interpretation.
What are the three patterns that break agent readability most often?
The first is single-page-app architecture with a client-rendered shell.
A React or Vue or Angular site that renders entirely in the browser ships an empty document shell on the first response and then populates it through JavaScript. To a human, the page loads in a second. To an agent that does not execute JavaScript, the page is the empty shell. The same framework can ship a populated server-rendered response (Next.js, Nuxt, SvelteKit, Angular Universal all support this), and the fix is enabling that mode on the routes that matter for AI visibility.
The second is paywall or cookie-consent walls that block content until a user interacts.
A wall that hides the body of the page until the visitor clicks "accept" reads to an agent as a wall with no body behind it. The fix is making sure the content is in the HTML response regardless of consent state, and that the consent banner is an overlay rather than a content gate.
The third is heavy navigation, lazy-loaded sections, or modal-only product details that hide critical content behind a click.
A page that contains its core answer only after a buyer clicks "see details" or "expand" is a page whose core answer the agent does not see. The fix is making sure the steady drip of fresh content the AI engines reward lives in the initial response, and that any "more details" links open new pages rather than reveal content inside the current one.
A small site that fixes these three patterns is agent-ready in roughly the same week. None of the fixes change what the human visitor sees. All of them change what the agent reads.
Other questions worth answering
Does adding an llms.txt file at your root URL meaningfully change how retrieval crawlers find new content?
Not meaningfully yet. The llms.txt convention emerged from a 2024 community proposal — a single root-URL document listing URLs an AI client should prioritize. Major retrieval crawlers do not parse it as a directive, and OpenAI and Anthropic have published nothing committing to honor it. Current audit guidance treats llms.txt as low-confidence, so put effort into HTML access first.
How do you confirm a visitor calling itself GPTBot is genuine rather than a copycat header?
OpenAI publishes verified IP ranges for GPTBot. A reverse DNS lookup against those ranges confirms the bot. Anthropic and Perplexity publish equivalent verification information for ClaudeBot and PerplexityBot. The 2024 Vercel and MERJ analysis tallied more than 500 million GPTBot requests using this style of verification step.
Has sitemap.xml become a stronger signal for retrieval crawlers in 2026?
Yes, on early evidence. GPTBot and ClaudeBot both started requesting sitemap.xml files in March 2026, the first such activity tracked in either crawler’s history. Earlier server logs showed neither crawler touching the sitemap. Keep the sitemap current as low-cost insurance, but treat the signal as emerging rather than settled.
What role does schema markup serve once your HTML response delivers the answer text directly?
Two roles, both confirming. Schema markup is the parallel JSON-LD channel that restates your answer as structured data for retrieval crawlers. It carries entity facts the prose may leave implicit, such as business hours or canonical brand spellings. Vercel and MERJ’s 2024 measurement showed GPTBot reading HTML cleanly, which means schema works as confirmation rather than as the sole source.
How should you fix agent readiness on a small site without a rebuild?
Treat the next month as a four-step pass, one step a week.
Week one is the inspection pass. Run curl against your top five pages and read the output. Mark the pages whose body content is in the response. Mark the pages whose body content is missing.
Week two is the rendering pass.
For each page in the missing set, decide whether the fix is enabling server-side rendering on that route, switching the route to a static export, or moving the critical content out of a client-only component. Most modern frameworks can flip this per route without a full rebuild.
Week three is the wall pass. Audit the cookie banner, the paywall (if any), and the consent flow. Confirm that the body of the page is in the initial HTML response even when the wall has not been dismissed. Move the wall into an overlay layer if it is currently a content gate.
Week four is the navigation pass. Audit the lazy-loaded sections, the modal-only product details, and the "click to expand" sections that hide critical content from the initial render. Move the critical content into the initial page response. Leave the optional content where it is.
By the end of the month, the agent reads the same page a human reads. Whether the agent then recommends your business when buyers ask about your category becomes an editorial question — about answer-block quality, schema completeness, and source-attributed claims — rather than a structural one. You can check what your AI citations currently look like once a month from there.
If you have read this far and still are not sure whether your own site is agent-ready, that is normal. Most small site owners are not. You can contact me here, and we can run curl together against three of your pages and read the responses. There is no charge for this kind of call, and there is no pitch at the end.
I will tell you which pages are agent-ready, which are not, and what one or two fixes on your stack will move the most pages into the ready column. That is usually enough to turn a vague worry into a one-page plan.