# Why AI crawlers may miss JavaScript-rendered content

Your browser can show a complete page while another client receives only a loading message. Here's how to check where the content appears.

[Doğu Abaris](https://tityra.com/authors/dogu-abaris) · Published 2026-11-10

## A page can load without delivering its content

When you open a service page in your browser, the description may explain exactly what's included and how to get in touch. An automated client requesting the same URL might receive just the heading and a message saying 'Loading service details...'. Although the request succeeded, the response doesn't contain enough information to explain the service.

When JavaScript supplies the missing details, what an AI system receives depends on how its retrieval tool loads the page. Comparing the original HTML response with the page after its scripts have run helps you find that dependency, and the downloadable example below gives you a small, repeatable way to try it.

## What the retrieval tool can actually do

An HTTP fetch reads the response sent by a server, whereas a browser can also run JavaScript and request the data needed to build the page. If the description only appears during those later steps, a tool reading the original response will miss it. [Cloudflare's web fetch documentation](https://developers.cloudflare.com/agents/tools/web-fetch/) gives a useful example of this distinction by directing agents to a browser tool when content requires JavaScript or interaction.

The name of a crawler won't necessarily tell you which of those steps it performs. OpenAI documents OAI-SearchBot for search, GPTBot for model training and ChatGPT-User for user-triggered visits, but its crawler overview doesn't specify their JavaScript capabilities. Even a user-agent string that resembles a browser's is insufficient evidence that scripts will run, so check the documented capabilities and the result of the tool you're interested in.

[OpenAI: the roles of its crawlers and user-triggered visits](https://developers.openai.com/api/docs/bots)

## Running JavaScript is only part of loading a page

Google Search can render JavaScript, and [Google's JavaScript crawling guidance](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) explains how it processes those pages and why server-rendered or pre-rendered content can help other clients. That capability still depends on the page's resources loading successfully: a script might fail, or the request supplying its data might return an error, leaving the browser at a loading message even when the HTML request returned HTTP 200.

Some content also depends on what happens after the page opens. A specification may appear only when someone selects a product option, or another part of an article may load after a click on 'Load more'. When checking that content, record the interaction needed to reveal it so you can distinguish what the client received on arrival from what appeared during use.

## Look at the response before blaming the framework

A site can use React and still send its main content as readable HTML. [React's hydration documentation](https://react.dev/reference/react-dom/client/hydrateRoot) describes how JavaScript attaches to HTML that the server has already generated, adding interactivity to content that's present before the scripts load. Finding React or a large JavaScript bundle on a site therefore tells you little about whether its description is available in the response.

The useful question is whether the particular response contains the information you need. Server rendering can return a loading fallback for some content too, so inspect the words under the relevant heading rather than relying on the framework or rendering method as proof.

[React: server rendering and loading fallbacks](https://react.dev/reference/react-dom/server/renderToString)

## The same service, delivered in two different ways

To make the difference visible, a local comparison on 10 November 2026 checked two pages describing a fictional website care plan. The before page adds its details with JavaScript, while the after page includes the same wording directly in its HTML. Both HTML requests returned HTTP 200, although the before response stopped at the heading and loading message and the after response already contained the description, scope exclusions and consultation next step. In the browser, both pages displayed those details once the before page's script had run.

A further check kept the before document unchanged and made its script request return HTTP 503. This time, the browser stayed at 'Loading service details...', showing how a successful HTML request can leave essential content unavailable when another resource fails.

### Downloadable HTML delivery example

The html-delivery-v1 files let you inspect the same pages and compare what arrives in the response with what appears in the browser.

- Test setup: two HTML pages and a shared script served locally over HTTP, without accounts, cookies, a consent flow or external data requests.
- Inspection: an ordinary HTTP client read each response, followed by a JavaScript-enabled browser opening the pages without clicks or scrolling.
- Failure check: a temporary local server returned 503 for the before page script to reproduce a failed resource load; that server is not included in the downloads.

[Before: details added by JavaScript](/examples/html-delivery/before.html) · [After: details in the HTML](/examples/html-delivery/after.html) · [Service-content script](/examples/html-delivery/service-content.js) · [Node verifier](/examples/html-delivery/verify.mjs)

| Check | Before | After |
| --- | --- | --- |
| Service description in the initial main content | Absent | Present |
| Scope and exclusions in the initial main content | Absent | Present |
| Next step in the initial main content | Absent | Present |
| Service details in the browser with normal loading | Present after the script runs | Present without a script |

#### Repeat the check

Save all four files in one folder and run the Node verifier to check the HTML and script insertion using its minimal document stub. For the browser comparison, serve that folder with Python 3 and open http://127.0.0.1:8000/before.html and /after.html. In the Network panel, inspect each HTML response alongside the displayed page, then block service-content.js and reload before.html to see what remains when its script cannot load.

```sh
node verify.mjs
python3 -m http.server 8000 --bind 127.0.0.1
```

**Recommended change:** The after page makes the service details available in the HTML response by including them in the document, so reading that information no longer depends on the script running.

**Limits:** This comparison covers the HTTP and browser behaviour of fictional pages marked noindex, so you'd need a separate test to find out how a particular AI crawler retrieves or uses their content.

## Follow one important fact through your own page

Choose a public page you control and a detail that someone needs from it, such as a service exclusion or a product specification. Open it in a fresh session so saved preferences and previously loaded data don't hide a dependency, then look for that detail in the HTML response and in the rendered page before interacting with it.

Pay attention to where the words appear. Text stored inside a script payload may still need JavaScript to turn it into a readable paragraph or table, so finding the phrase somewhere in the source doesn't establish that it's part of the delivered main content. If it appears only after a script runs or an action is taken, follow that step to the request that supplies it.

Keep the URL, time and client with the observation, together with the response status and any session conditions or actions needed to reproduce it. Those details help the next person investigate the same result and give you a useful starting point for a retest.

- Present only after scripts run: identify the component or data request supplying the missing content.
- Absent from both views: check failed requests, access restrictions and whether you opened the expected page state.
- Visible only after interaction: record the action needed and check whether the client performs it.
- Present in both views: investigate the actual retrieval result and how the information is interpreted before changing the rendering.

## Keep essential information available from the start

If important public copy arrives only after client-side JavaScript runs, consider generating it on the server or at build time. The page can retain its filters, forms and other interactions while delivering the essential information under clear headings, with links and table relationships that preserve its meaning.

For content loaded through interaction, a directly addressable URL can provide another way to reach it. [Google's lazy-loading guidance](https://developers.google.com/search/docs/crawling-indexing/javascript/lazy-loading) recommends persistent, linked URLs for paginated content because Google Search doesn't scroll or click to reveal it. Treat that as guidance for Google Search and check the behaviour of any AI retrieval tool you also want to support.

Once a change is ready, repeat the original checks against the same detail to confirm that it now appears in the delivered main content and still displays correctly after the scripts run. If you also try an AI retrieval tool, save its name, date and output with the evidence so the team can see what changed and whether the fix worked.

[Use the website content audit guide to record evidence and plan a retest](/blog/how-to-audit-website-content-for-ai-agents)

## Sources and further reading

- [Google: JavaScript rendering and crawling](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)

- [Cloudflare: web fetch and when to use a browser tool](https://developers.cloudflare.com/agents/tools/web-fetch/)

- [OpenAI: overview of crawlers](https://developers.openai.com/api/docs/bots)

- [React: hydrating server-rendered HTML](https://react.dev/reference/react-dom/client/hydrateRoot)

- [React: server rendering and loading fallbacks](https://react.dev/reference/react-dom/server/renderToString)

- [Google: lazy-loaded content and paginated loading](https://developers.google.com/search/docs/crawling-indexing/javascript/lazy-loading)

## About the author

### [Doğu Abaris](https://tityra.com/authors/dogu-abaris)

Doğu Abaris is a software and data engineer and a co-founder of Tityra. For more than a decade, he has built maintainable software for production, working across front-end and back-end development. He also contributes to open-source software and web standards.

[Tityra audit methodology](https://tityra.com/audit-methodology) · [Blog](https://tityra.com/blog)