How AI agents interact
with websites

Copy linkStart with what the agent needs to do

Imagine asking an AI agent to find a museum's step-free entrance and check the available times for a visit. The entrance information might be in a paragraph on the access page, while the visiting times sit behind a date picker. Reading the paragraph and using that calendar need different capabilities, even though both are part of the same request.

The agent works with the tools its application gives it, which might return a screenshot, page text or a description of the controls. Browser-based agents can combine these views as they work, while a tool connected to an API may retrieve the visiting times directly.

<h1>Plan your visit</h1>
<button
  aria-label="Next month">
  <svg>...</svg>
</button>

heading "Plan your visit"
  level: 1

button "Next month"

Copy linkA screenshot shows the rendered page

With a screenshot and a vision-capable model, an agent can inspect the page's visible layout. It can look for an icon, read a label or locate a control by its position, then use a browser tool to click at the corresponding coordinates. This can help with visual interfaces such as maps or drawing canvases, where the relevant area may have little useful structure in an accessibility snapshot.

Playwright MCP's vision mode is a documented example of this approach. It adds coordinate-based interaction that works with screenshots alongside accessibility snapshots, so the client can use the representation that helps with the current step.

A screenshot records a particular view at a particular moment. If a menu opens or the page moves after a click, the agent needs an updated view to work from. The same applies to information below the viewport, which won't appear in a screenshot of the area currently on screen.

Copy linkThe DOM exposes the document structure

The Document Object Model, usually shortened to DOM, is the browser's in-memory representation of a document. It organises elements and text into a tree that software can inspect and change, as MDN's introduction to the DOM explains. A browser tool can use that structure to find a heading, follow a link or inspect a form field without identifying its position in an image.

The agent may receive selected elements or extracted text rather than the whole tree. The tool's choice of what to include matters, as does the page state when the inspection happens. JavaScript can add or change elements after the original HTML arrives, so inspecting the current DOM can reveal content that wasn't present in the server's first response.

For the museum example, the entrance description needs to stay associated with the right building and address. Clear headings and links help preserve that relationship when a tool extracts the text from its visual layout.

Why AI crawlers may miss JavaScript-rendered content

Copy linkThe accessibility tree describes meaning and controls

Browsers also expose an accessibility tree, which represents information such as an element's role, accessible name and state. A control can be described as a button named 'Next month', for example, while a heading can carry its text and level. Chrome's accessibility tree explanation shows how Chromium derives this representation from the DOM and its semantic information, including ARIA attributes.

This gives software a way to distinguish a control's purpose from its appearance. Playwright MCP's accessibility snapshots use roles, names and references that a client can use to target an element in a later action. A screenshot can still help when a control's purpose is only apparent visually.

If a calendar arrow has no accessible name, its direction may be clear to someone looking at the page while its purpose remains unclear in the snapshot. Using the appropriate HTML control and giving it a meaningful name helps explain the action. An accessibility tree isn't a separate page to maintain; its usefulness depends on the semantics of the page you've built.

Playwright MCP: accessibility snapshots and element references

Copy linkAn agent can also work without a browser

An agent looking up the entrance description may only need an HTTP request and readable text. Cloudflare's web fetch tool, for example, can retrieve a page and turn its HTML into Markdown. Its documentation directs agents towards a browser when the work requires JavaScript, a browser session or interaction with the page.

Where a service provides an API, an agent with a suitable tool can request data or perform a supported action without navigating the website. MCP can make such capabilities available through a standard protocol, allowing a client to discover a server's tools and call them with structured arguments. A museum could, for instance, expose a tool for checking tour availability instead of requiring every client to operate its calendar.

That route depends on what the service actually exposes and which permissions the client has. A public information tool and an account tool that can manage bookings have different access requirements, and connecting to an MCP server doesn't remove those requirements.

MCP: server tools, resources and user control

Copy linkEach action needs a fresh look at the result

In the museum scenario, the agent might open the calendar, choose a date and then read the available times. After choosing a date, it should check that the calendar shows the intended day and that the available times belong to that selection.

Playwright's vision documentation illustrates this pattern by taking another screenshot after an interaction to inspect the result. A client working with structured snapshots can likewise inspect the updated controls, while an API tool may return the outcome in its response. The important part is checking the resulting state before using it as the basis for the next step or an answer.

For a website team, this makes feedback part of the interface. A selected date, an unavailable slot or a failed request needs a clear indication that the tool can pick up. If the page continues to display old times while new ones are loading, a client needs a way to tell which result belongs to its selection.

Copy linkCheck your website with a specific task in mind

Choose a task that matters to your visitors and write down what a correct result would require. For the museum, that could mean identifying the step-free entrance for the right building and reporting times for the selected day. Then inspect the information and controls through the tools of the particular agent you're assessing.

Keep the page URL, date and client or tool version with the evidence, along with the session conditions and actions taken. Record signed-in and public journeys separately, and repeat the check with any other clients you want to support so the team can reproduce each finding.

Information

Can the client retrieve what the task needs, with its conditions and context intact?

Controls

Can it identify the relevant controls and understand their current state?

Feedback

Can it distinguish a new result from loading, an error or an earlier result?

Evidence

Can someone repeat the steps and check the answer against the recorded evidence?

Record findings and plan a retest with the website content audit guide