# How AI agents interact with websites AI agents can read pages, use browser controls or call tools directly, with each route shaping what they can see and do. [Doğu Abaris](https://tityra.com/authors/dogu-abaris) · Published 2026-11-11 ## Start with what the agent needs to do Imagine asking an AI agent to find a museum's step-free entrance and check the available times for a visit. The entrance information might be in a paragraph on the access page, while the visiting times sit behind a date picker. Reading the paragraph and using that calendar need different capabilities, even though both are part of the same request. The agent works with the tools its application gives it, which might return a screenshot, page text or a description of the controls. Browser-based agents can combine these views as they work, while a tool connected to an API may retrieve the visiting times directly. ### The same page, different views An illustrated museum page, with simplified DOM and accessibility snippets. #### Screenshot Illustrated page heading: Plan your visit. Calendar control: Next month. The page layout and calendar arrow. #### DOM ```html

Plan your visit

``` HTML elements and attributes that a browser tool can inspect. #### Accessibility tree ```text heading "Plan your visit" level: 1 button "Next month" ``` The heading and a named button, expressed through their roles. ## A screenshot shows the rendered page With a screenshot and a vision-capable model, an agent can inspect the page's visible layout. It can look for an icon, read a label or locate a control by its position, then use a browser tool to click at the corresponding coordinates. This can help with visual interfaces such as maps or drawing canvases, where the relevant area may have little useful structure in an accessibility snapshot. [Playwright MCP's vision mode](https://playwright.dev/mcp/vision-mode) is a documented example of this approach. It adds coordinate-based interaction that works with screenshots alongside accessibility snapshots, so the client can use the representation that helps with the current step. A screenshot records a particular view at a particular moment. If a menu opens or the page moves after a click, the agent needs an updated view to work from. The same applies to information below the viewport, which won't appear in a screenshot of the area currently on screen. ## The DOM exposes the document structure The Document Object Model, usually shortened to DOM, is the browser's in-memory representation of a document. It organises elements and text into a tree that software can inspect and change, as [MDN's introduction to the DOM](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model) explains. A browser tool can use that structure to find a heading, follow a link or inspect a form field without identifying its position in an image. The agent may receive selected elements or extracted text rather than the whole tree. The tool's choice of what to include matters, as does the page state when the inspection happens. JavaScript can add or change elements after the original HTML arrives, so inspecting the current DOM can reveal content that wasn't present in the server's first response. For the museum example, the entrance description needs to stay associated with the right building and address. Clear headings and links help preserve that relationship when a tool extracts the text from its visual layout. [Why AI crawlers may miss JavaScript-rendered content](/blog/why-ai-crawlers-may-miss-javascript-rendered-content) ## The accessibility tree describes meaning and controls Browsers also expose an accessibility tree, which represents information such as an element's role, accessible name and state. A control can be described as a button named 'Next month', for example, while a heading can carry its text and level. [Chrome's accessibility tree explanation](https://developer.chrome.com/blog/full-accessibility-tree) shows how Chromium derives this representation from the DOM and its semantic information, including ARIA attributes. This gives software a way to distinguish a control's purpose from its appearance. Playwright MCP's accessibility snapshots use roles, names and references that a client can use to target an element in a later action. A screenshot can still help when a control's purpose is only apparent visually. If a calendar arrow has no accessible name, its direction may be clear to someone looking at the page while its purpose remains unclear in the snapshot. Using the appropriate HTML control and giving it a meaningful name helps explain the action. An accessibility tree isn't a separate page to maintain; its usefulness depends on the semantics of the page you've built. [Playwright MCP: accessibility snapshots and element references](https://playwright.dev/mcp/snapshots) ## An agent can also work without a browser An agent looking up the entrance description may only need an HTTP request and readable text. [Cloudflare's web fetch tool](https://developers.cloudflare.com/agents/tools/web-fetch/), for example, can retrieve a page and turn its HTML into Markdown. Its documentation directs agents towards a browser when the work requires JavaScript, a browser session or interaction with the page. Where a service provides an API, an agent with a suitable tool can request data or perform a supported action without navigating the website. MCP can make such capabilities available through a standard protocol, allowing a client to discover a server's tools and call them with structured arguments. A museum could, for instance, expose a tool for checking tour availability instead of requiring every client to operate its calendar. That route depends on what the service actually exposes and which permissions the client has. A public information tool and an account tool that can manage bookings have different access requirements, and connecting to an MCP server doesn't remove those requirements. [MCP: server tools, resources and user control](https://modelcontextprotocol.io/docs/2026-07-28/learn/server-concepts) ## Each action needs a fresh look at the result In the museum scenario, the agent might open the calendar, choose a date and then read the available times. After choosing a date, it should check that the calendar shows the intended day and that the available times belong to that selection. Playwright's vision documentation illustrates this pattern by taking another screenshot after an interaction to inspect the result. A client working with structured snapshots can likewise inspect the updated controls, while an API tool may return the outcome in its response. The important part is checking the resulting state before using it as the basis for the next step or an answer. For a website team, this makes feedback part of the interface. A selected date, an unavailable slot or a failed request needs a clear indication that the tool can pick up. If the page continues to display old times while new ones are loading, a client needs a way to tell which result belongs to its selection. ### Follow the page as it changes - **Read the current view:** Find the calendar and identify the date controls. - **Take the next action:** Choose the day using the available browser tool. - **Check the new state:** Read the times for that day and check for a loading or error message. Use the updated state to decide whether another action is needed. ## Check your website with a specific task in mind Choose a task that matters to your visitors and write down what a correct result would require. For the museum, that could mean identifying the step-free entrance for the right building and reporting times for the selected day. Then inspect the information and controls through the tools of the particular agent you're assessing. Keep the page URL, date and client or tool version with the evidence, along with the session conditions and actions taken. Record signed-in and public journeys separately, and repeat the check with any other clients you want to support so the team can reproduce each finding. - **Information:** Can the client retrieve what the task needs, with its conditions and context intact? - **Controls:** Can it identify the relevant controls and understand their current state? - **Feedback:** Can it distinguish a new result from loading, an error or an earlier result? - **Evidence:** Can someone repeat the steps and check the answer against the recorded evidence? [Record findings and plan a retest with the website content audit guide](/blog/how-to-audit-website-content-for-ai-agents) ## Sources and further reading - [Playwright MCP: vision mode](https://playwright.dev/mcp/vision-mode) - [MDN: the Document Object Model](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model) - [Chrome: the accessibility tree](https://developer.chrome.com/blog/full-accessibility-tree) - [Playwright MCP: accessibility snapshots](https://playwright.dev/mcp/snapshots) - [Cloudflare: web fetch](https://developers.cloudflare.com/agents/tools/web-fetch/) - [MCP: server concepts and tools](https://modelcontextprotocol.io/docs/2026-07-28/learn/server-concepts) ## About the author ### [Doğu Abaris](https://tityra.com/authors/dogu-abaris) Doğu Abaris is a software and data engineer and a co-founder of Tityra. For more than a decade, he has built maintainable software for production, working across front-end and back-end development. He also contributes to open-source software and web standards. ## Try a focused check on your own website Use the audit guide to record what a client receives and give your team a repeatable way to check a change. [Read the audit guide](/blog/how-to-audit-website-content-for-ai-agents) [Tityra audit methodology](https://tityra.com/audit-methodology) · [Blog](https://tityra.com/blog)