GuideAdvanced

Computer Use and Browser Agents

Many systems a business depends on have no API: a supplier's portal, an old internal system, a desktop application nobody dares to touch. For an agent to work there it has to do what a person does: look at the screen, move the mouse, type, and check that what it expected actually happened. This guide teaches you to build that agent end to end: the loop that observes, acts and verifies with screenshots; the driver that turns each model action into real clicks with Playwright or with xdotool; the coordinate scaling that prevents clicks in the wrong place; the element-reference mode that costs less than screenshots; the container with a virtual desktop where the agent cannot touch anything it shouldn't; and the defenses against on-screen text that tries to give the agent orders, with human confirmation before any irreversible action. You learn it with **Halcyon Parts**, a fictional auto parts distributor. Every morning someone copies by hand the status of open orders from the portal of its supplier Kestrel Bearings, which has no API, into Halcyon's reorder desk, and requests a reorder from another supplier for whatever is backordered. Your agent does that work in a real Chromium inside Docker: it accepts the cookie banner, pages through the list, waits for the delivery date to load, fills in the form, finds out that the save button takes a moment to turn on, asks for permission before sending a purchase order and, in the trap scenario, ignores a supplier note that asks it to open another page and order 500 units. It also operates a desktop application on a virtual X server. All the code runs on your laptop without an API key or accounts: the sites are local, the browser and the desktop live in containers without a network, and the model is a simulated pilot with fixed rules that answers with the same shape as Claude's Messages API and that makes, when you ask it to, the mistakes the providers document: assuming a step worked without looking, using stale references, obeying instructions written on the page. You see the same numbers as the guide. An optional `--live` mode uses the real API. Every piece is connected to the computer use tools of Anthropic, OpenAI and Google as they document them as of October 2026.