How computer-use agents differ: driving the screen versus driving the machine
Computer-use AI agents fall into two designs. Screen-driven agents take a screenshot, infer where to click, and repeat — a general approach that works anywhere but is slow and breaks when layouts change. Channel-driven agents read the structured interfaces underneath the screen, such as the DOM or accessibility tree, and fall back to pixels only when nothing else is available.
In short
- Screen-driven agents work on any surface, including ones that expose nothing, but pay a vision model call per step.
- Channel-driven agents are faster and more stable, because labels and roles survive a layout change that breaks coordinates.
- The two are not mutually exclusive: the robust design is a ladder that prefers structure and keeps pixels as a fallback.
- Twinly is channel-driven with a pixel fallback, running locally on macOS and Windows.
- Where an agent runs matters as much as how it clicks: on your machine, in your sessions, versus in a remote browser.
The two designs
Screen-driven. The agent receives a screenshot and outputs a coordinate to click. This is general — it works on anything that renders, including applications that expose no structured interface at all. The costs are that every step needs a vision model call, small or low-contrast controls get misread, and a layout change invalidates what the model learned a moment ago.
Channel-driven. The agent asks what the application actually exposes and uses that: a typed API, the DOM, the accessibility tree. A control resolved by label and role stays valid when the page is restyled, and needs no vision call to find. The cost is that some surfaces genuinely expose nothing, which is why a pixel fallback is still necessary.
The ladder Twinly uses
Twinly probes which channels are live on the current surface and commits to the highest-fidelity one, in this order:
- First-party API
- Declared tools
- DOM
- Accessibility tree
- Canvas model
- OCR
- Pixels
Pixels remain the seventh rung rather than the first, so a screenshot-only surface still works — it is just not the assumed case.
The axes that matter besides clicking
| Question to ask | Why it matters |
|---|---|
| Where does it run? | On your machine, it uses your existing logged-in sessions. In a remote browser, you must re-authenticate and trust a third party with those credentials. |
| What can it do without asking? | Whether irreversible actions — sending, paying, deleting — require explicit approval, and whether that gate can be turned off. |
| Is there a record? | Whether tool calls are logged in a form that would reveal after-the-fact alteration. |
| What will it refuse? | Whether it will type credentials, card numbers, or government identifiers into a field. |
| What happens when it fails? | Whether it reports honestly or claims success it cannot verify. |
Common questions
What is a computer-use AI agent?
- A computer-use AI agent is software that operates a computer's interface the way a person would — reading what is on screen and acting on it — in order to complete tasks in applications that expose no API for the job.
Why do screenshot-based agents fail on real websites?
- Because a coordinate inferred from an image is only valid for that exact rendering. When a page reflows, loads slowly, or is restyled, the inferred position goes stale, and each retry costs another vision model call.
Is driving the DOM always better than driving pixels?
- Not always, which is why a fallback is necessary. Some surfaces — text-free 3D and WebGL canvases, for instance — expose no structured channel at all. The robust design prefers structure where it exists and keeps pixels available where it does not.
Does it matter whether an agent runs locally or in the cloud?
- It changes what you have to hand over. An agent on your own machine uses sessions you are already signed into. A remote agent needs its own authenticated access to each service, which means trusting a third party with that access.