Computer use: a bounded control plane
Summary
Desktop control through the same bounded, verified contract as every other effect, and a clear line between a control-plane check and a task score.
Probe latency
Key results
| Measurement | Result | Source |
|---|---|---|
| Read-only capability probes passed | 30/30 | core-microbenchmarks.json |
| Probe latency, p50 | 0.032 ms | core-microbenchmarks.json |
| Probe latency, p95 | 0.052 ms | core-microbenchmarks.json |
MELRA exposes computer use through the same bounded operation contract as files, terminals and browsers. The adapters provide:
- read-only discovery of what the machine can do;
- inspection: the frontmost window, the display geometry, and the addressable elements in the accessibility tree, or, where there is no tree, words read off a screenshot with a confidence;
- screenshots, with their size and SHA-256 recorded as evidence;
- pointer movement, click and drag, by normalised or pixel coordinates, or by a role-and-name target resolved against the inspection;
- bounded text entry, an allowlist of named keys, and bounded vertical scrolling.
On macOS the adapters are typed native scripts that need the system's Screen Recording or Accessibility permission, and they refuse to type while secure input is held. Windows uses PowerShell and .NET, with UI Automation for the element tree. Linux supports screenshot tools, and xdotool on X11 when it is installed. On an unsupported platform the adapter reports the capability as absent rather than pretending it is there.
The result
Thirty read-only capability probes all passed, with a median latency of 0.032 ms, on a Mac using the native adapter. This is a control-plane microbenchmark only. It is not a task-success score and not a perception score.
What we learned
- A robust loop is observe, target, act, observe again, verify.
- Accessibility and semantic targets come first. Vision is the fallback when there is no structured evidence about the interface.
- Coordinates should not depend on screen resolution where that can be avoided, and they must name the coordinate space they are in.
- Focus, the active window, the display and secure-input state are part of the safety boundary.
- A change in pixels alone is weak evidence. A task needs its own check of the final state.
- A target that worked before can be replayed only after drift checks and independent validation.
Benchmark status
MELRA has not submitted an official OSWorld or OSWorld-MCP result. Those benchmarks measure an agent's whole decision-making, application state, GUI execution and goal completion. A public score needs a controlled virtual machine, a pinned model and policy, released traces and the official evaluator.
Since this research was first written, MELRA has added accessibility-tree inspection and semantic targeting on macOS and Windows, the screenshot-and-OCR fallback, the Windows adapter, secure-input detection on macOS, and an observation after each action that a task can verify. Still open: checking that focus has not moved between approval and action, verification across multiple displays, before-and-after screenshot evidence for changes, and an official OSWorld-MCP subset before any leaderboard claim.