Vercel Labs has released agent-desktop, an open-source CLI for AI agent computer use on macOS, Windows and Linux. agent-desktop reads each app through the operating system’s accessibility tree, not through screenshots. An agent gets a JSON snapshot with a ref for every element it can act on. Version 1.0.2 shipped on October 30, 2026, under Apache-2.0. For developers, any shell-capable coding agent can now drive native apps without a vision model. TL;DR

Library: Rust CLI and a C library. 60 command names, 56 operational. Runs on: macOS 13+, Windows 10 1809+ or Server 2019+, Ubuntu 24.04+, on x64 and ARM64. Plain CPU, no GPU. Performance: Deterministic, retry-safe ref actions. No independent computer-use benchmark yet. Best: 78% to 96% fewer tokens on dense Electron apps via skeleton traversal (self-reported).

What is agent-desktop? agent-desktop is a Rust CLI and C library from Vercel Labs for reading and operating desktop apps. It is a tool that agents call, not an agent. It embeds no LLM and has no GUI. Every command prints one JSON envelope and exits. The observe, act and verify loop stays in your agent. How does agent-desktop work? Each command flows through a fixed chain. The CLI or C library calls agent-desktop-core, which reaches the OS only through a PlatformAdapter trait. One platform crate then talks to the native accessibility API:

macOS: AX API Windows: UI Automation Linux: AT-SPI

Core never imports a platform crate. Each binary is compiled to drive exactly one OS. Snapshots and refs A snapshot assigns qualified refs such as @s8f3k2p9:e8. The snapshot ID inside the ref makes it self-describing. A bare @e8 is rejected with INVALID_ARGS. When the UI changes, an action returns STALE_REF or AMBIGUOUS_TARGET. The agent then takes a fresh snapshot. Every error carries a disposition. Agents should retry only when it says safe. Ref resolution is strict and fails closed rather than guessing. Skeleton and drill-down Dense apps can expose thousands of accessibility nodes. A --skeleton snapshot reads only 3 levels deep. Cut containers report a children_count and get a drill ref. snapshot --root then reads one region into the same snapshot. The builder reported a 78% to 96% token cut versus full-tree dumps in Slack, VS Code and Notion. In Apple Maps, he reported 3,811 tokens per look falling to 273. A simulated 52-step task dropped from 198k to 31k tokens. These are self-reported figures, not an independent benchmark. Headless by default Ref actions go through the accessibility API without moving the pointer or taking focus. That is what lets agents work while you keep using the machine. The global --headed flag allows physical input when needed. Every ref action also auto-waits up to 5,000 ms for a visible, enabled, stable target. Trace a command Step

Map, drill, act, verify. Click a step. Example output is trimmed from the official docs (GNOME Calculator).

1 Skeleton 2 Drill 3 Act 4 Verify

Next step →

Builder-reported numbers for skeleton drilling, next to Anthropic's documented screenshot cost.

Apple Maps, tokens per look (self-reported) Full window snapshot 3,811 3,811 Skeleton, then drill 273 Simulated 52-step task, total tokens (self-reported) Full window every step 198k 198k Skeleton, then drill 31k 31k Reference: pixel-based computer use One Claude computer-use screenshot (Anthropic docs) ~1,000 to 1,800 ~1.8k max

↻ Replay Builder also reports 78% to 96% savings on Slack, VS Code and Notion. No independent benchmark yet.

Same 60 command names everywhere. Click a platform for requirements.

macOS AX API macOS 13+, Intel or Apple Silicon. Grant Accessibility permission with permissions --guide . Shell surfaces still planned. Windows UI Automation Windows 10 1809+ or Server 2019+, x64 or ARM64. No grant for apps at your terminal's integrity level. Higher-integrity apps refuse input. Linux AT-SPI Ubuntu 24.04+, glibc 2.35 floor, x86_64 or aarch64. Needs at-spi2-core . Mouse input and cursor overlay need X11.

one install for all 3

npm install -g agent-desktop agent-desktop skills get desktop

Where it falls short. Click to expand.

! No tree, no vision It reads accessibility trees and never matches pixels. A UI that publishes no tree is invisible to it. ! Parallel, with a lease Multi-agent cursors are presentation-only. A machine-wide interaction lease still serializes physical input. Headless semantic actions avoid the pointer entirely. ! Slow cold Electron apps A cold Chromium or Electron app can take 10 to 25 seconds to expose its web tree. ! Private data in outputs Snapshots, clipboard reads and screenshots can contain private desktop content. Keep them out of public logs.

Sources: github.com/vercel-labs/agent-desktop · agent-browser.dev/desktop · Anthropic docs © Marktechpost

(function(){ var R=document.getElementById('mtp-ad'); function rs(){try{parent.postMessage({mtpH:R.offsetHeight+40},'')}catch(e){}} var tabs=R.querySelectorAll('.tab'),panes=R.querySelectorAll('.pane'); tabs.forEach(function(t){t.onclick=function(){tabs.forEach(function(x){x.classList.remove('on')});panes.forEach(function(x){x.classList.remove('on')});t.classList.add('on');var p=R.querySelector('[data-p="'+t.dataset.t+'"]');p.classList.add('on');if(t.dataset.t==='2')bars();setTimeout(rs,60);}}); / chain / var L=R.querySelectorAll('#ad-chain .layer'),ci=-1,tm=null; function lite(i){L.forEach(function(x,j){x.classList.toggle('lit',j===i)})} R.querySelector('#ad-step').onclick=function(){clearInterval(tm);ci=(ci+1)%L.length;lite(ci)}; R.querySelector('#ad-play').onclick=function(){clearInterval(tm);ci=-1;tm=setInterval(function(){ci++;if(ci>=L.length){clearInterval(tm);L.forEach(function(x){x.classList.add('lit')});return}lite(ci)},450)}; / loop / var S=[ [' # 1. map a dense window: depth clamped to 3 \n agent-desktop snapshot --app gnome-calculator --skeleton -i --compact\n\n{ "snapshot_id": "s2txcl95uvubmn" , "ref_count": 2,\n "children": [\n { "role": "group", "ref_id": "@s2txcl95uvubmn:e1" , "children_count": 2 },\n { "role": "group", "ref_id": "@s2txcl95uvubmn:e2" , "children_count": 1 } ] }','Only top levels are read. Cut containers carry children_count and a drill ref.'], [' # 2. drill into one region by ref \n agent-desktop snapshot --root @s2txcl95uvubmn:e1 -i --compact\n\n{ "snapshot_id": "s2txcl95uvubmn" , "ref_count": 33,\n "children": [\n { "role": "button", "name": "C", "ref_id": "@s2txcl95uvubmn:e8" },\n { "role": "button", "name": "7", "ref_id": "@s2txcl95uvubmn:e11" ,\n "available_actions": ["Click","SetFocus"] } ] }','New refs join the same snapshot. Cost follows the region, not the whole window.'], [' # 3. act on a qualified ref: headless, pointer stays put \n agent-desktop click @s2txcl95uvubmn:e11\n\n # a bare ref is refused \n agent-desktop click @e11\n{ "ok": false, "error": { "code": "INVALID_ARGS" } }','Auto-wait holds the action up to 5,000 ms until the target is visible, enabled and stable.'], [' # 4. verify: success does not prove the UI changed \n agent-desktop wait --element @s2txcl95uvubmn:e6 --predicate value --value 7\n\n # stale after a re-drill? \n{ "code": "STALE_REF" ,\n "disposition": { "delivery": "not_delivered", "retry": "safe" } }','Errors say whether a retry is safe. On STALE_REF, snapshot again and retry.']]; var si=0,code=R.querySelector('#ad-code'),note=R.querySelector('#ad-note'),st=R.querySelectorAll('.st'); function show(i){si=i;st.forEach(function(x,j){x.classList.toggle('on',j===i)});code.style.opacity=0;setTimeout(function(){code.innerHTML=S[i][0];note.textContent=S[i][1];code.style.opacity=1;rs()},150)} code.style.transition='opacity .15s'; st.forEach(function(x){x.onclick=function(){show(+x.dataset.s)}}); R.querySelector('#ad-next').onclick=function(){show((si+1)%4)}; show(0); / bars / function bars(){var f=R.querySelectorAll('#ad-bars .fill');f.forEach(function(x){x.style.transition='none';x.style.width='0'});void R.offsetWidth;setTimeout(function(){f.forEach(function(x,i){x.style.transition='width 1.1s cubic-bezier(.2,.8,.2,1) '+(i0.12)+'s';x.style.width=x.dataset.w+'%'})},30)} R.querySelector('#ad-replay').onclick=bars; /* expanders */ R.querySelectorAll('.oc,.li').forEach(function(x){x.onclick=function(){x.classList.toggle('on');setTimeout(rs,380)}}); window.addEventListener('load',rs);setTimeout(function(){R.querySelector('#ad-play').click()},700);window.addEventListener('resize',rs);setTimeout(rs,300); })();

" style="width:100%;max-width:880px;height:600px;border:0;display:block;margin:0 auto;background:transparent;" loading="lazy">

How does it compare with pixel-based computer use? Model-side computer use tools work from screenshots and pixel coordinates. Anthropic’s docs say each screenshot costs roughly 1,000 to 1,800 input tokens. Its toolset definition adds about 4,500 input tokens per request. OpenAI’s tool similarly returns click, type and scroll actions your code executes. agent-desktop is model-agnostic and returns structure, not images. Microsoft’s UFO² also uses UI Automation, but ships as a full Windows agent. Featureagent-desktop (Vercel Labs)Claude computer useOpenAI computer useMicrosoft UFO²TypeCLI + C library toolAPI client toolsetAPI toolFull agent frameworkHow it sees the UIOS accessibility treeScreenshotsScreenshots or code (PyAutoGUI, Playwright)UIA + visual hybridModel bundledNone, any agentClaude modelsOpenAI modelsConfigurable LLMPlatformsmacOS, Windows, LinuxAny env you host (reference: Linux Docker)Browser or desktop you hostWindows (UFO²)Moves your cursorNo, headless by defaultYes, pixel clicksYes, pixel clicksGUI + API actionsToken cost per look273 (Apple Maps skeleton, self-reported)~1,000 to 1,800 per screenshotNot disclosedNot disclosedLicenseApache-2.0Proprietary APIProprietary APIMITPriceFreeStandard token pricingStandard token pricingFree (plus LLM costs) How do you install and use agent-desktop? Install with npm install -g agent-desktop. Bun, Yarn and pnpm work too. The package bundles native binaries for x64 and ARM64.

macOS: grant Accessibility permission with agent-desktop permissions --guide. Windows: no grant needed for apps at your terminal’s integrity level. Linux: needs a graphical session with at-spi2-core.

Agents learn the tool from bundled docs via agent-desktop skills get desktop. A skill is also listed on skills.sh. The C library lets Python, Swift, Go, Ruby and Node hosts call it in process. For Electron web contents, launch --cdp opens a DevTools port for agent-browser. Key Takeaways

agent-desktop gives any agent computer use through OS accessibility trees. It runs on macOS, Windows and Linux, x64 and ARM64, under Apache-2.0. Headless ref actions never move your pointer or steal focus. Skeleton snapshots cut tokens 78% to 96% (self-reported). Apps without an accessibility tree remain out of reach.

Check out the GitHub repo, docs and npm package. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Vercel Labs Open-Sources agent-desktop for Agent Computer Use appeared first on MarkTechPost.