Overview & result
Agent Desktop gives any AI agent reliable computer use capabilities across native operating systems. Built in Rust, it inspects real application UI structures through accessibility trees rather than guessing coordinates from raw screenshots. Jev evaluates candidate buttons, menus, and text fields in milliseconds, ensuring reliable retries and instant action dispatch.
From the original post
Agent Desktop gives any agent reliable computer use on the desktop. Built with Rust, it sees any app's real UI structure through OS accessibility trees and operates it.
How Jev fits in the loop
- Queries the platform accessibility engine for active window hierarchies and control IDs.
- Passes candidate interactive nodes to Jev to select the target element and action.
- Dispatches deterministic mouse and keyboard events directly through OS APIs.
How to reproduce
- Clone https://github.com/lahfir/agent-desktop and compile with cargo build --release.
- Grant Accessibility permissions in macOS System Settings or Windows Security.
- Run the agent daemon to execute desktop workflows across native applications.
Why this build matters
Provides a faster, more dependable alternative to vision-based screenshot clicking, saving millions of visual tokens.
Limitations
- Requires OS-level accessibility privileges.
- Applications with custom non-standard GUI renderers (e.g., games) may lack accessibility nodes.