Overview & Result
Agent Desktop gives any AI agent reliable computer use capabilities across native operating systems. Built in Rust, it inspects real application UI structures through accessibility trees rather than guessing coordinates from raw screenshots. Jev evaluates candidate buttons, menus, and text fields in milliseconds, ensuring reliable retries and instant action dispatch.
How Jev fits in the loop
- Queries the platform accessibility engine for active window hierarchies and control IDs.
- Passes candidate interactive nodes to Jev to select the target element and action.
- Dispatches deterministic mouse and keyboard events directly through OS APIs.
How to reproduce
- Clone https://github.com/lahfir/agent-desktop and compile with cargo build --release.
- Grant Accessibility permissions in macOS System Settings or Windows Security.
- Run the agent daemon to execute desktop workflows across native applications.
Why this build matters
Provides a faster, more dependable alternative to vision-based screenshot clicking, saving millions of visual tokens.
Limitations
- Requires OS-level accessibility privileges.
- Applications with custom non-standard GUI renderers (e.g., games) may lack accessibility nodes.