Overview & Result
An operating system control loop that replaces heavy screenshot uploads with local edge segmentation. CoreML detects bounding boxes, local OCR reads labels, and Jev chooses the next click from text candidates alone.
How Jev fits in the loop
- Local CoreML model detects all UI elements on the Mac desktop.
- On-device OCR extracts element labels and coordinates.
- Jev evaluates the label list against user intent and returns the best target.
- Local OS accessibility API triggers mouse and keyboard events.
How to reproduce
- Inspect the source implementation at https://x.com/milindlabs/status/2100631847155994852.
- Configure local Jev API credentials and register required tool/model bindings.
- Execute test prompts and observe decision latency and routing outputs.
Why this build matters
Radically cuts bandwidth and latency while guaranteeing enterprise privacy by never sending screen captures to cloud models.
Limitations
- Metrics are author-reported from the initial release unless independently verified.
- Requires access to the respective agent framework or runtime environment.
- Generative model execution remains external to the Jev decision step.