Overview & result
An operating system control loop that replaces heavy screenshot uploads with local edge segmentation. CoreML detects bounding boxes, local OCR reads labels, and Jev chooses the next click from text candidates alone.
From the original post
Okay so Jev can actually do computer use really well
Without any screenshots, or LLMs and no Pixels leave my mac
I dont even read the Dom elements
A local CoreML model segments every button and UI element on screen.
On-device OCR reads the labels. That text is all Jev gets.
It returns a probability across those elements and tells me the best one to click.
Then it clicks, re-runs detection, and decides again. In a loop until the goal is done.
~90ms per decision. Faster than any LLM computer use I've tried.
Blazing fast computer use, without any latency
@typesafeai is building something really interesting
How Jev fits in the loop
- Local CoreML model detects all UI elements on the Mac desktop.
- On-device OCR extracts element labels and coordinates.
- Jev evaluates the label list against user intent and returns the best target.
- Local OS accessibility API triggers mouse and keyboard events.
How to reproduce
- Inspect the source implementation at https://x.com/milindlabs/status/2100631847155994852.
- Configure local Jev API credentials and register required tool/model bindings.
- Execute test prompts and observe decision latency and routing outputs.
Why this build matters
Radically cuts bandwidth and latency while guaranteeing enterprise privacy by never sending screen captures to cloud models.
Limitations
- Metrics are author-reported from the initial release unless independently verified.
- Requires access to the respective agent framework or runtime environment.
- Generative model execution remains external to the Jev decision step.