Overview & result
Aaaaaand this is now open-source here: A tiny AI pointer companion for your Mac driven by JEV or Gemini Live - Bring your own keys - Local OmniParser running on CoreML - Jev drives the pointer - Ctrl + K to type a task Voice mode next if people want it! https://github.com/milind-soni/tiptour-macos
How Jev fits in the loop
- Press Ctrl+K and type a click-based task.
- TipTour detects controls on screen locally and JEV chooses among them.
- TipTour executes the click and validates the action.
- The loop stops when JEV chooses none or reports completion, on a failure, or at 12 actions.
How to reproduce
- Inspect the original showcase and source material at https://github.com/milind-soni/tiptour-macos
- Verify the bounded prompt and input candidate schema configured for Jev
- Benchmark decision latency and classification accuracy against baseline models
Why this build matters
It pairs two modes in one menu bar app: Gemini realtime for talking about the screen and JEV for typed click tasks, each with your own key kept in the macOS Keychain.
Limitations
- JEV mode supports single, double and right clicks only; it cannot see images or generate text.
- JEV acts on its top-ranked target without a confidence cutoff.
- A task stops at a 12-action limit.