Overview & result
Benchmark evaluation testing Jev paired with Mercury 2.5 against GPT-6 Astra. Jev picked the required tool from WebMCP, while the fast LLM generated arguments, achieving 100% completion at a fraction of standard cost.
From the original post
We just ran Jev on our WebMCP benchmark.
The result: basically broke the benchmark.
Jev + Mercury 2.5 (a fast, low-cost LLM) using WebMCP solved 100% of the tasks at roughly 112× lower model cost than GPT-6 Astra using computer use with code execution. Compared to Astra using screenshot-based computer use, the model cost was 245× lower (!).
We also compared Jev operating the browser with and without WebMCP.
We used Browser Use’s open-source Ultrafast, with some improvements to the harness to make it more reliable across the benchmark.
Jev’s browser-control accuracy on its own was not amazing - adding WebMCP nearly doubled the number of solved tasks, from 25/49 to 49/49, while reducing model cost by 18% (more on why below).
The benchmark and methodology are fully open and reproducible.
Full results:
How Jev fits in the loop
- Extract WebMCP available tools on benchmark step.
- Jev Choice identifies correct tool function in tens of milliseconds.
- Fast small LLM populates parameters and verifies result.
How to reproduce
- Inspect the source implementation at https://x.com/0xidanlevin/status/2100937437325205568.
- Configure local Jev API credentials and register required tool/model bindings.
- Execute test prompts and observe decision latency and routing outputs.
Why this build matters
Rigorous empirical proof that offloading decision boundaries from frontier models achieves parity at drastic savings.
Reported performance
Reported by authorCost: 112× lower model cost
Latency: ~7 s run time
Limitations
- Metrics are author-reported from the initial release unless independently verified.
- Requires access to the respective agent framework or runtime environment.
- Generative model execution remains external to the Jev decision step.