Overview & result
In the classic Wikiracing challenge (navigating between two unrelated Wikipedia articles using only internal links), Jev acts as the real-time choice model, evaluating candidate links and advancing hops in tens of milliseconds rather than waiting seconds per generative step.
From the original post
TypeSafe AI official benchmark: Jev System One model navigating Wikipedia (Wikiracing) side-by-side against general-purpose frontier LLMs. Operating in tens of milliseconds per hop (<50ms) with zero prompt tokens wasted, Jev rapidly evaluates ~140 candidate links per page to win the race with over 90% latency reduction.
How Jev fits in the loop
- Extract all viable article links from the current Wikipedia DOM.
- Format the target topic and candidate link list as a typed Choice schema.
- Jev evaluates semantic proximity and outputs the highest probability link.
- Playwright navigates to the next page and checks termination criteria.
How to reproduce
- Fetch the candidate links on the current Wikipedia article using the MediaWiki API or DOM parser.
- Pass the target concept and the link candidate array into a bounded Jev Choice call.
- Execute the chosen transition in Playwright or browser session until the target page is reached.
Why this build matters
Proves that in high-cardinality action spaces where choices are discrete and bounded, a System 1 classifier outpaces generative models by orders of magnitude in speed and cost.
Reported performance
Reported by authorLatency: <50ms per link selection hop
Limitations
- This benchmark was conducted by the vendor on Wikipedia structured linking.
- Real-world web navigation frequently encounters dynamic SPAs, popups, and anti-bot challenges that require broader fallback logic.
- Does not invent new search keywords or synthesis outside the provided link set.