Overview & result
Inspired by @typesafeai , here is a Jev-compatible public API to play with It runs a comparable open model (Qwen3.6-35B-A3B), and just uses SGLang radix cache to preserve the prefill reuse / really fast parallel systemone generation - 64 tasks in <1s. github.com/ekzhang/openje…
How Jev fits in the loop
- The server validates the request's schema, answer count, body size and token budget.
- It renders the chat template once and splits out a prefix shared by all questions.
- The shared prefix is sent first to warm SGLang's radix cache.
- Each question is then sent concurrently on top of that prefix, served by Qwen3.6-35B-A3B.
How to reproduce
- Inspect the original showcase and source material at https://github.com/ekzhang/openjev-sglang
- Verify the bounded prompt and input candidate schema configured for Jev
- Benchmark decision latency and classification accuracy against baseline models
Why this build matters
It reproduced the Jev HTTP API on an open model soon after launch; the author notes SGLang now ships a native decisions endpoint built the same way.
Limitations
- The author calls it an early experiment and says to use SGLang's native API instead.
- Each container runs on one B200 GPU.
- Defaults cap a request at 64 questions and 16 simultaneous evaluations.