Overview & result
smkrv shows jev-calibrate, built on Jev: Tune your Jev questions against your own labels, then confirm on held-out data.
From the original post
jev-calibrate by @smkrv: Tune your Jev questions against your own labels, then confirm on held-out data.
How Jev fits in the loop
- You write the Jev questions you want to rely on.
- You label a set of examples with the correct answers.
- jev-calibrate runs each question against the labels and reports how often it is right, including when it is confidently wrong.
- You use the per-question result to decide whether, and at what confidence, its answers can be trusted.
How to reproduce
- Review the source documentation and repository at https://github.com/smkrv/jev-calibrate.
- Configure environment with TypeSafe Jev / System One API keys.
- Run test prompts against the decision endpoints.
Why this build matters
Shows Jev in practical use for research and data analysis.
Limitations
- Small sets give wide margins: 20 correct out of 20 still allows a true precision of 0.86 (one-sided 95% bound).
- It measures one question at a time, not whether the pipeline around it got faster or cheaper.
- Text only, as the API is.