Overview & result
stas4000 shows jev-papers, built on Jev: A thousand arXiv papers, one Jev decision each, checked by an LLM judge.
From the original post
jev-papers by @stas4000: A thousand arXiv papers, one Jev decision each, checked by an LLM judge.
How Jev fits in the loop
- fetch.py pulls 1,000 recent AI papers from the public arXiv API.
- classify.py asks Jev one 24-way choice question per paper, four papers per request, through OpenRouter's decisions endpoint.
- eval.py asks a strong chat model the same question on a seeded sample of 100 papers and reports where the two disagree.
- A static page shows the result as a topic map.
How to reproduce
- Review the source documentation and repository at https://github.com/stas4000/jev-papers.
- Configure environment with TypeSafe Jev / System One API keys.
- Run test prompts against the decision endpoints.
Why this build matters
Shows Jev in practical use for research and data analysis.
Limitations
- The judge is another model, not ground truth: the README says 85% is agreement, not accuracy.
- The check covers a sample of 100 of the 1,000 papers.
- Topics are 24 one-line criteria in a file; changing them changes the map.