COMMUNITY OBSERVEDMetric claims: AUTHOR REPORTEDSOURCE AUDITED🤖 Jev Agent

jev-papers

A thousand arXiv papers, one Jev decision each, checked by an LLM judge.

Overview & result

stas4000 shows jev-papers, built on Jev: A thousand arXiv papers, one Jev decision each, checked by an LLM judge.

From the original post

jev-papers by @stas4000: A thousand arXiv papers, one Jev decision each, checked by an LLM judge.

— stas4000

How Jev fits in the loop

  1. fetch.py pulls 1,000 recent AI papers from the public arXiv API.
  2. classify.py asks Jev one 24-way choice question per paper, four papers per request, through OpenRouter's decisions endpoint.
  3. eval.py asks a strong chat model the same question on a seeded sample of 100 papers and reports where the two disagree.
  4. A static page shows the result as a topic map.

How to reproduce

  1. Review the source documentation and repository at https://github.com/stas4000/jev-papers.
  2. Configure environment with TypeSafe Jev / System One API keys.
  3. Run test prompts against the decision endpoints.

Why this build matters

Shows Jev in practical use for research and data analysis.

Limitations

  • The judge is another model, not ground truth: the README says 85% is agreement, not accuracy.
  • The check covers a sample of 100 of the 1,000 papers.
  • Topics are 24 one-line criteria in a file; changing them changes the map.

Patterns