Overview & Result
jev-curate is a high-throughput dataset curation engine written in Rust by Akash Priyadarshi. It streams gigabyte-scale Parquet and JSONL datasets through Jev decision pipelines, scoring quality, topic relevance, and safety risks with typed Choice, Score, and Noul judgments at measured speeds up to 24 rows per second.
How Jev fits in the loop
- Streams records from Parquet or JSONL files into memory buffers.
- Dispatches speculative concurrent evaluation requests to Jev using typed criteria.
- Partitions accepted samples into training splits and discards substandard rows.
How to reproduce
- Clone https://github.com/AkashPriyadarshii/jev-curate and build with cargo build --release.
- Provide a sample Parquet or JSONL pretraining file.
- Execute the pipeline: jev-curate --input data.jsonl --rules quality.toml.
Why this build matters
Dramatically lowers the financial cost of filtering millions of synthetic or scraped data rows for pretraining and fine-tuning pipelines.
Limitations
- Requires Rust toolchain to compile.
- Throughput is bounded by network concurrency to Jev API endpoints.