Jev-as-a-Judge: Structured Decisions for Brazilian Portuguese IR
This presentation examines Jev, a System One model designed for bounded decision-making in information retrieval, evaluating its performance in two Brazilian Portuguese tasks: automatic relevance assessment and candidate reranking. Despite only 53-65% exact label agreement with reference judgments, Jev preserves retrieval system rankings with 0.90-0.93 correlation and dramatically improves BM25 ranking effectiveness on the Quati passage collection, demonstrating that structured probability-based decisions can substitute expensive manual annotation and enhance first-stage retrieval without generative overhead.Script
Can a model built for structured decisions replace expensive human annotation in information retrieval? Jev, a System One model, tackles two hard problems in Brazilian Portuguese: generating relevance judgments and reranking search results, without generating a single word of text.
The researchers tested Jev on four Portuguese collections spanning web passages, jurisprudence, and legislative documents. For each query and document pair, Jev selects a predefined relevance label rather than generating explanatory text, treating annotation as a bounded classification decision.
Here is where the results become counterintuitive. Jev agreed with human judgments on only 53 to 65 percent of individual labels, yet when researchers used those imperfect judgments to rank fifteen retrieval systems, the system ordering remained almost identical, with correlations between 0.90 and 0.93.
As a reranker, Jev independently scored each of the top 1000 BM25 candidates using relevance probabilities. On the Quati passage collection, this pointwise approach increased effectiveness from 0.33 to 0.63, moving the first relevant passage dramatically earlier in the ranked list.
The study leaves critical questions unanswered. All reranking experiments used short web passages retrieved by a single BM25 configuration. Whether probability-based scoring transfers to full-length legal documents, where relevance depends on distributed evidence and document structure, remains untested.
Jev demonstrates that imperfect individual judgments can still preserve the conclusions that matter: which retrieval systems work better than others. If you want to explore how structured decisions are reshaping evaluation in specialized languages, visit EmergentMind.com to dive deeper and create your own videos.
