Retrieval-Judgment-Exploration (RJE) Pattern

• RJE is a three-stage workflow that retrieves candidate evidence, assesses its sufficiency, and selectively expands the search when needed.
• It is applied in domains such as legal judgment prediction, KGQA, and query-decomposition RAG to enhance retrieval accuracy and efficiency.
• Empirical studies demonstrate that integrating an explicit judgment step with conditional exploration improves performance over conventional retrieval methods.

Retrieval–Judgment–Exploration (RJE) denotes a three-stage workflow in which a system first retrieves candidate evidence, then judges whether the available evidence is sufficient, and finally explores additional evidence or alternative retrieval directions when it is not. The label appears in multiple technically distinct settings: Athena’s legal judgment prediction pipeline, a knowledge graph question answering framework explicitly named RJE, and a bandit-based retrieval-augmented generation formulation for query decomposition (Peng et al., 2024, Lin et al., 25 Sep 2025, Petcu et al., 21 Oct 2025). Across these usages, the common structural claim is that retrieval alone is not treated as final; an explicit judgment step mediates whether further search is necessary.

1. Scope and problem formulations

In legal judgment prediction, Athena applies RJE to a labeled LJP dataset in which each case $l_i$ carries a ground-truth accusation label $j_i$. The retrieval target is a compact legal knowledge base keyed by distinct accusation names $\{a_1,\dots,a_M\}$, and the downstream prediction target is a single accusation label for a new case (Peng et al., 2024).

In KGQA, the framework is defined over a knowledge graph

$\mathcal{G}=(E,R),$

where $E$ is the entity set and $R$ is the relation set. Given a natural-language question $q$, the goal is to return an answer set

$A_q \subseteq E.$

A small set of topic entities

$T=\{e_{t_1},\dots,e_{t_n}\}$

is assumed to have already been linked from $q$ into $j_i$0. The central retrieval objects are reasoning paths

$j_i$1

with candidate pool

$j_i$2

(Lin et al., 25 Sep 2025).

In query-decomposition RAG, RJE is formulated as an exploitation-exploration problem. A complex user query $j_i$3 is decomposed into $j_i$4 simpler sub-queries $j_i$5, and each sub-query is associated with a ranked list of candidate documents $j_i$6. The system then allocates a limited judgment budget across these ranked lists, deciding whether to exploit the currently most informative sub-query or explore a different one (Petcu et al., 21 Oct 2025).

This distribution of formulations suggests that RJE is not a single invariant algorithm. A plausible implication is that it is better understood as a reusable architectural pattern whose retrieval object may be accusation descriptions, reasoning paths, or ranked documents, while the judgment object may be label sufficiency, path sufficiency, or document relevance.

2. Canonical stages and their operational roles

The three stages are structurally consistent across the published variants, but their semantics are domain-specific.

Setting	Retrieval stage	Judgment stage
Athena	retrieve top-$j_i$7 accusation candidates from a vectorized accusation KB	parse the LLM reply for Final Judgment: and map the accusation name back to the label set
KGQA RJE	retrieve relation paths, instantiate them into reasoning paths, and rank top-$j_i$8 paths	ask the LLM whether $j_i$9 is sufficient
Bandit RAG	retrieve documents from ranked lists for decomposed sub-queries	obtain binary relevance judgments $\{a_1,\dots,a_M\}$0 or continuous ranking scores

In Athena, retrieval is a preprocessing component that supplies few-shot exemplars consisting of accusation name, definition, and synthetic example. Judgment is embedded in the prompted LLM output: the prompt requires a structured chain of norms $\{a_1,\dots,a_M\}$1 facts $\{a_1,\dots,a_M\}$2 judgment, and the final label is read from the line Final Judgment: 〈Accusation Name〉 (Peng et al., 2024).

In KGQA, the judgment stage is formalized explicitly as

$\{a_1,\dots,a_M\}$3

If $\{a_1,\dots,a_M\}$4, the LLM emits its answer immediately; otherwise the system proceeds to exploration (Lin et al., 25 Sep 2025).

In the bandit formulation, judgment is not a single sufficiency verdict but a sequential relevance signal. Each arm pull corresponds to retrieving one more document from a sub-query’s ranked list and observing a reward, either binary or continuous. Exploration and exploitation are then governed by posterior updates over sub-query utility (Petcu et al., 21 Oct 2025).

A common misconception is to equate RJE with generic RAG. The published formulations indicate a narrower pattern: retrieval is followed by an explicit evaluative checkpoint, and exploration is conditional on that checkpoint rather than being an unconditional extension of the initial retrieval set.

3. Athena and legal judgment prediction

Athena constructs a legal knowledge base by starting from a labeled LJP dataset, collecting all distinct accusation names $\{a_1,\dots,a_M\}$5, and using an LLM such as GPT-4 to generate for each accusation $\{a_1,\dots,a_M\}$6 a human-readable definition $\{a_1,\dots,a_M\}$7 and a short synthetic example case $\{a_1,\dots,a_M\}$8. For each accusation, the tuple $\{a_1,\dots,a_M\}$9 is stored as metadata alongside its embedding. Vectorization uses OpenAI’s text-embedding-ada-002, with dimensionality $\mathcal{G}=(E,R),$0, under the embedding function

$\mathcal{G}=(E,R),$1

Semantic retrieval for a new legal case $\mathcal{G}=(E,R),$2 uses cosine similarity,

$\mathcal{G}=(E,R),$3

to rank accusations by similarity between the query embedding $\mathcal{G}=(E,R),$4 and the stored example embeddings $\mathcal{G}=(E,R),$5, returning the top-$\mathcal{G}=(E,R),$6 candidates $\mathcal{G}=(E,R),$7 (Peng et al., 2024).

The prompt template has three parts: case facts, retrieved candidates, and output format instructions. For each retrieved candidate $\mathcal{G}=(E,R),$8, the prompt includes accusation name, definition $\mathcal{G}=(E,R),$9, and example $E$0. The output instructions may require the model to list the relevant legal norms, summarize the facts against each retrieved candidate, and state Final Judgment: followed by the single most likely accusation label. The template may optionally append “Let’s think step by step.”, but Athena’s own template already imposes a structured chain of norms $E$1 facts $E$2 judgment (Peng et al., 2024).

The exploration component in Athena is realized through ablation and diagnostic analysis rather than graph traversal. The in-context window-size ablation varies

$E$3

with $E$4 as the baseline. Accuracy first rises, then dips, then plateaus, reproducing a “lost-in-the-middle” phenomenon tied to the transformer’s relative positional encoding. The reported accuracy curve for GPT-4o on CAIL2018 peaks around $E$5, while Retrieval Hit Rate climbs with $E$6 even when judgment accuracy can fall because the model “forgets” earlier context. Athena also compares two KB-building schemes—{accusation name only} versus {name + LLM-generated definition + example}—and reports that rewritten descriptions yield a +10 pts Hit Rate boost at small k (e.g. k=5), while the original scheme needs nearly twice the $E$7 to reach the same Hit Rate. The long-tailed character of CAIL2018 is analyzed through

$E$8

and

$E$9

(Peng et al., 2024).

Quantitatively, GPT-4o + Athena (k=∞) → 0.91 accuracy on 256-case sample, and With tuned k=32, GPT-4o + Athena → 0.95 accuracy (95 %). Athena reports only overall accuracy, although the paper also states that one can define precision, recall, and $R$0. In discussion, retrieval is described as injecting up-to-date, domain-specific knowledge into each prompt so that the LLM leverages grounded context rather than trying to recall legal norms from pretraining, and Athena is reported to outperform bare prompt / zero-shot, zero-shot CoT, and legal-syllogism prompting (Peng et al., 2024).

4. The explicit RJE framework for KGQA

The KGQA framework defines Retrieval, Judgment, and Exploration as a fixed control flow. In the retrieval stage, a fine-tuned PLM generates a set of relation paths

$R$1

where each relation path $R$2 is scored via beam search. These are instantiated into full reasoning paths $R$3 by retrieving all valid entity instantiations in $R$4. To suppress noise, a lightweight ranker re-scores each reasoning path. Given

$R$5

the score is

$R$6

where $R$7 extracts the $R$8 embedding. The top-$R$9 paths

$q$0

are retained, and the ranker is trained with weak supervision using the margin-ranking loss

$q$1

(Lin et al., 25 Sep 2025).

The judgment stage prompts the LLM with $q$2 and asks a binary sufficiency question. If the evidence is insufficient, exploration begins. Exploration is iterative, with cap $q$3 equal to 4 for CWQ, 2 for WebQSP. The system maintains the accumulated paths $q$4 and their entities $q$5. The exploration loop comprises five named steps: Question Decomposition; Exploration Entity Selection, which chooses a small frontier entity set $q$6; Retriever-assisted Relation Exploration, which issues SPARQL to list incident relations $q$7, pre-filters them with the relation-path retriever to top-$q$8 candidates, and prompts the LLM to select a subset $q$9; Entity Exploration, which retrieves tail entities and filters them with a train-free BERT before LLM selection; and Path Update and Answer Generation, which extends $A_q \subseteq E.$0 and repeats sufficiency judgment if necessary. The reported settings are temperature = 0.3, top-K = 10 initial paths, default N = 30, and $A_q \subseteq E.$1 or 4 (Lin et al., 25 Sep 2025).

Empirically, the paper reports Hits@1 on CWQ / WebQSP for several systems: LLM-Only (CoT, SC): 38.8/62.2 → 45.4/61.1, ReasoningLM: 69.0/78.5, PoG w/ChatGPT: 63.2/82.0, PoG w/GPT-4: 75.0/87.3, RJE w/Llama-3B: 62.9/82.6, RJE w/Llama-8B: 71.5/89.2, and RJE w/GPT-4o-mini: 77.1/92.5 (best overall). For Macro-$A_q \subseteq E.$2, RJE w/DeepSeek-V3: 70.2/78.9 (state-of-the-art). The paper also reports improvements over PoG on CWQ of +41.5% with 3B, +27.9% with 8B, +9.9% with 14B, and +9.9% with GPT-4o-mini. Efficiency results include average LLM calls per query on CWQ of ToG 22.6, PoG 13.3, RJE 7.9; token usage per CWQ query of PoG 7,803 vs RJE 5,769 input tokens and PoG 353 vs RJE 247 output tokens; average LLM calls per WebQSP query of ToG 15.9, PoG 9.0, RJE 4.1; and wall-clock time per CWQ query of ToG 96.5s, PoG 23.3s, RJE 16.3s (Lin et al., 25 Sep 2025).

The significance of this formulation lies in its explicit separation between sufficiency assessment and further search. Rather than treating retrieval depth as fixed, the framework lets the LLM terminate early when the current path set is adequate or continue expanding only selected frontiers when it is not.

5. Bandit-based RJE for query decomposition in RAG

In the query-decomposition setting, RJE is defined as a three-step cycle built around a multi-armed bandit formulation of RAG. Retrieval decomposes a complex query $A_q \subseteq E.$3 into $A_q \subseteq E.$4 sub-queries and pre-fetches a ranked list of $A_q \subseteq E.$5 candidate documents for each sub-query. Judgment iteratively selects one sub-query, pulls its next document, and obtains either an explicit relevance judgment $A_q \subseteq E.$6 or a continuous ranking score. Exploration versus exploitation then updates a posterior belief about each sub-query’s utility and decides whether to continue pulling from the currently strongest sub-query or switch to another one whose utility remains uncertain (Petcu et al., 21 Oct 2025).

Two reward models are studied. In the Bernoulli model, utility is binary and each arm $A_q \subseteq E.$7 has a Beta prior $A_q \subseteq E.$8 with initial values $A_q \subseteq E.$9. After observing reward $T=\{e_{t_1},\dots,e_{t_n}\}$0 on chosen arm $T=\{e_{t_1},\dots,e_{t_n}\}$1,

$T=\{e_{t_1},\dots,e_{t_n}\}$2

In the Gaussian model, utility is a real-valued score $T=\{e_{t_1},\dots,e_{t_n}\}$3, with a conjugate Gaussian prior $T=\{e_{t_1},\dots,e_{t_n}\}$4. The decision rules studied are Thompson Sampling and UCB. For UCB, the empirical mean is

$T=\{e_{t_1},\dots,e_{t_n}\}$5

and the selected arm is

$T=\{e_{t_1},\dots,e_{t_n}\}$6

The best performing policy is reported as a Rank-aware top-k Bernoulli UCB with [diversity](https://www.emergentmind.com/topics/diversity-beta-recall), with instantaneous reward

$T=\{e_{t_1},\dots,e_{t_n}\}$7

This combines top-$T=\{e_{t_1},\dots,e_{t_n}\}$8 averaging of binary judgments, a cosine-similarity diversity penalty, and a logarithmic term that ensures untried arms receive infinite UCB bonus as $T=\{e_{t_1},\dots,e_{t_n}\}$9 (Petcu et al., 21 Oct 2025).

The experimental setup includes NeuCLIR24, where each query is decomposed into 16 LLM-generated sub-queries, and ResearchyQuestions, described as 100 K Bing questions with hierarchical decomposition into two levels of sub-queries. Budgets are $q$0, and the experiments use 1 000 random seeds × 10 runs of query decompositions. Evaluation includes document-level precision@$q$1, $q$2-nDCG@$q$3, and downstream long-form generation via Auto-ARGUE, reporting Citation Relevance, Citation Support, Nugget Coverage, and Sentence Support. The reported findings are +35% document-level precision, +15% α-nDCG, 6–10% boosts in citation support, 7–9% in nugget coverage, and 7–10% in sentence support. Ablations report that full exploitation and full exploration both underperform, ε-greedy works well with very small budgets, Bernoulli posteriors outperform Gaussian posteriors on noisy real retrieval scores, Top-k averaging (k=3…5) yields a 5–10% lift over single-document pulls, diversity penalties modestly help, and hierarchical bandits give an additional 20–30% precision gain under tight budgets on ResearchyQuestions (Petcu et al., 21 Oct 2025).

In this formulation, “Judgment” is literal supervision or pseudo-supervision over retrieved evidence, and “Exploration” is a resource-allocation policy over decomposed retrieval channels. This differs from the KGQA formulation, where judgment is a sufficiency verdict over reasoning paths, and from Athena, where the most prominent exploratory analyses concern context size and KB construction.

6. Interpretive issues, misconceptions, and extensions

The published record shows that “RJE” is polysemous. In Athena, the end-to-end Retrieval–Judgment–Exploration pipeline is tightly coupled to prompt engineering for legal judgment prediction, with exploration expressed through in-context window-size ablation, query rewriting analysis, and data distribution analysis (Peng et al., 2024). In the KGQA framework, exploration is a conditional graph expansion procedure over frontier entities and relations, and the method is designed to address the limitation that retrieval-based methods are constrained by the quality of retrieved information while agent-based methods rely heavily on proprietary LLMs (Lin et al., 25 Sep 2025). In the bandit-based RAG formulation, exploration is the explicit exploration-exploitation trade-off over sub-queries, governed by posterior updates and decision rules such as Thompson Sampling and UCB (Petcu et al., 21 Oct 2025).

Accordingly, it is inaccurate to treat RJE as a single standard pipeline with fixed modules. A plausible implication is that the term identifies a control principle—retrieve, assess sufficiency or utility, and expand selectively—rather than one canonical implementation. The main invariant is the presence of a judgment step that mediates between initial retrieval and further evidence acquisition.

The proposed extensions are likewise domain-specific. Athena lists Multi-vector indexing, Hybrid sparse + dense retrieval, Reflection / self-critique modules, Dynamic context window resizing at runtime, “Why-not” explanations, and Legal-expert-in-the-loop refinement of query rewriting schemas and near-synonym disambiguation (Peng et al., 2024). The KGQA framework’s auxiliary modules—Reasoning Path Ranking, Question Decomposition, and Retriever-assisted Exploration—already indicate one direction for modularization around small-sized LLMs without fine-tuning (Lin et al., 25 Sep 2025). The bandit formulation points instead toward improved posterior models, rank-aware reward shaping, and hierarchical exploration under budget constraints (Petcu et al., 21 Oct 2025).

Taken together, these works position Retrieval–Judgment–Exploration as a design pattern for evidence-grounded LLM systems in which retrieval quality, sufficiency assessment, and adaptive search are treated as separate but interdependent components. The technical differences across legal judgment prediction, KGQA, and RAG are substantial, but the shared premise is consistent: retrieval should be coupled to an explicit judgment mechanism, and further exploration should be conditional rather than indiscriminate.