PromptRR: Prompt-Driven Retrieval and Reranking

• PromptRR is a prompt-driven, graph-based retrieval-augmented generation framework that uses sequential LLM prompts for entity extraction, fact gating, and passage reranking.
• It employs a three-stage prompting protocol to construct a symbolic knowledge graph and applies Personalized PageRank for scalable multi-hop reasoning.
• Empirical results on HotpotQA and 2WikiMultiHopQA show significant gains in retrieval recall and QA accuracy through optimized prompt engineering and fact selection.

A Prompt-Driven Retrieval and Reranking Pipeline in Graph-Based Retrieval-Augmented Generation (PromptRR)

Retrieval-augmented generation (RAG) frameworks integrate external knowledge retrieval with LLMs to enable complex reasoning, especially in multi-hop question answering. The PROPEX-RAG system introduces a prompt-driven, graph-based RAG pipeline that systematically incorporates lightweight, human-interpretable prompts at each stage of the retrieval and answer generation process, achieving state-of-the-art multi-hop QA performance on HotpotQA and 2WikiMultiHopQA (Sarnaik et al., 3 Nov 2025). This architecture, termed Prompt-Driven Retrieval and Reranking (PromptRR), leverages LLMs both offline and online to optimize entity graph construction, fact selection, semantic passage reranking, and answer synthesis, demonstrating the critical role of prompt engineering in retrieval accuracy and multi-hop reasoning.

1. Prompt Staging and Template Design

PromptRR interleaves three LLM-based prompting stages to orchestrate both the construction of the symbolic knowledge graph and online retrieval:

1. Entity Extraction ("Entity-Seed Extraction"):
   • Prompted LLM (e.g., GPT-4.1-mini) receives a passage and returns a structured JSON array of salient entities, each labeled by type (Person, Organization, Location).
   • Output: { 'entity': ..., 'type': ... }, without commentary.
2. Fact Triple Generation ("Fact-Triple Elicitation"):
   • LLM is prompted to extract factual assertions from a passage, formatted as triples (subject, predicate, object), where subjects and objects exactly match the prior extracted entities.
   • Predicates are canonicalized to short verb phrases ("born in", "headquartered at"), and output as one triple per line.
   • Each triple is embedded (e.g., with text-embedding-3-large) for vector search.
3. Fact Gating and Passage Reranking ("Fact-Gating & Rerank"):
   • LLM selects relevant triples for a query by returning a keep/drop decision per triple, then scores top-k retrieved passages on: (a) mention of any kept triple subject/object, (b) triple–query semantic alignment.
   • Example JSON output: { "kept_triples": [indices…], "passage_scores": [ { "passage": 1, "score": 4 }, ... ] }.

This hierarchical decomposition of prompt roles enables explicit control over entity induction, fact graph sparsification, and semantic passage scoring.

2. Symbolic Graph Construction and Fact Representation

The offline index is a heterogeneous, row-normalized directed graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$:

• Nodes ($\mathcal{V}$): partitioned into entities ($E$) and passages ($P$).
• Edges ($\mathcal{E}$):
  • Mentioned-in: $e \rightarrow p$
  • Synonymy: $e_i \leftrightarrow e_j$, weight $w_{sim}$
  • Contextual-cooccurrence: $e_i \leftrightarrow e_j$, weight $w_{rel}$ from triple co-occurrence statistics.

All extracted facts are stored as triples $\mathcal{V}$0 and utilized for multi-hop reasoning and downstream retrieval. The adjacency matrix $\mathcal{V}$1 is row-normalized.

3. Entity-Guided Retrieval with Personalized PageRank

PromptRR’s online retrieval harnesses Personalized PageRank (PPR), with entity seeding and triple gating directed by LLM calls:

• Query Embedding and Triple Retrieval:
  • The user query $\mathcal{V}$2 is embedded ($\mathcal{V}$3).
  • Candidate triples $\mathcal{V}$4 are retrieved by top-k cosine similarity.
• LLM Fact Gating and Entity Selection:
  • LLM gates $\mathcal{V}$5 (keep/drop) and extracts seed entities ($\mathcal{V}$6).
• PPR Initialization and Iteration:
  • Restart vector $\mathcal{V}$7 is uniform over $\mathcal{V}$8; other entries zero.
  • Power iteration: $\mathcal{V}$9, $E$0.
• Final Passage Scoring:
  • For each passage $E$1, compute $E$2, where overlap counts matched query entities or retained facts, $E$3.
  • Top passages are selected for answer generation.

This entity-guided traversal scheme enables scalable, context-aware multi-hop retrieval over the symbolic graph.

4. Prompt-Driven Fact Selection and Multi-Stage Reranking

Fact selection and passage reranking rely on prompt-based LLM inference at two key decision points:

• First Gating (Triples): The “keep-drop” prompt discards irrelevant facts, yielding $E$4.
• Hybrid Passage Scoring: LLM passage scores (from fact gating) are linearly combined with the PPR entity-based score to form a hybrid relevance score.
• Second Fine-Grained Reranking: Supplementary LLM prompts (“Given the kept facts, is passage X helpful to answer Q? Rate 1–5.”) refine the ranking prior to final answer generation.

Empirical ablations indicate that omitting fact gating or removing fine reranking degrades both retrieval recall and end-to-end QA metrics.

5. Selective and Cost-Efficient LLM Invocation

PromptRR achieves high efficiency and interpretability by invoking the LLM strictly at necessary junctures:

1. Offline indexing: Entity extraction and triple elicitation (once per document).
2. Online retrieval, early: Triple relevance gating (on the order of 5–10 triples).
3. Online retrieval, late: Final QA generation over the highest-scoring passages.

Final Answer Generation Prompt:

$E$5

This mode of operation avoids retraining of large retrievers and re-rankers, and directly exposes the intermediate entity/fact/passages to system designers for interpretability.

6. Benchmark Results and Ablation Studies

On HotpotQA and 2WikiMultiHopQA, PromptRR achieves:

• Retrieval Recall@5: 97.1% (HotpotQA), 98.1% (2WikiMultiHopQA)
• QA F1 / Exact Match: F1 80.7% / EM 79.9% (HotpotQA), F1 78.9% / EM 76.4% (2WikiMultiHopQA)
• Ablation Impact:
  • Removing fact gating: Recall@5 drops ~4 points.
  • Replacing PPR with dense retrieval: F1 drops ~6 points.
  • Using fewer seed entities: EM drops ~3 points.

These results quantify the contribution of prompt-based entity seeding, fact gating, and passage reranking to both retrieval and multi-hop QA end performance.

7. Implications, Design Principles, and Future Directions

PromptRR establishes the importance of well-designed, lightweight prompts for fine-grained control over entity induction, graph traversal, and answer synthesis in graph-based RAG. The explicit integration of prompt-driven decision points enables modular improvements without retraining foundation models, and the calibration of prompt templates can directly increase retrieval accuracy and QA F1.

This paradigm—prompt-aware, symbolic fact graph traversal orchestrated by selective, interpretable LLM prompting—offers a robust, scalable path for future multi-hop and complex reasoning systems, and provides empirical evidence that prompt engineering is a lever for unlocking advanced reasoning in retrieval-augmented generative architectures (Sarnaik et al., 3 Nov 2025).