RUBEN: Rule-Based Explanations for Retrieval-Augmented LLM Systems

Abstract: This paper demonstrates RUBEN, an interactive tool for discovering minimal rules to explain the outputs of retrieval-augmented LLMs in data-driven applications. We leverage novel pruning strategies to efficiently identify a minimal set of rules that subsume all others. We further demonstrate novel applications of these rules for LLM safety, specifically to test the resiliency of safety training and effectiveness of adversarial prompt injections.

• The paper introduces RUBEN, which formulates minimal if-then rules linking specific retrieval sources to LLM outputs, improving source attribution.
• It employs a lattice-based traversal with dynamic pruning to efficiently isolate minimal adversarial source sets triggering safety violations in weak models.
• Empirical evaluations across finance, software security, and PII scenarios demonstrate RUBEN’s value in enhancing LLM safety auditing and compliance.

RUBEN: Rule-Based Explanations for Retrieval-Augmented LLM Systems

Introduction and Motivation

This paper introduces RUBEN, an interactive tool that formulates minimal, rule-based explanations for the outputs of retrieval-augmented LLMs, with a pronounced focus on LLM safety analysis and adversarial robustness. LLMs employing retrieval-augmented generation (RAG) often blend external knowledge into their responses, confounding efforts to attribute model outputs to their underlying sources. Existing attribution methods—including feature attribution and counterfactual explanations—are limited in specificity, consistency, or usability, especially in operational, real-world systems.

RUBEN directly addresses these deficiencies by mining minimal if-then rules that associate the presence of specific RAG sources with observed model behavior, especially when outputs violate intended constraints (e.g., safety guardrails). By doing so, RUBEN promises significantly increased transparency, supporting rigorous safety evaluation, adversarial testing, and systematic provenance recovery in LLM deployments.

System Architecture and Rule Mining

RUBEN’s architecture is oriented around modularity and efficiency (Figure 1). Users define a query, optional adversarial instructions, and an output predicate $O$ which asserts whether the model’s response satisfies certain conditions (e.g., violates a safety constraint). A retriever component selects relevant sources $D_q$ for the query. Adversarial instructions can be interactively injected into these sources, simulating data poisoning or prompt injection attacks before passing them to the LLM.

Figure 1: The architecture of RUBEN, illustrating user query flow, retrieval, adversarial modification, LLM inference, and rule mining workflow.

Central to RUBEN is a rule miner that explores the power set $\mathcal{P}(D_q)$ of retrieved sources. For each subset $S \subseteq D_q$, the LLM is evaluated and its output assessed against the predicate $O$. The goal is to enumerate all minimal $S$ such that the presence of $S$ guarantees satisfaction of $O$—with “minimal” meaning that no subset suffices, enhancing explanation specificity. The rule mining algorithm leverages lattice-based traversal and dynamic pruning based on the Apriori property, significantly reducing the search and computational cost by eliminating infeasible branches early.

Rule Formalism and Pragmatic Flexibility

RUBEN employs a highly general rule formulation paradigm:

• Antecedent: Defined by the presence of specific RAG sources in the LLM prompt.
• Consequent: Defined by any explicit predicate on the model output, permitting both syntactic rules (via string matching or regex) and sophisticated semantic evaluations (including “LLM-as-a-judge” approaches).

This framework enables RUBEN to identify not only direct information leakage or instruction-following failures, but also nuanced output behaviors relevant to safety, bias, and reliability. Practical non-determinism in $M$ or $O$ is acknowledged and traced as a variable impacting consistency across repeated tests.

Empirical Scenarios: LLM Safety Evaluation

Three demonstration use cases exemplify the practical utility of RUBEN in adversarial and safety-critical environments:

• Finance Data Poisoning: Adversarial instructions to output forbidden contact information are inserted into retrieved sources. With weaker LLMs, RUBEN identifies minimal sets of adversarial sources sufficient to subvert safeguards, effectively quantifying the ease of exploitation and the model’s resilience. Notably, stronger LLMs show no such vulnerabilities, evidenced by the absence of minimal rules.
• Software Security: Counterfeit sources containing malicious code or instructions are injected. RUBEN discovers that even a single poisoned source can bypass weak LLM safety training, but not robust ones. This underscores the use of RUBEN as a practical instrument for rapid iterative red-teaming.
• PII Red Teaming in Employment Domains: RUBEN is used to probe for successful leakage of personally identifiable information (PII) under adversarial conditions. Across tested models and scenarios, no rules are found, indicating robust defenses against PII exfiltration via RAG-source manipulation.

  Figure 2: Using RUBEN to test adversarial prompt injections against an LLM's safety training, with user-controlled injection and real-time rule discovery.

These use cases demonstrate RUBEN’s ability to isolate contributing sources, quantify minimal risk surfaces, and provide actionable insights for fortifying LLM safety training.

Technical Insights and Numerical Results

RUBEN’s rule mining algorithm, based on top-down level traversal of the source lattice with intelligent caching, is validated for real-time operability. Empirical evaluations (referenced in a technical report) demonstrate that substantial pruning of the search space is realized via Apriori-style optimizations, making it feasible to deploy RUBEN interactively even as candidate source sets grow moderately large.

Crucially, RUBEN is able to surface minimal rules identifying as few as one or two sources sufficient to force a safety violation in weak LLMs, while returning no rules for robust models, thus substantiating and quantifying the marginal value of adversarial instructions and model upgrades in adversarial alignment settings.

Implications and Future Directions

RUBEN contributes a principled, interpretable methodology for tracing LLM behavior to RAG sources, enhancing user understanding, compliance auditing, and regulatory verification. The mathematical formalism underlying the rule miner paves the way for generalization to larger, distributed retrieval pipelines and the integration of richer semantic output predicates, including those evaluated over structured data and model confidence measures.

Future work will likely explore:

• Integration with automated adversarial testing pipelines.
• Expansion to more complex, higher-cardinality RAG architectures.
• Longitudinal analysis of rule stability and transferability across datasets, domains, and LLM releases.
• Extension to continual learning scenarios, measuring risk drift as new sources are admitted.
• Advanced visualization for rule hierarchies and interactive exploration.

Conclusion

RUBEN establishes a rigorous, interactive, and highly interpretable framework for rule-based explanation of retrieval-augmented LLM systems. By enabling succinct, minimal provenance tracing—under adversarial, safety-critical, and standard use conditions—RUBEN advances the state of practical explainability and safety auditing in contemporary LLM deployments. Its use of efficient, lattice-based rule mining with dynamic pruning ensures scalability to real-world tasks, and its flexibility in predicate specification renders it adaptable for future safety, compliance, and trustworthiness applications in AI.