Rule2Text: Translating KG Rules to NL

• Rule2Text is a framework that converts mined knowledge graph rules into intuitive natural language explanations using integrated neural and neurosymbolic techniques.
• It employs variable-type extraction, chain-of-thought prompting, and LLM orchestration to optimize explanation correctness and clarity.
• Evaluation merges human annotation with LLM-based judging, demonstrating improvements in metrics like BLEU, ROUGE, and METEOR for robust rule translation.

Rule2Text refers to neural and neurosymbolic frameworks for converting knowledge graph (KG) rules into intelligible natural language explanations. These systems aim to bridge the gap between the formal, logical rules mined from KGs—often expressed in Horn clause or triple-based notation—and human-readable text, improving both interpretability and operational transparency for KG-driven applications. State-of-the-art Rule2Text frameworks integrate LLMs, prompt engineering, rule-type inference, and evaluation protocols to enable scalable, high-fidelity natural-language rule explanation, sometimes extending to domain adaptation through supervised fine-tuning on curated explanations (Shirvani-Mahdavi et al., 14 Aug 2025, Shirvani-Mahdavi et al., 31 Jul 2025). The general methodology combines rule extraction/mining, sophisticated prompt design, variable typing, chain-of-thought induction, LLM or symbolic generator orchestration, and both human and automatic evaluation pipelines for explanation correctness and fluency.

1. System Architecture and Rule Processing Pipeline

Rule2Text frameworks typically operate on KG rules mined via systems such as AMIE 3.5.1, producing rules of the form

$$B_1 \land B_2 \land \cdots \land B_n \;\Longrightarrow\; H$$

where each $B_i$ or $H$ is a triple $\mathtt{r(h, t)}$ over variables and/or constants. The core pipeline consists of the following stages:

1. Rule Acquisition: Logical rules are extracted from KGs using high-throughput rule miners (e.g., AMIE 3.5.1) with metrics such as support, head coverage, and confidence for rule selection.
2. Variable-Type Extraction: If the KG has explicit type schemas, relations are mapped to type assignments for rule variables; otherwise, the system samples valid rule instantiations and uses an LLM to infer probable types per variable. This is critical for accurately contextualizing rule elements in natural language.
3. Prompt Engineering: The prompt construction module generates one of several prompt styles for LLMs:
   • Always includes KG labeling syntax, rule body/head, (optionally) variable types, and an English-language instruction.
   • Supports several strategies: zero-shot, few-shot (with in-context exemplars), variable-type elaboration, and chain-of-thought (CoT) decomposition.
4. LLM Interface: Off-the-shelf LLMs (e.g., GPT-3.5 Turbo, GPT-4o mini, Gemini 2.0 Flash) serve as rule-to-text generators, initially prompted in various configurations; later, superior open-source models (e.g., Zephyr-7B-β) are fine-tuned on curated explanation corpora for scalable deployment.
5. Output Synthesis: Each input rule, enhanced as needed by variable type or explicit rule decomposition, is mapped to a succinct English explanation designed to capture the semantics of the formal rule.

This architecture modularly separates rule mining, type inference, prompt crafting, model scoring, and explanation synthesis, facilitating adaptation to different datasets, rule qualities, and types of target language output (Shirvani-Mahdavi et al., 14 Aug 2025).

2. Prompting Strategies for Explanation Generation

Four principal prompting techniques are evaluated for generating explanations from KG rules, each with distinct properties and observed effects:

• Zero-Shot Prompting: Presents the rule directly, requesting an English paraphrase:

  Given the following Horn rule: [B₁ ∧ ... ∧ Bₙ ⇒ H] produce a concise English explanation of what the rule expresses.

  No in-context examples are included. Yields baseline correctness and fluency.
• Few-Shot Prompting: Prepends two in-context rule→explanation pairs before presenting the new target rule. No statistically significant performance improvement over zero-shot is observed.
• Variable-Type Incorporation: Includes explicit type assignments for all variables; correctness increases by ~0.27 (on a 1–5 scale) over baseline. For example:

  Rule: {?a} {/time/event/instance_of_recurring_event} {World Series} ⇒ {World Series} {/sports/sports_championship/events} {?a} Variable types: ?a: /sports/sports_championship_event Please explain in English.

• Chain-of-Thought (CoT) Prompting: Decomposes generation into intermediate reasoning steps (variable listing and typing, atom paraphrasing, logical flow confirmation), culminating in a final sentence. This approach, particularly when combined with variable-type annotation, achieves the highest human-evaluated correctness and clarity; Gemini 2.0 Flash outperforms both GPT-4o mini and GPT-3.5 Turbo on these metrics (Shirvani-Mahdavi et al., 14 Aug 2025, Shirvani-Mahdavi et al., 31 Jul 2025).

3. Evaluation Methodologies

Evaluation within Rule2Text frameworks is multi-tiered, blending expert human annotation with LLM-based judging mechanisms:

• Human Evaluation: Annotators rate explanations on:
  • Correctness (Does the NL paraphrase include all rule elements in the correct logical order? 1–5 scale)
  • Clarity (How naturally is it phrased? 1–5 scale)
  • Hallucinations and Omissions (counts for extra/missed relations or entities)
  • Rule logicalness (1–3 scale for the inherent rule, not the explanation)

Each explanation is rated independently and adjudicated by majority or follow-up consensus (Shirvani-Mahdavi et al., 14 Aug 2025).

• LLM-as-a-Judge Framework: Inspired by Zheng et al. (2023), this module receives the rule, variable types, an instantiation, and the explanation. It assesses coverage and fidelity via structured yes/no sub-questions, then produces a graded score (1–5). Multiple passes and in-context anchoring examples enhance robustness. Empirical agreement with expert annotators is strong (Spearman’s $\rho = 0.69$, Krippendorff’s $\alpha = 0.59$) (Shirvani-Mahdavi et al., 14 Aug 2025).

This dual evaluation design supports both high-quality small-scale assessment (human) and scalable, reliable calibration (LLM-as-judge).

4. Experimental Results and Model Performance

Rule2Text frameworks have been validated on a range of Freebase-derived and biomedical datasets. Key results include:

Phase / Domain	Model	Correctness	Clarity	Perplexity	BLEU	ROUGE	METEOR
Phase 1 (zero/few-shot) FB15k	GPT-3.5 group	4.36	4.67	~36.1	—	—	—
Phase 2 (types) Freebase	—	3.94 → 4.21	4.12→4.19	—	—	—	—
Phase 3 (CoT + types)	Gemini 2.0 Flash	4.67	4.70	27.2	—	—	—
Zephyr FT (Freebase)	Zephyr-7B-β	—	—	—	0.71	0.33	0.66
Zephyr FT (BioKG)	Zephyr-7B-β	—	—	—	0.55	0.78	0.81

In the Biomedical (ogbl-biokg) domain, fine-tuning a modest open-source LLM (Zephyr-7B-β) using data augmented by LLM-as-judge and human-corrected explanations achieves substantial improvements: BLEU rises from 0.38 to 0.55, ROUGE from 0.02 to 0.78, METEOR from 0.36 to 0.81. These findings demonstrate that fine-tuning on domain-grounded explanations can yield large domain-adaptation gains, particularly in high-heterogeneity KGs (Shirvani-Mahdavi et al., 14 Aug 2025).

5. Type Inference and Rule Disambiguation

Variable-type inference is a critical module in the system, needed when KG schemas do not provide explicit types for variables. This module operates by sampling three valid triple instantiations per rule and prompting the LLM to infer the "most specific but generalizable KG type" per variable.

• The method yields high assignment accuracy (human-rated correctness 4.53/5).
• Augmenting natural-language prompts with these high-confidence variable-type annotations demonstrably increases explanation correctness and clarity in downstream LLM generation.

This approach is essential in scaling Rule2Text to KGs lacking explicit ontological typing, and is integrated end-to-end with both prompt construction and explanation generation (Shirvani-Mahdavi et al., 14 Aug 2025).

6. Data, Code, and Practical Implementation

All code and datasets referenced in recent Rule2Text work are released publicly at https://github.com/idirlab/KGRule2NL, including rule mining scripts, LLM prompt templates, variable-type extraction routines, evaluation code, and curated explanation corpora.

For practical deployment and extension:

• Rules are first mined via AMIE 3.5.1, filtered by head coverage, confidence, and maximum body atoms.
• Type inference is performed where needed, using a validated LLM-based instantiation sampling procedure.
• Prompt engineering follows best-performing strategies, with chain-of-thought and explicit typing as the default for highest correctness/fluency.
• Explanations are produced in batch using the best model available (Gemini 2.0 Flash or a fine-tuned Zephyr-7B-β), with LLM-as-judge providing scalable secondary quality assurance.
• Human-in-the-loop correction remains a best practice for gold-standard dataset curation and fine-tuning (Shirvani-Mahdavi et al., 14 Aug 2025).

7. Significance and Limitations

Rule2Text frameworks enable the translation of thousands of mined logical rules into clear, accurate natural language, significantly elevating the transparency and usability of knowledge-graph-based reasoning pipelines. The integration of type-aware prompt decomposition and chain-of-thought induction has proven effective, with evaluation metrics confirming both high absolute and relative model performance.

Limitations include persistent challenges in explaining complex or concatenated rules (e.g., rules with mediator nodes or more than two atoms), occasional hallucinations due to ambiguous predicate labels, and the dependency on well-typed rule instantiations for accurate type inference. Current research focuses on building larger gold-standard explanation corpora and on automated prompt/evaluation refinement to further improve output reliability, especially for KGs with non-standard or less interpretable relation labeling (Shirvani-Mahdavi et al., 14 Aug 2025, Shirvani-Mahdavi et al., 31 Jul 2025).