Logic-Preserving Distillation

• The paper demonstrates that logic-preserving distillation retains modular symbolic reasoning by decomposing a teacher’s strategy into verifiable function bases.
• It employs stagewise cross-entropy and dense retrieval techniques to achieve 100% rule compliance and more robust decision-making than standard KD.
• Experimental results validate its efficacy in both language models and quantum protocols, highlighting scalable, interpretable, and fault-tolerant architectures.

Logic-preserving distillation refers to a class of protocols and frameworks in both artificial intelligence and quantum information processing that aim to transfer or refine “logical” features—from modular reasoning pipelines in LLMs to error-corrected logical entanglement in quantum systems—while maintaining explicit logical or symbolic structure throughout the distillation procedure. In contrast to standard output-matching distillation, logic-preserving approaches are expressly designed so that the distilled student or purified state mirrors not merely the outputs, but the compositional and interpretable logical mechanisms or encodings of the teacher or parent system.

1. Formalizing Logic-Preserving Distillation in LLMs

Logic Distillation (LD) addresses the challenge of equipping small LLMs (S-LLMs) with the symbolic/logical reasoning competencies observed in LLMs (L-LLMs), especially in interactive planning and decision-making contexts. The key insight is that while traditional knowledge distillation (KD) protocols train S-LLMs to mimic outputs, this practice fails to transfer the modular logical pipelines underpinning the teacher’s decisions, resulting in brittle student behavior and rule violations (Chen et al., 2024).

Mathematically, let $x$ represent the combination of rule set and current state, $p_{\theta_L}$ denote the L-LLM, and $p_{\theta_S}$ the S-LLM student. The aim is to construct $p_{\theta_S}$ such that it, given $x$, yields an action sequence $O$ satisfying all rules and achieving the intended task. Rather than direct trajectory supervision, LD factors the teacher’s solution into intermediate code-level functions, then trains the student to orchestrate these atomic operations in the correct sequence—preserving the symbolic logic pipeline.

2. Construction and Utilization of Function Bases

The LD framework builds a “function base” ($D_f$), instantiated by prompting the L-LLM to emit:

• A set of code-style, self-contained functions $f_i$ that each encapsulate a distinct logical sub-operation,
• An associated “user manual” $u_i$ for each $f_i$, detailing signature, usage, intent, and invocation context.

Let $y = (y_1, y_2, \dots)$ be the generated token sequence representing all function- and manual-defining output. The decomposition is modeled as:

$$p_{\theta_L}(f,u \mid x) = \prod_{i=1}^{|y|} p_{\theta_L}(y_i \mid x, y_{<i})$$

The function base is thus $D_f = \{(f_i, u_i)\}_{i=1}^N$.

During inference, the S-LLM does not operate monolithically; instead, it acts as an orchestrator, selecting and invoking functions from $D_f$ at each stage. This enforces preservation of the original system’s logical compositionality.

3. Training, Function Retrieval, and Invocation Mechanisms

Training in LD focuses on function selection at each decision stage. The S-LLM must select the correct function $f_j$ from a candidate set $D_{cand}^{(j)}$ based on $x$, manuals $u$, and current context. The principal loss is the stagewise cross-entropy:

$$L_{sel} = -\sum_{j=1}^J \log p_{\theta_S}(f_j^* \mid x, u, D_{cand}^{(j)})$$

where $f_j^*$ is the supervised function at stage $j$.

For scalability, retrieval is handled by a dense retriever $p_{\theta_R}$ with bi-encoder architecture:

$$p_{\theta_R}(f \mid x) \propto \exp(d(f)^\top q(x))$$

$d(f)$ encodes a function’s code and manual; $q(x)$ encodes the instruction and state. At each stage, the retriever ranks $D_f$ to yield $K$ most pertinent candidates:

$D_{cand}^{(j)} = \textrm{argtop}_K_{f \in D_f} d(f)^\top q(x)$

Optionally, the retriever is trained using the InfoNCE contrastive loss:

$$L_{R} = -\sum_{(x,f^+)} \log \frac{\exp(d(f^+)^\top q(x) / \tau)}{\sum_{f'\in D_f} \exp(d(f')^\top q(x) / \tau)}$$

During inference, for each $f \in D_{cand}^{(j)}$, the S-LLM computes a score and applies softmax:

$$p_{\theta_S}(f \mid x, u, D_{cand}^{(j)}) = \text{softmax}(\text{score}_{\theta_S}(x, u, f))$$

The selected $f_j$ is invoked, its output feeds the next stage, and the process iterates for $J$ steps, ultimately producing the output $O$.

4. Logical Structure Preservation and Interpretability

Logic is preserved throughout the process by:

• Fixing the set of admissible operations (functions) by L-LLM during function-base construction;
• Constraining the student to select among these verified primitives via explicit supervision;
• Guaranteeing each function’s semantic and correctness properties by formal documentation (manuals) and code-based implementation.

By structuring planning and decision-making as a modular pipeline of verifiable reasoning steps, the distilled student model inherits robust generalization and rule compliance. Cross-entropy over function indices (log $K$) is a lower-entropy, more stable alternative to standard full-vocabulary losses.

5. Experimental Evaluation and Rule Compliance

Empirical results on interactive planning domains, such as the Pursuit Game and its emergency (restricted zone) variant, provide quantitative evidence for logic-preserving distillation’s efficacy.

Model	Success	Fail w/o Violation	Fail w/ Violation	Avg. Steps (succ.)
S-LLM	0%	0%	100%	—
L-LLM	96%	4%	0%	14.22
KD (LoRA)	88.5%	10.5%	1%	15.23
LD	100%	0%	0%	13.26

In both standard and emergency settings, LD achieves perfect rule compliance and expedient solutions, improving over baseline S-LLM and outperforming knowledge distillation which only matches outputs (Chen et al., 2024).

6. Logic-Preserving Distillation in Quantum Information

In quantum information, logic-preserving distillation refers to protocols designed to purify logical-qubit entanglement, maintaining logical encoding throughout error correction. Using cross-Kerr nonlinearity and quantum nondemolition (QND) parity checks, the protocol distinguishes, preserves, and corrects logic-level Bell states by:

• Keeping logic encoding intact across rounds;
• Correcting logic bit-flip, phase-flip, and physical bit-flip errors;
• Realizing deterministic physical error correction and probabilistic logic-Bell distillation, with fidelity boosts of $F' = F^2/(F^2 + (1-F)^2)$ and success probabilities $P_s = [F^2 + (1-F)^2]/4$ per round (Sheng et al., 2016).

This methodology is crucial for fault-tolerant long-distance entanglement distribution.

7. Implications and Resource Challenges

Logic-preserving distillation protocols result in models and entangled states that are both interpretable and robust. In LLM settings, they produce lightweight, rule-compliant agents capable of generalizing to new instructions. In quantum communication, they ensure that logical-qubit entanglement fidelity can be boosted systematically while preserving the logical code.

Resource challenges arise in both domains: in LD, function base construction and retrieval may grow with domain complexity; in quantum logic-qubit distillation, resource consumption scales exponentially with target fidelity, and practical success rates are impacted by detection inefficiencies in QND gates and limitations in physical cross-Kerr strengths.

A plausible implication is that logic-preserving protocols may underpin scalable, robust, and interpretable architectures in future AI and quantum systems, provided limitations in resources and hardware are systematically addressed (Chen et al., 2024, Sheng et al., 2016).