Logic-Preserving Contrastive Learning

• Logic-Preserving Contrastive Learning is a framework that preserves task-relevant logical relationships in dense retrieval by distinguishing answer-sufficient evidence from distractors.
• It employs a dual-encoder architecture with LLM-guided query rewriting to embed complex discourse structures into the query representation without extra inference resources.
• Hierarchical contrastive supervision stratifies negatives into fine-grained classes, boosting retrieval accuracy by emphasizing logical alignment over mere lexical similarity.

Logic-preserving contrastive learning denotes a family of contrastive training strategies in which the preserved invariant is not merely surface similarity, but some task-relevant semantic, relational, evidential, or structural property that should remain stable in representation space. In the most direct formulation, dense retrieval is trained so that answer-sufficient evidence outranks passages that are lexically or topically attractive but logically non-answering, even when queries are rewritten with complex discourse structure (Zhang et al., 1 Feb 2026). Related work broadens the preservation target to faithfulness and knowledge retention during alignment (Korkmaz et al., 25 May 2025), semantic attribute predicates under benign transformations (Nissani, 2023), admissible higher-order topology in cellular complexes (Qin et al., 27 May 2025), label dependencies under missing labels (Ma et al., 2022), latent component identity in Gaussian mixtures (Bansal et al., 2024), and task-relevant semantics selected by the inductive bias of the function class and optimization procedure (Saunshi et al., 2022).

1. Conceptual scope and meaning

The phrase has no single canonical meaning across the cited literature. Its most explicit retrieval-oriented definition appears in LORE, where the central failure mode is that dense retrievers overfit to surface overlap and retrieve passages that “look related” to a query while failing to satisfy the query’s logical requirements. LORE therefore frames preservation as maintaining alignment between the query’s intended meaning and evidential sufficiency under discourse-rich perturbations, rather than preserving only semantic or lexical proximity (Zhang et al., 1 Feb 2026).

This notion is narrower than formal symbolic logic and broader than ordinary semantic matching. LORE does not define symbolic operators for conjunction, disjunction, negation, comparison, temporal ordering, or causal graphs, and it does not impose a formal logical consistency regularizer. Instead, it operationalizes “logic-oriented” retrieval through Rhetorical Structure Theory (RST) guided query rewriting and through a distinction between chunks that are sufficient to answer the query and chunks that merely become attractive because of discourse-linked distractor material. The preserved object is therefore the ordering induced by answer sufficiency under compositional and discourse perturbation, not theorem-proving validity in a symbolic system (Zhang et al., 1 Feb 2026).

Other papers use closely related but not identical preservation targets. In debiasing, the preserved object is factual accuracy, knowledge retention, faithfulness, and intelligibility during alignment (Korkmaz et al., 25 May 2025). In self-supervised vision, it is the preservation of semantic attributes under identity-preserving transforms (Nissani, 2023). In topological deep learning, it is the admissibility of higher-order relational structure in cellular complexes (Qin et al., 27 May 2025). In multi-label learning, it is local low-rank and global structural properties of the label matrix under missing labels (Ma et al., 2022). In theory, it can be the Fisher-optimal latent subspace associated with same-component augmentations (Bansal et al., 2024), or, negatively, the paper may show that low contrastive loss and augmentation design alone do not determine which semantics survive unless the learner has the appropriate inductive bias (Saunshi et al., 2022).

2. Retrieval-centered realization: LORE

LORE is a post-pretraining contrastive fine-tuning method for dense retrievers. Its starting point is a labeled retrieval corpus containing a query $q$, candidate chunks $\{c_k\}$, and boolean labels $y_k \in \{\text{True}, \text{False}\}$. For each query, some false-labeled chunks are sampled as distractors. These chunks, together with the original query, are then given to an LLM that rewrites the query into a more natural form while embedding distractor content but preserving the original meaning. The rewriting is guided by one of eight discourse relations: Sequential, Transitional, Supplementary, Contrastive, Causal, Parallel, Hypothetical, and Explanatory (Zhang et al., 1 Feb 2026).

The rewrite induces a three-tier candidate partition for each query $q$ and candidate set $\mathcal{C}(q)$. The set $P$ contains chunks with $y_k=\text{True}$, namely answer-sufficient positive evidence. The set $N1$ contains false-labeled chunks used during query rewriting as distractors; these are the “hard false attractions” that appear relevant to the rewritten query but do not help answer it. The set $N2$ contains the remaining false-labeled chunks not used in the rewrite. The resulting training set is called CoEnTrain. If only part of the data is rewritten, the remaining examples keep their original queries and boolean labels and are mapped into the same three-tier schema for consistency (Zhang et al., 1 Feb 2026).

Architecturally, LORE uses a standard dual-encoder setup with separate query and document encoders, denoted $\mathcal{M}_q$ and $\{c_k\}$0. Queries and chunks are embedded in a shared vector space as

$\{c_k\}$1

with cosine similarity

$\{c_k\}$2

The document encoder is frozen, and only the query encoder is fine-tuned. This design is methodologically important because it teaches the query representation to encode logical structure without rebuilding the document side of the index. At retrieval time, the incoming query is encoded with the fine-tuned query encoder and ordinary nearest-neighbor search is performed over the existing document index, making the method index-compatible and free of any inference-time pre-retrieval analysis stage (Zhang et al., 1 Feb 2026).

The paper states that LORE “requires no external supervision, resources, or pre-retrieval analysis,” but it also makes clear that the training set is enriched offline with LLM-generated rewrites and discourse annotations. This indicates that the absence of external resources applies to inference-time analysis and external symbolic structures, not to a wholly annotation-free training pipeline (Zhang et al., 1 Feb 2026).

3. Hierarchical contrastive supervision and the preservation mechanism

LORE’s central design choice is to stratify negatives rather than collapse all non-positives into a single class. The adjusted similarities are defined in logit space as

$\{c_k\}$3

Here, $\{c_k\}$4 is the temperature, and $\{c_k\}$5 makes distractors $\{c_k\}$6 more competitive than ordinary negatives $\{c_k\}$7. Since adding $\{c_k\}$8 or $\{c_k\}$9 before exponentiation multiplies the corresponding denominator contributions by $y_k \in \{\text{True}, \text{False}\}$0 or $y_k \in \{\text{True}, \text{False}\}$1, the objective increases competitive pressure specifically from discourse-plausible but logically useless evidence (Zhang et al., 1 Feb 2026).

For each positive $y_k \in \{\text{True}, \text{False}\}$2, the positive probability is

$y_k \in \{\text{True}, \text{False}\}$3

and the query loss is

$y_k \in \{\text{True}, \text{False}\}$4

The objective therefore encodes the hierarchy

$y_k \in \{\text{True}, \text{False}\}$5

The “fine-grained” aspect is not a multi-level encoder or token-level contrastive loss, but the partition of negatives into semantically different subclasses. Positives are also treated individually rather than pooled against one another in the denominator; the authors state that this avoids “dilution” (Zhang et al., 1 Feb 2026).

This hierarchy is the paper’s main logic-preserving mechanism. The learned query embedding is pushed to remain close to chunks that are sufficient to answer the query, while especially plausible distractors are pushed away more strongly than ordinary negatives. The preserved relation is therefore graded evidential relevance under logically or discourse-augmented queries, not binary topical similarity (Zhang et al., 1 Feb 2026).

The reported implementation details are deliberately lightweight. Datasets are HotpotQA, MS MARCO, and MuSiQue. Fine-grained annotations via query rewriting are applied to the MS MARCO training and validation data, while HotpotQA and MuSiQue remain unannotated during training and are later used to assess transfer. The embedding backbones are Qwen/Qwen3-Embedding-0.6B and BAAI/bge-m3. Training is performed on a single A800 GPU with learning rate $y_k \in \{\text{True}, \text{False}\}$6, temperature $y_k \in \{\text{True}, \text{False}\}$7, $y_k \in \{\text{True}, \text{False}\}$8, $y_k \in \{\text{True}, \text{False}\}$9, batch size 32, for 1 epoch, with three random seeds $q$0 and mean/std reported (Zhang et al., 1 Feb 2026).

4. Adjacent forms of structure preservation

The broader literature shows that contrastive preservation can target several different kinds of structure.

Domain	Preservation target	Representative mechanism
Alignment and debiasing (Korkmaz et al., 25 May 2025)	Truthfulness, knowledge retention, faithfulness, intelligibility	Positive and negative examples with joint LM and contrastive loss
Semantic attributes (Nissani, 2023)	Attribute predicates under identity-preserving transforms	SimCLR-style positives from benign transformations
Cellular TDL (Qin et al., 27 May 2025)	Admissible higher-order topology	Parameter perturbation instead of structural corruption
Multi-label learning (Ma et al., 2022)	Local low-rank and global label structure	Nuclear-norm contrastive embedding with label correction
GMM theory (Bansal et al., 2024)	Fisher-optimal latent structure under same-component pairings	Population InfoNCE with augmentation-enabled mixtures
Inductive-bias theory (Saunshi et al., 2022)	Task-relevant semantics selected by hypothesis class	Function-class-aware transfer analysis

In debiasing and alignment, the preserved object is not formal logic but a cluster of quality properties closely related to reasoning-preserving behavior: truthfulness, knowledge retention, faithfulness in summarization, and coherent non-abstaining generation. The method constructs positive examples by backtranslation through German, French, and Spanish, and constructs negatives through adversarial toxic generation, low-confidence generation, entity swapping, and masked regeneration. A token-level projection head and named-entity-focused pooling make the contrastive objective especially sensitive to semantically critical spans, while the joint loss

$q$1

preserves the base language modeling objective alongside contrastive alignment (Korkmaz et al., 25 May 2025).

In attribute-oriented self-supervision, the preservation hypothesis is that identity-preserving transformations may also preserve semantically meaningful attributes. The paper’s conjecture is that if the transformations preserve object identity and the identity of intrinsic attributes, then the learned representation should support not only class recognition but also the presence or absence decision of attribute predicates. Its empirical proxy is “hyper-separability,” namely linear recoverability of class-union or attribute-like super-classes from contrastive representations trained with elastic distortion (Nissani, 2023).

In topological deep learning on cellular complexes, preservation is explicitly structural. CellCLAT argues that node dropping, edge perturbation, or subgraph sampling can produce invalid cellular objects because they break attaching maps and higher-order incidence constraints. It therefore perturbs encoder parameters rather than the cellular structure itself and combines this with 2-cell trimming through a bi-level meta-learning scheduler. The preserved object is the legality of the cellular complex during augmentation, while task-irrelevant higher-order cells are suppressed only at the representation and gradient-contribution level (Qin et al., 27 May 2025).

In multi-label learning with missing labels, CLML replaces pairwise positives and negatives with a rank-structured objective,

$q$2

so that embeddings of samples sharing a semantic label form low-rank subspaces while the full batch remains globally discriminative. Label correction converts some apparent negatives into positives when model confidence exceeds a threshold, thereby addressing the false-negative problem induced by missing labels (Ma et al., 2022).

Two theoretical papers sharpen the general lesson. In Gaussian mixture models, augmentations are defined as another draw from the same latent mixture component. Under that construction, population InfoNCE recovers the Fisher subspace

$q$3

showing that when positive pairs preserve latent identity, contrastive learning recovers the corresponding discriminative subspace rather than merely high-variance directions (Bansal et al., 2024). By contrast, the inductive-bias analysis shows that low contrastive loss and augmentation properties alone do not determine downstream semantics: there can exist representations with the same or similar contrastive loss and radically different downstream behavior, so the function class and optimization procedure are decisive in selecting which structure is actually preserved (Saunshi et al., 2022).

5. Empirical evidence

LORE’s empirical evidence is strongest on disturbed, logic-heavy retrieval tests, where performance is measured both by positive recall at top-$q$4 and by distractor recall for $q$5, where lower is better. On Qwen for HotpotQA disturbed @3, positive recall rises from 33.76 for the raw model and 59.71 for InfoNCE to 65.45 for LORE, while distractor recall drops from 70.03 and 50.51 to 23.81. On MuSiQue disturbed @3, positive recall goes from 39.55 to 53.65 to 59.28, and distractor recall from 83.88 to 70.36 to 36.87. On MS MARCO disturbed @3, positive recall goes from 46.98 to 51.63 to 69.30, with distractor recall dropping from 84.25 and 82.03 to 20.15. For bge-m3 on MS MARCO disturbed @5, positive recall rises from 75.20 to 78.17 to 86.10, while distractor recall falls from 93.51 and 92.31 to 44.89. The ablation pattern is also informative: on raw datasets, LORE is often similar to InfoNCE and sometimes slightly lower, whereas on disturbed datasets it consistently yields better positive recall and dramatically lower distractor recall (Zhang et al., 1 Feb 2026).

The qualitative diagnostics support the same interpretation. Figure 1 tracks score evolution for the three chunk categories: early in training, distractors $q$6 receive high scores because of surface overlap; later, $q$7 scores decrease and converge toward $q$8, while positive $q$9 scores remain stably higher. This is consistent with the intended separation of logical relevance from superficial similarity (Zhang et al., 1 Feb 2026).

In alignment and debiasing, the reported result is that the contrastive framework is the first method in that evaluation to consistently improve both toxicity reduction and faithfulness preservation simultaneously. On Reddit TL;DR, GPT2 shows toxicity improving by $\mathcal{C}(q)$0 and faithfulness improving by $\mathcal{C}(q)$1; Phi2 shows toxicity improving by $\mathcal{C}(q)$2 and faithfulness improving by $\mathcal{C}(q)$3; Llama2-7B shows toxicity improving by $\mathcal{C}(q)$4 and faithfulness improving by $\mathcal{C}(q)$5. The same paper also reports severe failure modes for several baselines, including 94.9% repetitive text for Sentence Debiasing on GPT2 and 100% empty responses for Instructive Debiasing on Phi2 (Korkmaz et al., 25 May 2025).

CellCLAT reports the best unsupervised classification accuracy on all six TU benchmark datasets: NCI1 $\mathcal{C}(q)$6, PROTEINS $\mathcal{C}(q)$7, MUTAG $\mathcal{C}(q)$8, NCI109 $\mathcal{C}(q)$9, IMDB-B $P$0, and IMDB-M $P$1, with average rank $P$2. In semi-supervised learning, it also obtains the best average rank $P$3. The ablation is especially relevant: CellCL improves over SimGRACE, trimming 0-cells or 1-cells hurts, and 2-CellTrim improves most, which supports the distinction between preserving all topology during augmentation and suppressing redundant higher-order topology during pooling (Qin et al., 27 May 2025).

CLML reports gains of about 1.2% on MSCOCO, 1.6% on VOC, and 1.3% on NUS-WIDE, and SPLC + CLML on COCO with 75% labels retained reaches 80.4 mAP, surpassing BCE with full labels at 80.3. In the semantic-attribute study, 93% of MNIST super-classes and 98% of sampled balanced EMNIST super-classes are better than multiclass error under the contrastive model, whereas 77% and 68%, respectively, are worse than multiclass error under the supervised baseline. These results do not establish formal logic preservation, but they do show that preserving semantic structure in the contrastive objective can make nontrivial predicates more recoverable (Ma et al., 2022, Nissani, 2023).

6. Limits, misconceptions, and open directions

A recurring misconception is to treat these methods as preserving formal logic in the strict symbolic sense. The cited literature does not support that interpretation. LORE is explicit that its notion of “logic-oriented” is broad and discourse-centered rather than formal-logical; it uses RST-style perturbations and answer sufficiency rather than symbolic operators or explicit reasoning graphs (Zhang et al., 1 Feb 2026). The alignment paper is similarly careful: it is relevant to a broader notion of logic-preserving alignment because it preserves factuality, knowledge, and faithfulness, but it does not evaluate theorem proving, deductive consistency, or chain-of-thought faithfulness (Korkmaz et al., 25 May 2025).

Another limitation is evaluation scope. LORE’s abstract states that it “consistently improves retrieval utility and downstream generation,” but the main body described in the provided text reports retrieval results only and gives no explicit generation metrics such as EM, F1, or answer accuracy. The paper also notes dependence on the quality and diversity of the constructed logical/discourse rewrites and leaves broader generalization to future work (Zhang et al., 1 Feb 2026). The debiasing paper notes that named-entity-focused pooling may be too narrow for tasks whose key information lies in complex relationships rather than specific entities (Korkmaz et al., 25 May 2025). The attribute paper relies on super-class membership as a proxy for attributes, which makes its evidence suggestive rather than direct (Nissani, 2023).

The theory papers add two more cautions. The Gaussian-mixture analysis shows that contrastive learning can recover a Fisher-optimal subspace when positive pairs preserve latent mixture identity, but that result concerns latent class-conditioned semantics, not symbolic or compositional logic (Bansal et al., 2024). The inductive-bias analysis shows that contrastive loss and augmentation design alone are insufficient to explain which semantics survive, because different architectures and algorithms can achieve similar contrastive losses yet differ sharply in downstream behavior. For any program of logic-preserving contrastive learning, this implies that the hypothesis class and optimization procedure must embody the desired structure rather than relying on the loss alone (Saunshi et al., 2022).

The literature nevertheless suggests a coherent research trajectory. LORE indicates that future retrievers may benefit from supervision built around failure-inducing logical confounders, graded negative types, and query-side adaptation when index compatibility matters (Zhang et al., 1 Feb 2026). The alignment work suggests replacing toxicity-centered positives and negatives with reasoning-centered positives and negatives such as faithful chain-of-thoughts, contradiction-free explanations, and subtly invalid but fluent reasoning traces, though that extension is proposed rather than implemented (Korkmaz et al., 25 May 2025). The attribute and GMM papers suggest a common principle: augmentations should preserve the predicates or latent identities one wants the representation to encode (Nissani, 2023, Bansal et al., 2024). The inductive-bias results further suggest that a fully explicit logic-preserving contrastive framework would likely require structure-aware encoders, relation-preserving transformations, and optimization biases aligned with compositional or symbolic constraints rather than generic instance discrimination (Saunshi et al., 2022).