DualCSE: Dual Embedding for Sentences

• DualCSE is a dual-embedding framework that generates two parallel representations—explicit for literal meaning and implicit for contextual nuances.
• It employs both cross-encoder and bi-encoder architectures to separately decode semantic layers for improved sentence understanding in NLP tasks.
• The model uses a multi-term contrastive loss to enforce inter- and intra-sentence semantic alignment, boosting performance in entailment and text classification.

DualCSE refers to a dual-embedding approach for sentence representation that enables concurrent encoding of explicit and implicit semantics within the same shared representation space. Recent advances in sentence embedding methods have increased the effectiveness of natural language processing applications, but conventional systems are often restricted to single-vector representations that primarily capture explicit meaning. DualCSE remedies this limitation by assigning each sentence two parallel embeddings: one for explicit (literal) semantics and one for implicit (latent) semantics, thereby facilitating more nuanced semantic analysis in tasks such as information retrieval and text classification (Oda et al., 10 Oct 2025).

1. Dual Embedding Framework

DualCSE distinguishes itself by mapping each sentence to two distinct embedding vectors:

• Explicit semantic embedding $r$: Encodes the overt, literal interpretation of the sentence.
• Implicit semantic embedding $u$: Encodes underlying, context-dependent or inferred semantics.

These embeddings coexist in a shared vector space. For practical instantiations, a sentence such as “She conquered his heart.” yields $r$ for the direct, literal meaning (“defeated in battle”) and $u$ for the implied meaning (“won his love”), allowing downstream models to select which semantic layer to leverage.

2. Architecture and Representation Assignment

DualCSE is implemented in two architecture variants:

• Cross-encoder variant: Employs a single transformer encoder; distinct input prompts indicate which semantic perspective to encode (e.g., use “[CLS] s [SEP] explicit” and “[CLS] s [SEP] implicit” to obtain $r$ and $u$, respectively).
• Bi-encoder variant: Utilizes two independent encoder models, each dedicated to either explicit or implicit semantic extraction for maximal separation.

In both cases, the [CLS] token output from the transformer's final hidden layer is used as the embedding.

3. Contrastive Learning Objective

The dual representations are learned via a multi-term contrastive loss function designed to enforce inter- and intra-sentence semantic separation and alignment:

• Inter-sentence relations: For a premise $s_i$, the explicit embedding $r_i$ is pulled toward the explicit entailment hypothesis embedding $r_i^{+1}$; similarly, $u_i$ is pulled toward implicit entailment embedding $r_i^{+2}$; contradiction embeddings $r_i^-$ are pushed apart.
• Intra-sentence relations: The explicit and implicit embeddings for the same sentence $(r_i, u_i)$ are explicitly decorrelated to enforce separation; however, for less ambiguous hypotheses, explicit and implicit representations are encouraged to be similar.

The key component is the similarity-weighted exponential:

$$v(\mathbf{h}_1, \mathbf{h}_2) = \exp(\text{sim}(\mathbf{h}_1, \mathbf{h}_2)/\tau)$$

where sim is cosine similarity and $\tau$ is a temperature parameter.

The overall instance loss (see Eq. (2) from the paper) is:

$$\begin{align*} l_i =& -\log\left[\frac{v(r_i, r_i^{+1})}{\sum_{j}(v(r_i, r_j^{+1}) + v(r_i, r_j^-) + v(r_i, u_j))}\right] -\log\left[\frac{v(u_i, r_i^{+2})}{\sum_{j}(v(u_i, r_j^{+2}) + v(u_i, r_j^-) + v(u_i, r_j))}\right] \ &-\log\left[\frac{v(r_i^{+1}, u_i^{+1})}{\sum_j v(r_i^{+1}, u_j^{+1})}\right] -\log\left[\frac{v(r_i^{+2}, u_i^{+2})}{\sum_j v(r_i^{+2}, u_j^{+2})}\right] -\log\left[\frac{v(r_i^{-}, u_i^{-})}{\sum_j v(r_i^{-}, u_j^{-})}\right] \end{align*}$$

This structure ensures explicit and implicit meanings are encoded distinctly but can relate appropriately to their respective hypotheses or semantic categories.

4. Training Data and Semantic Differentiation

DualCSE is trained using natural language inference datasets (e.g., INLI), which provide explicit and implicit entailment hypotheses. This allows the model to learn how to associate both overt and inferred sentence meanings—capturing, for instance, both direct and pragmatic interpretations present in human communication.

Explicit embeddings typically align to hypothesis types labeled as “explicit-entailment,” while implicit embeddings are aligned to “implied-entailment,” reflecting human judgments of overt and subtle semantic relationships, respectively.

5. Downstream Tasks and Empirical Evaluation

DualCSE supports several tasks that exploit its dual embeddings:

• Recognizing Textual Entailment (RTE): Entailment is predicted when either explicit–explicit or implicit–explicit embedding pairs for premise and hypothesis surpass a cosine similarity threshold $\gamma$:

$$\text{Entailment if } \max[\cos(r_1, r_2), \cos(u_1, r_2)] > \gamma$$

• Estimating Implicitness Score (EIS): A sentence’s implicitness is quantified as:

$\operatorname{imp}(s) = 1 - \cos(r, u)$

This score allows one to compare the degree of implicitness between sentences or select for non-literal meaning in downstream analysis.

• Information Retrieval: By constructing separate retrieval indices for explicit and implicit embeddings, queries can be resolved according to required semantic depth (literal factoid retrieval versus pragmatic, sentiment-oriented answers).

Experimental results on the INLI dataset demonstrate that DualCSE outperforms baseline sentence representation methods (e.g., SimCSE trained on SNLI+MNLI) in tasks involving both explicit and implicit entailment, with notable improvements in handling implicit semantics.

6. Semantic Control and Application Implications

The explicit separation of literal and implied meanings affords fine-grained semantic control in NLP applications:

• Information Retrieval: Systems can select which embedding to query, improving relevance for direct questions (explicit) or sentiment/metaphorical queries (implicit).
• Text Classification: Detecting figurative language, irony, or pragmatic intent is enhanced by leveraging implicit embeddings.
• Implicitness Estimation: The implicitness score facilitates analysis of non-literal, obfuscated, or sentiment-laden text (e.g., in opinion mining or hate speech detection).

This division between $r$ and $u$ is pivotal for tasks that demand understanding both propositional content and pragmatic subtext.

7. Model Variants and Operational Considerations

DualCSE can be instantiated as either a cross-encoder (single model, prompt-driven semantic switch) or bi-encoder (parallel encoding, separation at architectural level). The cross-encoder variant leverages prompt engineering to induce semantic differentiation, while the bi-encoder affords more computational parallelism at test time, at the expense of increased parameter count.

Both architectures utilize the [CLS] token for embedding output and operate in a shared representation space, ensuring interoperability for retrieval and similarity tasks.

Summary Table: Dual Embedding Assignment

Embedding Type	Representation Symbol	Semantic Content
Explicit	$r$	Literal/Overset
Implicit	$u$	Latent/Subtextual

The DualCSE framework provides a principled method to capture and control dual layers of sentence meaning, validated through contrastive training and empirical evaluation on inference tasks. Its modularity and effectiveness at separating literal and implicit semantics position it as a foundation for sophisticated language understanding and retrieval systems (Oda et al., 10 Oct 2025).