Sequence-Cross Perspective Contrastive Learning

• Sequence-Cross Perspective Contrastive Learning is a paradigm that aligns multiple sequential views to enforce semantic consistency across local, global, cross-lingual, and cross-modal representations.
• The approach leverages InfoNCE-style objectives with view-specific projectors and pooled embeddings to optimize both fine-grained and holistic sequence semantics.
• Empirical results demonstrate enhanced performance in unsupervised learning, multilingual pretraining, protein modeling, and sequential recommendation tasks.

Sequence-Cross Perspective Contrastive Learning is a contrastive learning paradigm in which a sequence is represented through complementary perspectives and the training objective explicitly aligns those perspectives in a shared or coordinated representation space. The perspectives may be local and global views of the same sentence, sequence-level and token-level views across languages, sequence and structure views of proteins, cross-subsequence views in recommendation, or spatial and temporal views of mobility traces. Across these settings, the common principle is that a useful sequence representation should remain semantically consistent across distinct but related views, while mismatched views are repelled by contrastive objectives (Wu et al., 2023, Zhang et al., 2023, Ranganath et al., 6 Mar 2026, Gong et al., 2024).

1. Conceptual definition and scope

In the literature, the term does not denote a single standardized architecture. Rather, it denotes a family of methods that contrast multiple perspectives derived from one underlying sequential object. In HiCL, the two perspectives are local segment-level and global sequence-level relationships within one sentence (Wu et al., 2023). In VECO 2.0, they are sequence-to-sequence and token-to-token cross-lingual alignments (Zhang et al., 2023). In ProtAlign and CCPL, the paired perspectives are protein sequence and protein structure (Ranganath et al., 6 Mar 2026, Zheng et al., 2023). In STCCR, they are “spatial topic” and “temporal intention” views of a check-in sequence (Gong et al., 2024). In recommender systems, the perspectives include augmented sequence views, cross-user subsequences sharing a target item, same-target and similar-sequence collaborative views, source/target/mixed cross-domain sequences, and soft versus hard denoising signals (Xie et al., 2020, Qin et al., 2023, Wang, 12 May 2026, Ma et al., 2023, Zhu et al., 2024).

The objectives are usually symmetric or jointly optimized. A common pattern is an InfoNCE-style formulation with in-batch negatives, cosine or dot-product similarity, and either no projection head or lightweight view-specific projectors. The notion of “perspective” therefore spans both granularity and modality: sentence segments versus whole sentences, tokens versus sentences, sequence versus structure, and sequence versus graph or 3D geometry (Wu et al., 2023, Zhang et al., 2023, Xu et al., 28 May 2026).

A central consequence is that the paradigm is broader than stochastic augmentation. Some methods construct views by dropout alone, as in SimCSE-style sentence learning or CL4SRec-style recommendation (Wu et al., 2023, Xie et al., 2020). Others reject heuristic perturbations as semantically unstable and instead mine views from natural sequence structure, cross-user semantics, collaborative contexts, or physically paired modalities (Qin et al., 2023, Cui et al., 6 Mar 2025, Wang, 12 May 2026, Ranganath et al., 6 Mar 2026).

2. Hierarchical and multi-granularity learning within and around sequences

HiCL provides a canonical within-sequence formulation. Given a batch of sequences, each sequence is sliced into non-overlapping token segments with fixed slicing length $L$, each segment is encoded by the same Transformer encoder $E_\theta$, and the segment embeddings are aggregated into a sequence embedding by length-weighted average pooling,

$$h_i = \sum_{j=1}^{l_i} w_{i,j} h_{i,j}, \qquad w_{i,j} = \frac{|seg_{i,j}|}{\sum_{k=1}^{l_i} |seg_{i,k}|}.$$

HiCL then optimizes a local contrastive loss over segment pairs and a global contrastive loss over sequence pairs, using cosine similarity $s(a,b)=a^\top b / (\|a\| \|b\|)$, temperature $\tau=0.05$, and the combined objective

$$\mathcal{L} = \alpha\,\mathcal{L}_{\text{local}} + (1-\alpha)\,\mathcal{L}_{\text{global}}.$$

Local positives are the same segment encoded twice with different dropout masks; local negatives are segments from other sequences in the batch; segments from the same sequence are treated as neither positives nor negatives. Global positives are different dropout views of the full sequence representation; global negatives are in-batch sequence embeddings. Under SNCSE-style training, negated sentences are used as “soft” negatives at both segment and sequence levels (Wu et al., 2023).

The method operationalizes “cross perspectives” as alignment between fine-grained and whole-sequence semantics. Its rationale is explicit: local perspective enforces invariance and discriminability at the granular level, while global perspective maintains coherence and semantic completeness. The ablations are equally explicit. Weighted segment-length pooling consistently outperformed unweighted pooling; treating same-sequence segments as positives or negatives degraded performance; very short $L$ harmed performance; $L \approx 32$ offered a strong balance of accuracy and efficiency; both extremes $\alpha=0$ and $\alpha=1$ underperformed; and the variant HiCLv2, which added an explicit entailment objective $E_\theta$0 to force $E_\theta$1 close to its segments $E_\theta$2, did not help and sometimes hurt (Wu et al., 2023).

HiCL also couples representation quality to computational structure. Because Transformers have quadratic complexity in input length, segment encoding plus aggregation reduces training-time cost. With truncation size 512, the reported training time on RoBERTa-large was 152 minutes versus SimCSE’s 354.5 minutes, and the method supported larger effective batch sizes without OOM. On seven STS tasks, HiCL enhanced prior SOTA SNCSE with average $E_\theta$3 for BERT-large and $E_\theta$4 for RoBERTa-large; SNCSE-RoBERTa-large increased from 81.35 to 81.79 average Spearman’s correlation (Wu et al., 2023).

A closely related but cross-lingual multi-granularity formulation appears in VECO 2.0. There, sequence-level alignment maximizes similarity of parallel sentences and repels non-parallel sentences in-batch, while token-level alignment brings synonymous or translated tokens close using MUSE dictionary pairs mined within bilingual instances. On monolingual batches, $E_\theta$5; on bilingual batches, $E_\theta$6, with effectively $E_\theta$7. The method thereby extends the same principle of complementary perspectives from segment/sequence structure to sentence/token structure across languages (Zhang et al., 2023).

3. Cross-modal and cross-lingual instantiations

When the perspectives correspond to different modalities rather than different granularities, the same contrastive logic is retained but the encoders become modality-specific. ProtAlign uses ESM2 as the sequence encoder $E_\theta$8, Protein-MPNN as the structure encoder $E_\theta$9, and modality-specific single-query multi-head attention pooling heads with $$h_i = \sum_{j=1}^{l_i} w_{i,j} h_{i,j}, \qquad w_{i,j} = \frac{|seg_{i,j}|}{\sum_{k=1}^{l_i} |seg_{i,k}|}.$$0 heads followed by LayerNorm to map per-token outputs into a shared $$h_i = \sum_{j=1}^{l_i} w_{i,j} h_{i,j}, \qquad w_{i,j} = \frac{|seg_{i,j}|}{\sum_{k=1}^{l_i} |seg_{i,k}|}.$$1-dimensional embedding space. It trains with a symmetric CLIP-style InfoNCE loss over matched sequence–structure pairs and in-batch negatives, with $$h_i = \sum_{j=1}^{l_i} w_{i,j} h_{i,j}, \qquad w_{i,j} = \frac{|seg_{i,j}|}{\sum_{k=1}^{l_i} |seg_{i,k}|}.$$2 yielding the best retrieval performance. On the test set, the reported retrieval performance was Recall@1 = 42.7% and Recall@5 = 99.1% for CLIP, versus Recall@1 = 40.0% and Recall@5 = 97.6% for SigLIP with $$h_i = \sum_{j=1}^{l_i} w_{i,j} h_{i,j}, \qquad w_{i,j} = \frac{|seg_{i,j}|}{\sum_{k=1}^{l_i} |seg_{i,k}|}.$$3 (Ranganath et al., 6 Mar 2026).

CCPL also aligns protein sequence and structure, but with a frozen ESM-2 base teacher on the sequence side and a GVP-GNN plus Transformer student on the structure side. Its cross-modal contrastive objective is a symmetric CLIP-style cross-entropy loss over matched sequence–structure pairs, and it adds a self-supervised structural constraint through C$$h_i = \sum_{j=1}^{l_i} w_{i,j} h_{i,j}, \qquad w_{i,j} = \frac{|seg_{i,j}|}{\sum_{k=1}^{l_i} |seg_{i,k}|}.$$4-based contact-map prediction. The sequence–structure alignment is not only an internal pretraining signal: when finetuned for inverse folding, the residue-level pretrained model Design$$h_i = \sum_{j=1}^{l_i} w_{i,j} h_{i,j}, \qquad w_{i,j} = \frac{|seg_{i,j}|}{\sum_{k=1}^{l_i} |seg_{i,k}|}.$$5 achieved perplexities of 4.48, 3.76, and 3.28 and recoveries of 50.8%, 55.8%, and 60.3% on CATH, Ts50, and Ts500, respectively (Zheng et al., 2023).

TriMod-DTI extends the same idea from two to three modalities. For each drug and protein, it encodes sequence (1D), graph (2D), and 3D structure, then applies pairwise contrastive losses across the three modality pairs. The total training loss is

$$h_i = \sum_{j=1}^{l_i} w_{i,j} h_{i,j}, \qquad w_{i,j} = \frac{|seg_{i,j}|}{\sum_{k=1}^{l_i} |seg_{i,k}|}.$$6

where $$h_i = \sum_{j=1}^{l_i} w_{i,j} h_{i,j}, \qquad w_{i,j} = \frac{|seg_{i,j}|}{\sum_{k=1}^{l_i} |seg_{i,k}|}.$$7 and $$h_i = \sum_{j=1}^{l_i} w_{i,j} h_{i,j}, \qquad w_{i,j} = \frac{|seg_{i,j}|}{\sum_{k=1}^{l_i} |seg_{i,k}|}.$$8 average contrast over $$h_i = \sum_{j=1}^{l_i} w_{i,j} h_{i,j}, \qquad w_{i,j} = \frac{|seg_{i,j}|}{\sum_{k=1}^{l_i} |seg_{i,k}|}.$$9, $s(a,b)=a^\top b / (\|a\| \|b\|)$0, and $s(a,b)=a^\top b / (\|a\| \|b\|)$1. On Human, GPCR, and DrugBank, TriMod-DTI reported AUC values of $s(a,b)=a^\top b / (\|a\| \|b\|)$2, $s(a,b)=a^\top b / (\|a\| \|b\|)$3, and $s(a,b)=a^\top b / (\|a\| \|b\|)$4, respectively, and the full contrastive learning setting yielded $s(a,b)=a^\top b / (\|a\| \|b\|)$5 AUC and $s(a,b)=a^\top b / (\|a\| \|b\|)$6 AUPR over non-contrastive fusion (Xu et al., 28 May 2026).

VECO 2.0 occupies an intermediate position between cross-modal and multi-granularity learning. It is encoder-only, uses a shared 250k SentencePiece vocabulary across 109 languages, and complements MLM/TLM with sentence-to-sentence contrast and token-to-token synonym contrast. On XTREME’s 9 tasks across 40 languages, VECO 2.0 improved average by +15.4% over mBERT, +19.5% over XLM, +7.0% over XLM-R, and +2.1% over VECO, while surpassing HICTL on 9/12 metrics with +5.6% average improvement. Its strongest gains were especially aligned with the token-level objective: POS 75.4, NER 67.2, and TyDiQA-GoldP 71.1/54.7 EM (Zhang et al., 2023).

4. Sequential recommendation as a major development line

Sequential recommendation has become a major venue for Sequence-Cross Perspective Contrastive Learning. CL4SRec established an early sequence-level formulation by constructing two augmented views of each user sequence through item cropping, item masking, or item reordering, encoding both views with a shared SASRec-style Transformer, and optimizing $s(a,b)=a^\top b / (\|a\| \|b\|)$7. With leave-one-out evaluation on Beauty, Sports, Yelp, and ML-1M, it reported average improvements over second-best baselines of HR@5 +11.02%, HR@10 +9.69%, NDCG@5 +5.52%, and NDCG@10 +8.50% (Xie et al., 2020).

Later work increasingly replaced heuristic augmentation with semantically grounded cross-sequence views. ICSRec segments each user history into multiple subsequences by dynamic sliding, treats two subsequences as positives if both end with the same target item, and refines this with K-means prototypes for fine-grain intentions. Its total loss is $s(a,b)=a^\top b / (\|a\| \|b\|)$8. On Sports, Beauty, Toys, and ML-1M, representative NDCG gains over the best baseline were +44.39%, +41.95%, +41.34%, and +19.33% at NDCG@5, respectively (Qin et al., 2023). SRA-CL pushes this semantic turn further by retrieving semantically similar users and items with LLM-generated summaries and SimCSE-RoBERTa embeddings, then forming cross-sequence positives through a learnable synthesis module. On Sports, Yelp, Beauty, and Office, example improvements over the best baseline at HR@20/NDCG@20 were +11.82%/+9.12%, +9.29%/+7.68%, +6.05%/+5.97%, and +3.59%/+5.07% (Cui et al., 6 Mar 2025). QCMP-CL similarly uses same-target sequences and Jaccard-similar sequences as two collaborative perspectives, but adds a quality-aware weighting mechanism for multiple positives. On Beauty, Yelp, and Sports, its HR@10 values were 0.5632, 0.8698, and 0.5813, outperforming both CL4SRec and TCLARec in the reported comparisons (Wang, 12 May 2026).

A parallel line addresses the instability of stochastic views by learning them. Meta-SGCL uses a Seq2Seq VAE to generate semantically faithful sequence perspectives $s(a,b)=a^\top b / (\|a\| \|b\|)$9 and $\tau=0.05$0, aligns them with InfoNCE, and updates the view generator through a meta-optimized bilevel procedure. Its total loss combines recommendation, contrastive, and KL terms,

$\tau=0.05$1

On Amazon Clothing, Amazon Toys, and ML-1M, it improved over the best baseline by up to 25.66% on NDCG@5, 19.34% across HR@5/10 and NDCG@5/10 on Toys, and 16.25%–20.84% across HR@5/10 and NDCG@5/10 on ML-1M (Hao et al., 2023).

Other formulations generalize “perspective” beyond paired sequence views. MSDCCL contrasts a soft-denoised user representation with item embeddings labeled as relevant or noisy by hard denoising signals, thereby aligning soft and hard sequence-derived perspectives; on ML-100k, MSDCCL+BERT4Rec improved HR@20/NDCG@20/MRR@20 by 133.11%, 168.25%, and 217.20% over BERT4Rec (Zhu et al., 2024). Tri-CDR defines source, target, and mixed behavior sequences as three coordinated perspectives and combines coarse-grained InfoNCE alignment with a fine-grained margin-based distinction loss; on Amazon and Douban cross-domain settings, it achieved NDCG@10 improvements over the best baseline ranging from 1.25% to 5.18% (Ma et al., 2023). C$\tau=0.05$2DSR combines intra-sequence sequential relationships and inter-sequence item collaborative relationships, then maximizes mutual information between single-domain and cross-domain user representations via a JS-based discriminator; on Food–Kitchen, for example, Food-domain NDCG@10/HR@10 reached 9.71/14.54 (Cao et al., 2023). MCLSR extends the same logic to four views at two levels—sequential, user–item, user–user, and item–item—and reported improvements over the best baseline such as Recall@20 +10.3% and NDCG@20 +9.0% on Toys (Wang et al., 2022).

5. Pair construction, optimization design, and recurrent misconceptions

A recurrent technical issue is how positives and negatives are defined. The literature shows that these definitions are not interchangeable. HiCL explicitly reports that segments from the same sequence should be treated as neither positives nor negatives; forcing in-sequence segment entailment did not help and sometimes hurt (Wu et al., 2023). VECO 2.0 likewise shows that token-level contrast alone boosts NER by +0.9 and slightly hurts XNLI by -0.9, indicating that finer-grained contrast can perturb sentence-level semantics if it is not balanced by sequence-level alignment (Zhang et al., 2023). In recommendation, ICSRec and QCMP-CL both argue that heuristic masking, cropping, or perturbation may change latent intention, motivating positives derived from same-target subsequences or collaborative contexts rather than random edits (Qin et al., 2023, Wang, 12 May 2026).

Another recurrent issue is whether stronger alignment automatically requires heavier architecture. The answer is generally negative. HiCL uses length-weighted average pooling and states that because the majority of sequences split into $\tau=0.05$3 segments, attention pooling or additional Transformer layers were unnecessary and did not offer clear benefits (Wu et al., 2023). ProtAlign uses single-query attention pooling heads rather than cross-attention, requires no hard negative mining or queue/memory bank, and reports that CLIP converges faster and more stably than SigLIP (Ranganath et al., 6 Mar 2026). VECO 2.0 is encoder-only and explicitly avoids the extra cross-attention module used by VECO 1.0 (Zhang et al., 2023). STCCR, by contrast, does use a queue and prototypes because its objective is not only instance alignment but also discovery of shared spatial topics; its best settings were $\tau=0.05$4, queue length $\tau=0.05$5, and angular margin $\tau=0.05$6 (Gong et al., 2024).

A further misconception is that cross-perspective contrast is equivalent to adding more augmentations. Several systems are defined precisely by their rejection of that assumption. SRA-CL argues that current approaches struggle to generate reliable positive pairs because they rely on sparse collaborative signals or random perturbations, and therefore introduces semantic retrieval via LLMs to construct positives (Cui et al., 6 Mar 2025). Meta-SGCL replaces hand-crafted sequence edits with a conditional Seq2Seq VAE view generator (Hao et al., 2023). MSDCCL uses semantically grounded denoising signals instead of random dual-view perturbations (Zhu et al., 2024). These designs suggest that the central variable is not augmentation quantity but semantic validity of the paired perspectives.

6. Empirical impact, limitations, and future directions

Across domains, the empirical effect of the paradigm is consistent but domain-specific. In unsupervised sentence embedding, cross-perspective alignment improves both efficiency and robustness across text lengths, with HiCL showing larger gains when WikiText-103 is added and reporting transfer improvements on long-document tasks of Yelp +1.97% and IMDB +2.27% (Wu et al., 2023). In multilingual pretraining, VECO 2.0 improves sentence retrieval and token-level transfer simultaneously, with BUCC 86.2 and Tatoeba 81.8 alongside gains on NER and QA (Zhang et al., 2023). In protein modeling, ProtAlign demonstrates that explicit sequence–structure alignment is sufficient for high Recall@5 retrieval and interpretable latent neighborhoods, while CCPL shows that structure encoders aligned to sequence teachers can improve inverse folding (Ranganath et al., 6 Mar 2026, Zheng et al., 2023). In mobility modeling, STCCR improves next location prediction, time prediction, and trajectory-user linking by fusing “spatial topic” and “temporal intention” at the semantic level rather than the token level (Gong et al., 2024). In DTI, TriMod-DTI shows that sequence-based embeddings remain the strongest single modality, but pairwise alignment to graph and 3D views improves multimodal fusion (Xu et al., 28 May 2026).

The limitations are equally recurrent. HiCL notes that pre-training from scratch on very large corpora remains to be fully explored and that supervised NLI-based contrastive training was not investigated (Wu et al., 2023). VECO 2.0 identifies dictionary noise, polysemy, domain mismatch, and untuned temperatures and loss weights as open issues (Zhang et al., 2023). ProtAlign depends on structural data and reports no robustness analysis for structural noise or hard-negative mining (Ranganath et al., 6 Mar 2026). SRA-CL is sensitive to prompt quality and incurs offline LLM preprocessing cost (Cui et al., 6 Mar 2025). QCMP-CL still uses in-batch negatives and does not explicitly down-weight false negatives on the negative side (Wang, 12 May 2026). TriMod-DTI leaves some hyperparameters unreported and notes that 3D-only performance is relatively weak because only coordinates and bonds are used (Xu et al., 28 May 2026). STCCR assumes availability of POI metadata such as categories and coordinates (Gong et al., 2024).

Future directions in the cited literature extend the paradigm along three axes. One is broader granularity: HiCL proposes cross-document hierarchical contrast and dynamic semantic segmentation; MCLSR and Tri-CDR indicate that additional graph or domain views can be brought into the same contrastive system (Wu et al., 2023, Wang et al., 2022, Ma et al., 2023). A second is modality expansion: ProtAlign explicitly suggests alignment with MSA/profile, secondary-structure strings, functional annotations, and experimental observables, while TriMod-DTI already generalizes sequence-centered contrast to graph and 3D features (Ranganath et al., 6 Mar 2026, Xu et al., 28 May 2026). A third is more reliable view construction: SRA-CL, QCMP-CL, and Meta-SGCL all imply that the next stage of progress may depend less on generic augmentations than on retrieval, generation, or confidence-aware mechanisms that preserve the semantics of the underlying sequence (Cui et al., 6 Mar 2025, Wang, 12 May 2026, Hao et al., 2023).

Taken together, Sequence-Cross Perspective Contrastive Learning is best understood as a unifying strategy for forcing consistency across complementary sequence-derived views while preserving the distinctions that each view contributes. Its concrete implementations differ sharply by field, but the shared structure is stable: define semantically meaningful perspectives, assign positives and negatives in a way that respects the data-generating process, and optimize joint contrastive objectives so that sequence representations remain coherent across granularity, modality, domain, or signal type (Wu et al., 2023, Zhang et al., 2023, Ranganath et al., 6 Mar 2026, Qin et al., 2023).