---
title: 'NICL: Next-item Cross-modal Contrastive Learning'
url: https://www.emergentmind.com/topics/next-item-enhanced-cross-modal-contrastive-learning-nicl
type: topic
---

# NICL: Next-item Cross-modal Contrastive Learning

Next-item Enhanced Cross-modal Contrastive Learning (NICL) is a contrastive learning objective explicitly designed for multi-modal recommender systems to fuse cross-modal alignment and sequential transition modeling at the encoder level. NICL is introduced in the PMMRec architecture within "Multi-Modality is All You Need for Transferable Recommender Systems" [2312.09602] and is central to achieving high transferability and robustness in recommendation tasks without reliance on ID-based paradigms. The objective unifies text and vision modalities in a shared embedding space while explicitly endowing the item encoders themselves with transition (next-item) modeling capability.

## 1. Mathematical Formulation

NICL operates on a mini-batch of user behavior sequences. Each sequence of length $L$ for user $u$ consists of items $i^\mathit{u}_1,\,\ldots,i^\mathit{u}_L$, and each item is represented by modality-specific encoders:

- $t^\mathit{u}_l = \mathbf{t}^{cls}_{i^\mathit{u}_l}$: text encoder output (ℓ₂-normalized)
- $v^\mathit{u}_l = \mathbf{v}^{cls}_{i^\mathit{u}_l}$: vision encoder output (ℓ₂-normalized)

The similarity function is:
$$
\delta(a, b) = \exp(a^\top b)
$$
with no learnable or tunable temperature ($\tau=1$).

Given anchor $i^\mathit{u}_l$, let
$$
\mathcal{N}_{i^\mathit{u}_l} = \left\{ i^\mathit{k}_j\,|\,1\leq k\leq B,\,k\neq u,\,1\leq j \leq L \right\}\,,
$$
the in-batch negatives for all other users $k\neq u$, all positions $j$.

For each $l < L$, two symmetric losses are applied:

\[
\begin{aligned}
\mathcal{L}^{T,V}_{i^\mathit{u}_l} 
&= -\log \frac{\,\delta(t^\mathit{u}_l, v^\mathit{u}_l) + \delta(t^\mathit{u}_l, v^\mathit{u}_{l+1}) + \delta(t^\mathit{u}_l, t^\mathit{u}_{l+1})\,}
{\,\delta(t^\mathit{u}_l, v^\mathit{u}_l) + \sum_{i^\mathit{k}_j\in\mathcal{N}} \big[\delta(t^\mathit{u}_l, v^\mathit{k}_j) + \delta(t^\mathit{u}_l, t^\mathit{k}_j)\big]\,} \\

\mathcal{L}^{V,T}_{i^\mathit{u}_l}
&= -\log \frac{\,\delta(v^\mathit{u}_l, t^\mathit{u}_l) + \delta(v^\mathit{u}_l, t^\mathit{u}_{l+1}) + \delta(v^\mathit{u}_l, v^\mathit{u}_{l+1})\,}
{\,\delta(v^\mathit{u}_l, t^\mathit{u}_l) + \sum_{i^\mathit{k}_j\in\mathcal{N}} \big[\delta(v^\mathit{u}_l, t^\mathit{k}_j) + \delta(v^\mathit{u}_l, v^\mathit{k}_j)\big]\,}
\end{aligned}
\]

NICL is the average of these over both directions, summed for all users and positions:

\[
\mathcal{L}^{NICL} = \frac{1}{B\,(L-1)} \sum_{u=1}^{B}\sum_{l=1}^{L-1} \frac{1}{2}\left(\mathcal{L}^{T,V}_{i^\mathit{u}_l}+\mathcal{L}^{V,T}_{i^\mathit{u}_l}\right)
\]

This formulation explicitly acknowledges both primary (same-item, inter-modality) and next-item (cross- and intra-modality) positives, with all other positions in the batch serving as negatives in both modalities.

## 2. Construction of Positive and Negative Pairs

NICL distinguishes itself by the systematic inclusion of next-item transition information into contrastive objectives. The positive and negative pairings are constructed as follows:

| Pair Type                      | Anchor       | Positive Example          |
|-------------------------------|--------------|--------------------------|
| Inter-modality (same item)     | $t^\mathit{u}_l$/$v^\mathit{u}_l$ | $v^\mathit{u}_l$/$t^\mathit{u}_l$         |
| Inter-modality (next item)     | $t^\mathit{u}_l$/$v^\mathit{u}_l$ | $v^\mathit{u}_{l+1}$/$t^\mathit{u}_{l+1}$ |
| Intra-modality (next item)     | $t^\mathit{u}_l$/$v^\mathit{u}_l$ | $t^\mathit{u}_{l+1}$/$v^\mathit{u}_{l+1}$ |
| Inter-modality negatives       | $t^\mathit{u}_l$/$v^\mathit{u}_l$ | all $v^\mathit{k}_j$/$t^\mathit{k}_j$ for $k\neq u$, $\forall j$ |
| Intra-modality negatives       | $t^\mathit{u}_l$/$v^\mathit{u}_l$ | all $t^\mathit{k}_j$/$v^\mathit{k}_j$ for $k\neq u$, $\forall j$ |

This mixture ensures that the shared representation space encodes both modality alignment and ordered behavioral transitions.

## 3. Injection of Next-item Information

NICL's central innovation is its use of next-item positives, both cross-modally and intra-modally, thereby "front-loading" sequential transition knowledge into the item encoders themselves. Concretely:

- For anchor $t^\mathit{u}_l$, positives are $v^\mathit{u}_l$ (same-item, inter-modal), $v^\mathit{u}_{l+1}$ (next-item, cross-modal), and $t^\mathit{u}_{l+1}$ (next-item, intra-modal).
- Symmetric construction applies for anchors in the vision modality.

Unlike objectives that only align the same item or defer transition modeling to downstream sequence modules, NICL ensures that each modality-specific item encoder, via contrastive pairing, predicts the representation of immediate sequelae in the user's interaction sequence. This mechanism injects Markovian transition patterns directly into the learned representations.

## 4. Integration Within the PMMRec Architecture

NICL is executed within the PMMRec pretraining pipeline as a standalone loss involving only the outputs of the modality-specific encoders, prior to any fusion or user-level modeling. The core workflow is as follows:

```python
for each pre-training step:
    # Sample user sequences
    { s_u = [i_1^u ... i_L^u] } for u = 1,...,B

    # Item encoding
    for u in 1...B:
        for l in 1...L:
            t_l^u = TextEncoder(text_tokens(i_l^u))  # ℓ₂-normalized
            v_l^u = VisionEncoder(image_patches(i_l^u))  # ℓ₂-normalized

    # Fusion and user-level autoregression (DAP)
    for u in 1...B, l in 1...L:
        e_l^u = FusionModule([t_l^u_tokens ; v_l^u_patches])
    {h_l^u} = UserTransformer([e_1^u+pos_1 ... e_L^u+pos_L])
    L^DAP = mean next-item autoregressive loss using h_l^u vs. e_{l+1}^u

    # NICL objective—uses only t_l^u, v_l^u before fusion
    L^NICL = 0
    for u in 1...B:
        for l in 1...L-1:
            compute L^{T,V}_{i_l^u} and L^{V,T}_{i_l^u}
            L^NICL += 0.5 * (L^{T,V}_{i_l^u} + L^{V,T}_{i_l^u})
    L^NICL /= (B * (L-1))

    # Add self-supervised denoising objectives (NID, RCL)
    # ...
    # Final multi-task loss and parameter updates
    L_total = L^DAP + L^NICL + L^NID + L^RCL
    update all parameters
```

NICL thus sits orthogonally to downstream autoregressive or fusion-based objectives, focusing exclusively on aligning and sequence-enhancing the raw modality encodings.

## 5. Key Hyperparameters and Ablation Observations

Critical NICL hyperparameters and design choices are:

- **Batch size:** $B=64$ per GPU.
- **Sequence length:** $L$ is dataset-specific (e.g., $L=50$).
- **Similarity temperature:** No explicit temperature parameter; $\delta(a, b) = \exp(a^\top b)$.
- **Number of negatives:** $(B-1)\times L$ per anchor, per modality (both inter- and intra-modality).
- **Averaging:** Loss is computed over $B\cdot(L-1)$ anchors.

Ablations in the referenced work demonstrate:

- Removing NICL (keeping only next-item autoregression and denoising) reduces HR@10 by up to 1–2 points.
- Replacing NICL with vanilla cross-modal contrastive learning (without next-item positives) or with intra-modal enhanced contrastive learning (without cross-modal next-item positives) yields intermediate performance.
- Disabling either next-item positives or intra-modality negatives each results in a 3–5% relative drop in performance, confirming the necessity of both elements in achieving optimal transfer and recommendation accuracy.

## 6. Functional Role and Empirical Significance

NICL is largely responsible for two key attributes of PMMRec: the integration of cross-modal (text-image) alignment and the embedding of transition (next-item) behavior patterns at the item-encoder level. By blending inter- and intra-modality negatives with next-item positives, NICL:

1. Aligns text and vision item representations into a unified latent space suitable for downstream recommendation.
2. Encodes one-step Markovian transitions within the item encoders, without depending solely on higher-level user or sequence encoders.
3. Empowers the modality encoders with transferability; after pretraining, these encoders generalize across domains without retraining, as demonstrated by consistent improvements over strong baselines in multi-domain benchmarks.

The mechanism further distinguishes itself from traditional contrastive objectives that either only align the same item across modalities or rely solely on sequence-level modeling for transition information.

## 7. Context and Implications for Transferable Recommender Systems

NICL is formulated to address two deficiencies inherent in ID-based and unimodal recommendation approaches: the lack of transferability and the cold start problem. By relying exclusively on multi-modal content and embedding the sequential inductive bias at the encoder level, PMMRec—via NICL—sidesteps ID dependence and exhibits strong cross-domain transfer adaptivity.

A plausible implication is that the introduction of structured next-item contrastive objectives generalizes to other multi-modal or sequential learning tasks beyond recommendation, wherever cross-modal alignment and temporal dependence are jointly required.

In sum, NICL is a core component enabling PMMRec to outperform prior state-of-the-art on transfer and recommendation metrics, while supporting decoupled pretraining and reusability of its constituent modules [2312.09602].

Source: https://www.emergentmind.com/topics/next-item-enhanced-cross-modal-contrastive-learning-nicl