AutoCDSR+ Enhancement: Optimized CDSR

• The paper demonstrates an effective method for optimizing cross-domain recommendations through Pareto-optimal multi-objective learning.
• It introduces structured information bottleneck tokens to control cross-domain attention and mitigate negative transfer.
• Empirical results show up to 42% improvement in ranking metrics, validating the approach’s efficiency and plug-and-play design.

AutoCDSR$^+$ Enhancement is an advanced architectural and optimization technique for cross-domain sequential recommendation (CDSR) on transformer-based models. Developed as an augmentation to the foundational AutoCDSR framework, AutoCDSR$^+$ introduces structured bottlenecking and Pareto-optimal multi-objective learning, enabling robust and automated knowledge transfer between user histories in distinct domains. This approach is realized as a lightweight, plug-and-play module that brings principled improvements in recommendation performance while mitigating the challenges of negative transfer and uncontrolled information leakage across domains.

1. Theoretical Foundations and Motivation

Cross-domain sequential recommendation requires modeling user interactions distributed across multiple domains (e.g., books, movies, games). Naive concatenation of sequences and application of standard transformer self-attention typically lead to negative transfer, as transformer heads may amplify noise rather than harness complementary domain information. Previous CDSR methods frequently employ domain-specific modules—reweighting mechanisms, domain-aware blocks, etc.—which introduce substantial modeling and computational overhead.

AutoCDSR$^+$ reframes the role of self-attention in transformers as a potent, if adequately optimized, mechanism for automating knowledge transfer between domains. The paradigm shift is toward direct multi-objective optimization of self-attention, requiring neither auxiliary modules nor extensive manual tuning, and targeting automated, empirically validated mitigation of negative transfer.

2. Pareto-Optimal Multi-Objective Formulation

The core AutoCDSR methodology casts CDSR as a multi-objective problem, simultaneously:

• Minimizing standard recommendation loss, typically categorical cross-entropy ($\mathcal{L}_{\text{rec}}$), for next-item prediction.
• Dynamically minimizing the cross-domain attention intensity ($a_{\text{cd}}$), thereby controlling the degree of cross-domain transfer:

$$a_{\text{cd}} = \sum_{i=1}^M \sum_{j=1}^M \text{softmax}(\mathbf{A})_{i,j} \cdot \mathbb{I}(d(x_i) \neq d(x_j))$$

where $\mathbf{A}$ is the attention score matrix and $\mathbb{I}$ is the indicator for domain mismatch.

The joint loss is:

$$\mathcal{L} = \alpha_1 \cdot \mathcal{L}_{\text{rec}} + \alpha_2 \cdot a_{\text{cd}}$$

Fixed loss weight settings prove inadequate due to user- and sequence-level variance in domain complementarity. Accordingly, the framework invokes Pareto-optimality: a parameter set $\boldsymbol{\theta}^*$ is Pareto-optimal if no other configuration decreases either sub-objective without increasing the other.

Optimization leverages the Multiple Gradient Descent Algorithm (MGDA), constructing a preference-aware combination of gradients:

$$\min_{\alpha_1, \alpha_2} \left\lVert \alpha_1 g_{\text{rec}} + \alpha_2 g_{\text{cd}} \right\rVert_F \quad \text{s.t.} \;\; \alpha_1 + \alpha_2 = 1,\; \alpha_i \geq 0$$

Preference vectors prioritize solutions that minimize the recommendation loss, providing application-relevant tradeoff control. The result is automated, sequence-specific modulation of cross-domain attention—promoting transfer only when beneficial, mitigating negative transfer otherwise.

3. AutoCDSR$^+$: Structured Knowledge Transfer via Bottleneck Tokens

While Pareto-optimized self-attention regulates information flow across domains, unstructured transfer remains susceptible to leakage and residual noise. AutoCDSR$^+$ introduces Information Bottleneck (IB) tokens, inspired by advances in multimodal transformers:

• IB tokens are inserted in each domain-sequence as exclusive mediators for cross-domain information exchange.
• Attention masking enforces that domain sequence items attend only to local items and their corresponding IB token.
• Cross-domain communication is restricted such that IB tokens from distinct domains are the sole vectors for transfer. Items re-attend to IB tokens, enabling indirect, controlled domain knowledge acquisition.

The cross-domain attention metric for AutoCDSR$^+$ becomes:

$$a_{\text{cd}} = \sum_{d \in \mathcal{D}} \sum_{i = T}^{M^d} \sum_{j = 1}^T \mathbf{A}^d_{i,j}$$

where the sum selectively quantifies attention from domain items to cross-domain IB tokens.

This structuring ensures that cross-domain knowledge flow is both explicit and confined, strongly mitigating unwanted interference and further reducing negative transfer, while maintaining the Pareto-optimal multi-objective regime.

4. Empirical Performance and Comparative Advantages

Extensive experiments on datasets such as Amazon Reviews, KuaiRand-1K, and industrial-scale benchmarks demonstrate that both AutoCDSR and AutoCDSR$^+$ yield consistent and substantial improvements over base transformer models, including SASRec and BERT4Rec:

Model	Recall@10 (+)	NDCG@10 (+)
SASRec	9.8%	12.0%
BERT4Rec	16.0%	16.7%

Reported metrics (see Table 1 of main paper) confirm up to 42% increases on key ranking indices across cutoffs, and AutoCDSR$^+$ equipped transformers match or outperform leading bespoke CDSR methods, despite minimal architectural modification.

5. Implementation, Computational Complexity, and Deployment

AutoCDSR and AutoCDSR$^+$ are architected as wraparound modules for sequential recommenders:

• No new network blocks are required; IB tokens are additional input-level placeholders.
• Computational overheads are controlled: AutoCDSR is 9.3% slower than vanilla BERT4Rec; AutoCDSR$^+$ is 19.9% slower (due to IB token masking and processing), but remains ~4x faster than more complex CDSR competitors.
• The bulk of overhead is attributed to Pareto-optimal gradient computation, which converges rapidly ($\leq$100 iterations) and supports parallelization.
• Plug-and-play deployment is emphasized, with minimal hyperparameter tuning requirements due to adaptive multi-objective weighting.
• Robustness evaluations indicate AutoCDSR remains effective under noisy/missing domain labels, while AutoCDSR$^+$ is more sensitive—defaulting to AutoCDSR is recommended for less reliable domain annotation.

6. Comparative Summary Table

Aspect	AutoCDSR	AutoCDSR$^+$
Negative Transfer	Mitigated via Pareto loss	Further mitigated via bottlenecked attention
Knowledge Transfer	Pareto-optimized, automatic	+ explicit IB token gating
Additional Modules	None	Adds IB tokens, no structural change
Overhead	+9%	+20%
Plug-and-play	Yes	Yes
Domain label noise	Robust	Less robust
Empirical gains	Large	Slightly larger

7. Insights and Implications

Rigorous optimization of self-attention using a Pareto-front multi-objective framework enables both noise filtering and complementary knowledge harvesting between user behaviors in distinct domains. AutoCDSR$^+$ leverages the strategic insertion of IB tokens to further channel cross-domain transfer, producing an architecture wherein transfer is data-driven, explicit, and minimally intrusive.

The adaptivity of the approach—allocating cross-domain transfer capacity per user and sequence based on benefit to main task loss—addresses prior practicality gaps in CDSR arising from static loss weights or fixed domain-dependent reweighting. AutoCDSR$^+$ demonstrates that highly performant CDSR can be achieved with lightweight augmentations of backbone transformer architectures, reducing the barriers to practical deployment in large-scale recommendation settings.

References to Formulas and Algorithms

• Multi-objective Pareto-optimal gradient calculation:

$$\min_{\alpha_1, \alpha_2} \left\lVert \alpha_1 \nabla_{\theta} \mathcal{L}_{\text{rec}} + \alpha_2 \nabla_{\theta} \mathcal{L}_{\text{cd-attn}} \right\rVert_F \quad \text{s.t.} \;\; \alpha_1 + \alpha_2 = 1, \alpha_i \geq 0$$

• Preference-aware Pareto optimality for main task prioritization via preference vectors.

AutoCDSR$^+$ constitutes a substantive enhancement of automatic knowledge transfer in cross-domain sequential recommendation, balancing performance, efficiency, and deployment practicality as verified on real-world datasets (Ju et al., 27 May 2025).