HSTU-BLaIR: Hybrid Recommender System

• HSTU-BLaIR is a hybrid recommender system that combines Transformer-based sequential modeling with domain-specific contrastive text embeddings for improved next-item prediction.
• It operates via a two-stage pipeline, using BLaIR to generate fixed, semantically enriched item embeddings and HSTU to perform autoregressive sequence modeling.
• Empirical evaluations demonstrate that HSTU-BLaIR consistently outperforms text-agnostic and general-purpose embedding baselines, emphasizing its compute efficiency and domain adaptation.

HSTU-BLaIR is a hybrid recommender system architecture designed to advance next-item prediction in e-commerce environments. It synthesizes the Hierarchical Sequential Transduction Unit (HSTU), a generative autoregressive recommender built on stacked Transformers, with BLaIR, a lightweight contrastive text embedding encoder. BLaIR leverages domain-specific user reviews and item metadata to enrich item representations with semantic signals, complementing HSTU's sequential modeling capabilities. Empirical results on challenging real-world datasets demonstrate that HSTU-BLaIR surpasses both text-agnostic baselines and variants using large, general-purpose pretrained text embedding models, underscoring the relevance of domain-adaptive contrastive embeddings in compute-efficient settings (Liu, 13 Apr 2025).

1. System Architecture and Fusion Mechanism

HSTU-BLaIR operates as a two-stage hybrid pipeline. At its core:

• Textual Embedding Generation: BLaIR_BASE (~125M parameters) is pretrained on domain-specific review and metadata corpora, producing a 768-dimensional embedding ($e_\text{text}$) for each item. These are precomputed and fixed during downstream tuning.
• Item ID Embedding: HSTU randomly initializes a trainable embedding ($e_\text{item} \in \mathbb{R}^{256}$) for each item in the catalog.
• Embedding Fusion: A learnable linear projection ($W_\text{text} \in \mathbb{R}^{256 \times 768}$) transforms $e_\text{text}$ to match $e_\text{item}$'s dimension and fuses them via element-wise addition:

$$e_\text{combined} = e_\text{item} + W_\text{text} \cdot e_\text{text}$$

• Positional Encoding and Sequence Modeling: Input to HSTU's four-layer Transformer stack is formed by adding relative positional encoding ($e_\text{pos}$), yielding $e_\text{input} = e_\text{pos} + e_\text{combined}$. The sequence of fused embeddings is processed autoregressively, maintaining Transformer output dimensionality $d=256$.

The output interaction module computes dot-product scores between the current hidden state $h_t$ and all item embeddings. Negative sampling for the softmax is guided semantically, selecting hard negatives based on text similarity.

2. Mathematical Formulation

2.1 BLaIR Contrastive Objective

Although the precise objective is not fully displayed, BLaIR adopts a SimCSE-style InfoNCE loss over minibatches of augmented item texts:

$e_\text{item} \in \mathbb{R}^{256}$0

where $e_\text{item} \in \mathbb{R}^{256}$1 denotes cosine similarity, $e_\text{item} \in \mathbb{R}^{256}$2 is a temperature scaling hyperparameter, $e_\text{item} \in \mathbb{R}^{256}$3 is the batch size, $e_\text{item} \in \mathbb{R}^{256}$4 is a positive augmented view (e.g., dropout-induced) of anchor $e_\text{item} \in \mathbb{R}^{256}$5, and $e_\text{item} \in \mathbb{R}^{256}$6 serve as in-batch negatives.

2.2 Generative Sequential Objective

For autoregressive next-item prediction, HSTU utilizes a cross-entropy objective:

$e_\text{item} \in \mathbb{R}^{256}$7

with conditional probabilities computed as:

$e_\text{item} \in \mathbb{R}^{256}$8

where $e_\text{item} \in \mathbb{R}^{256}$9 is the hidden state up to position $W_\text{text} \in \mathbb{R}^{256 \times 768}$0 and $W_\text{text} \in \mathbb{R}^{256 \times 768}$1 is the total item vocabulary.

2.3 Training Regime

Objectives are optimized sequentially, not jointly. BLaIR is first pretrained, fixing $W_\text{text} \in \mathbb{R}^{256 \times 768}$2 for all items. Only HSTU's loss is then applied during fine-tuning:

$W_\text{text} \in \mathbb{R}^{256 \times 768}$3

The formulation $W_\text{text} \in \mathbb{R}^{256 \times 768}$4 is not adopted in the published work.

3. Training Protocols and Datasets

BLaIR Pretraining

Pretraining uses the first 80% of domain-specific Amazon Reviews 2023 data (chronologically ordered). The corpus includes only user reviews and item metadata for the target domain. Training parameters are: Adam optimizer (learning rate $W_\text{text} \in \mathbb{R}^{256 \times 768}$5), batch size 512, temperature $W_\text{text} \in \mathbb{R}^{256 \times 768}$6, and approximately three epochs until convergence.

HSTU-BLaIR Fine-Tuning

Fused embeddings from BLaIR and HSTU are used as inputs. Protocols involve full data shuffling, leave-one-out assignment for test and validation, and multi-epoch training (100 epochs). AdamW is employed ($W_\text{text} \in \mathbb{R}^{256 \times 768}$7, $W_\text{text} \in \mathbb{R}^{256 \times 768}$8), with batch size of 1,024 user sequences.

Evaluation Corpora

Experiments focus on two 5-core Amazon Reviews 2023 domains:

• Video Games: 25,612 items, 94,762 users, 814,585 interactions.
• Office Products: 77,551 items, 223,308 users, 1,800,877 interactions.

Preprocessing enforces 5-core filtering, chronological sorting of user histories, and leave-one-out sampling. For comparison with OpenAI's text-embedding-3-large (TE3L) model, item review texts longer than 8,179 tokens are truncated.

4. Model Properties and Fusion Strategy

BLaIR Encoder

• SimCSE-inspired, bidirectional Transformer.
• 125M parameters, 768-dim output.
• Trained exclusively on domain-specific texts.

HSTU Transformer

• 4 layers, 4 attention heads per layer.
• Hidden size 256, feed-forward inner dimension 512.
• Relative positional bias in the style of T5.

Embedding Fusion and Interaction

• $W_\text{text} \in \mathbb{R}^{256 \times 768}$9 trainable.
• $e_\text{text}$0 fixed, projected via $e_\text{text}$1.
• Fused by elementwise addition.
• Dot-product output scoring; negative sampling informed by textual similarity ("semantically informed negatives").

This configuration supports the integration of semantic evidence from text, while maintaining computational tractability in large-scale item catalogs.

5. Quantitative Evaluation and Empirical Findings

Performance was assessed on next-item ranking using HR@K and NDCG@K for $e_\text{text}$2, with metrics computed as follows:

• HR@K: Fraction of users for whom the true next item is within the top-K recommendations.
• NDCG@K: Discounted cumulative gain at top-K, normalized by ideal ranking.

Key metrics are summarized in the table below, including percentage improvements relative to HSTU.

Dataset	Model	HR@10	HR@50	HR@200	NDCG@10	NDCG@200
Video Games	SASRec	0.1028	0.2317	0.3941	0.0573	0.1097
	HSTU	0.1315	0.2765	0.4565	0.0741	0.1327
	HSTU-OpenAI(TE3L)	0.1328	0.2821	0.4645	0.0742	0.1341
	HSTU-BLaIR	0.1353	0.2852	0.4684	0.0760	0.1361
Office Products	SASRec	0.0281	0.0668	0.1331	0.0153	0.0335
	HSTU	0.0395	0.0880	0.1649	0.0223	0.0443
	HSTU-OpenAI(TE3L)	0.0477	0.1050	0.1940	0.0269	0.0526
	HSTU-BLaIR	0.0484	0.1068	0.1946	0.0271	0.0529

Principal observations:

• HSTU-BLaIR consistently surpasses the original HSTU by 2–3% absolute (Video Games) and 1–1.5% (Office Products).
• BLaIR-enhanced fusion outperforms the OpenAI TE3L embedding variant for all but one metric, and matches it for one.
• Office Products, a sparser domain, exhibits pronounced gains (up to +77% NDCG@10 over SASRec, +21.5% over HSTU).

6. Comparative and Ablation Analysis

While no formal ablation is conducted on fusion variants or embedding dimensionality, direct comparison of:

• HSTU (no text)
• HSTU-OpenAI (general large-scale LLM embeddings)
• HSTU-BLaIR (lightweight, domain-tuned contrastive embeddings)

— demonstrates that domain-specific contrastive pretraining produces more semantically effective item signals than large, general-purpose embedding models. Element-wise fusion of ID and text embeddings suffices to yield substantial performance improvements.

A plausible implication is that leveraging domain adaptation in textual encoders amplifies benefits when resource constraints preclude deployment of large universal models.

7. Prospective Directions

The paper notes that dynamic or context-adaptive fusion mechanisms and richer output interaction modules (beyond dot-product) constitute promising extensions. This suggests further gains may be achievable by refining the integration of semantic and sequential signals, potentially through deeper multimodal modeling or adaptive hard negative mining.

HSTU-BLaIR exemplifies the efficacy of compute-efficient, domain-focused contrastive embeddings in generative recommender systems. Future research may explore joint training regimes and advanced fusion architectures for even higher fidelity item modeling (Liu, 13 Apr 2025).