SEMCo: Sampled Entmax for Cold-Start

• The paper introduces SEMCo, a cold-start recommendation method that uses a sparse, α-entmax formulation to zero out uninformative negatives and enhance ranking performance.
• SEMCo reformulates recommendation by factorizing item-item similarities through a multimodal content encoder, sidestepping traditional collaborative filtering space alignment.
• It integrates content-space knowledge distillation to further refine item embeddings, leading to significant gains in ranking accuracy and more equitable item exposure.

Searching arXiv for the cited SEMCo paper and closely related references. Sampled Entmax for Cold-start (SEMCo) is a cold-item recommendation framework introduced in "Sparse Contrastive Learning for Content-Based Cold Item Recommendation" that replaces collaborative-filtering-space alignment with a purely content-based latent similarity model for new items (Meehan et al., 14 Apr 2026). It addresses the item cold-start problem by learning item embeddings directly from auxiliary content, while using user–item interactions only as supervision for arranging a content space in which item similarity correlates with user preferences. Its central technical contribution is a sparse generalization of sampled softmax based on the $\alpha$-entmax family, so that gradients for uninformative negatives can be zeroed out. The method is further extended with content-space knowledge distillation, and the reported results indicate improvements in ranking accuracy and more equitable item outcomes relative to CF-aligned cold-start baselines.

1. Conceptual basis and departure from CF-space alignment

In a collaborative filtering recommender system, the interaction matrix is

$$\mathbf{R} \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|},$$

where $\mathbf{R}_{ui}=1$ denotes an interaction between user $u$ and item $i$. The cold-start difficulty arises because new items have no historical interactions, so a CF model cannot assign them meaningful item embeddings or scores.

A common solution is to train a cold-start model that maps auxiliary item content—such as images, text, audio, or video—into the embedding space of a warm CF model. The SEMCo formulation rejects this strategy as fundamentally limited by an information gap between content and CF signals. In the stated argument, CF embeddings encode behavioral patterns, including co-consumption, popularity, feedback loops, exposure, and platform-specific biases, whereas content features encode semantic attributes. These are not equivalent sources of information. Many behavioral properties are not present in content, and a content encoder asked to imitate CF embeddings is thereby asked to reconstruct popularity bias and other CF-specific artifacts using content alone.

SEMCo therefore adopts what the authors call purely content-based modeling of cold items. No CF item or user embeddings are used as targets; no mapping into CF space is performed; and the learned latent space is a content space optimized for recommendation rather than a CF space. User–item interactions are still used, but only as supervision for learning a content representation in which similarity tracks co-preference. This suggests a deliberate decoupling between recommendation supervision and CF latent structure: the model exploits interaction data without inheriting a pre-trained CF geometry.

2. Item–item formulation and multimodal encoder

SEMCo reformulates cold-start prediction through item–item similarity. Let

$$\mathbf{B} \in \mathbb{R}^{|\mathcal{I}|\times|\mathcal{I}|}$$

model item–item relations, so that predicted user preferences are written as

$\mathbf{R}\mathbf{B}.$

Rather than learning $\mathbf{B}$ directly from $\mathbf{R}$, SEMCo factorizes it through a content encoder:

$$\mathbf{B} = \mathbf{Y}\mathbf{Y}^\top, \qquad \mathbf{Y} = f(\mathbf{X}_1,\dots,\mathbf{X}_M) \in \mathbb{R}^{|\mathcal{I}|\times d}.$$

Each row of $$\mathbf{R} \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|},$$0 is an item embedding derived entirely from content. The resulting preference scores become

$$\mathbf{R} \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|},$$1

This gives a concrete interpretation of users and items in the latent content space. The user embedding matrix is $$\mathbf{R} \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|},$$2, so each user vector is an interaction-weighted average of the content embeddings of consumed items. The predicted score for user $$\mathbf{R} \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|},$$3 and item $$\mathbf{R} \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|},$$4 is the dot product between the row of $$\mathbf{R} \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|},$$5 associated with $$\mathbf{R} \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|},$$6 and the item embedding $$\mathbf{R} \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|},$$7. For new items, the encoder can compute a fresh content embedding directly, and scoring requires only a dot product with $$\mathbf{R} \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|},$$8.

The encoder operates on $$\mathbf{R} \in \{0,1\}^{|\mathcal{U}|\times|\mathcal{I}|},$$9 content modalities represented by vectors $\mathbf{R}_{ui}=1$0, assumed to be outputs of pre-trained feature extractors such as BERT, ViT, or MusicFM. The architecture has three stages. First, each modality is passed through a single linear layer followed by batch normalization and ReLU to produce $\mathbf{R}_{ui}=1$1. Second, the modality representations are concatenated,

$\mathbf{R}_{ui}=1$2

and a two-layer MLP with hidden ReLU and final softmax yields attention weights $\mathbf{R}_{ui}=1$3. These are used to form

$\mathbf{R}_{ui}=1$4

Third, a final linear layer is applied and the result is L2-normalized:

$\mathbf{R}_{ui}=1$5

Because of the normalization, cosine similarity and dot product are equivalent. In this space, item similarity is $\mathbf{R}_{ui}=1$6, and the user–item score is the average, over items previously consumed by the user, of item–item similarities to the candidate item. Training is therefore naturally cast as contrastive learning over a recommendation-oriented content geometry.

3. Sampled entmax objective and sparsity mechanism

SEMCo is a sampled classification objective over in-batch negatives in which softmax is replaced by $\mathbf{R}_{ui}=1$7-entmax. For logits $\mathbf{R}_{ui}=1$8, $\mathbf{R}_{ui}=1$9-entmax is defined as

$u$0

where $u$1 is the probability simplex and $u$2 is the Tsallis $u$3-entropy with $u$4. When $u$5, the activation reduces to softmax; when $u$6, it becomes sparsemax. For $u$7, the solution has the thresholded closed form

$u$8

with threshold $u$9.

The associated Fenchel–Young loss with temperature $i$0 is

$i$1

and its gradient with respect to logits is

$i$2

SEMCo applies this in a standard in-batch negative sampling regime. A batch $i$3 is formed from positive interactions $i$4 with $i$5. For each user $i$6, the items in the batch constitute the candidate set: one positive item and the remainder as negatives. If the positive item occupies batch index $i$7, the target is the one-hot vector $i$8, the logits $i$9 are the cosine similarities between the user embedding and each item embedding in the batch, and the SEMCo loss is

$$\mathbf{B} \in \mathbb{R}^{|\mathcal{I}|\times|\mathcal{I}|}$$0

The defining property of the method is sparsity for $$\mathbf{B} \in \mathbb{R}^{|\mathcal{I}|\times|\mathcal{I}|}$$1. If an item's logit falls below the entmax threshold, its output probability becomes exactly zero. For negatives in a one-hot classification target, that also yields zero gradient. In the authors’ formulation, this enables zeroing gradients for uninformative negatives and produces sharper estimation of item relevance (Meehan et al., 14 Apr 2026). The paper evaluates $$\mathbf{B} \in \mathbb{R}^{|\mathcal{I}|\times|\mathcal{I}|}$$2 (Softmax), $$\mathbf{B} \in \mathbb{R}^{|\mathcal{I}|\times|\mathcal{I}|}$$3 (Entmax), and $$\mathbf{B} \in \mathbb{R}^{|\mathcal{I}|\times|\mathcal{I}|}$$4 (Sparsemax), and reports that Entmax and Sparsemax consistently outperform softmax SEMCo across datasets, while Sparsemax gives the strongest sparsity and the best overall ranking performance in the online distillation setting.

4. Optimization details and content-space distillation

The optimization setup is deliberately simple. The optimizer is Adam. The embedding dimension is fixed at $$\mathbf{B} \in \mathbb{R}^{|\mathcal{I}|\times|\mathcal{I}|}$$5 for SEMCo and baselines, the hidden size in the content encoder is 192, the batch size is 2048 interactions, the number of training epochs is 15, and the learning rate is 0.001 with cosine decay to 0. L2 regularization is tuned by grid search. For $$\mathbf{B} \in \mathbb{R}^{|\mathcal{I}|\times|\mathcal{I}|}$$6, the implementation sets $$\mathbf{B} \in \mathbb{R}^{|\mathcal{I}|\times|\mathcal{I}|}$$7 to improve training stability; for softmax, $$\mathbf{B} \in \mathbb{R}^{|\mathcal{I}|\times|\mathcal{I}|}$$8. The entmax and sparsemax computations add about 10–20% overhead relative to softmax.

The model is extended with knowledge distillation in the same content space rather than from a CF teacher. The teacher is a higher-capacity content encoder with hidden and output sizes 384 and output embeddings $$\mathbf{B} \in \mathbb{R}^{|\mathcal{I}|\times|\mathcal{I}|}$$9, where $\mathbf{R}\mathbf{B}.$0. The student is a smaller content encoder producing embeddings in $\mathbf{R}\mathbf{B}.$1 with $\mathbf{R}\mathbf{B}.$2. Distillation targets item–item similarity distributions rather than user–item scores.

Teacher embeddings are first projected to the student dimension through a two-layer MLP with hidden ReLU. For a batch of items $\mathbf{R}\mathbf{B}.$3, teacher and student cosine similarities are computed between each anchor item and the items in $\mathbf{R}\mathbf{B}.$4. The teacher similarity logits are passed through $\mathbf{R}\mathbf{B}.$5-entmax to form a sparse target, and the student is trained with

$\mathbf{R}\mathbf{B}.$6

The overall objective is

$\mathbf{R}\mathbf{B}.$7

Two variants are described. In SEMCo-Offline, the teacher is trained first and the student is then trained with the combined objective. In SEMCo-Online, teacher and student are trained simultaneously, and the student distills from an exponential moving average of the teacher parameters in a mean-teacher-style arrangement. The reported interpretation is that sparse entmax supervision focuses the student on the most meaningful similarities; this suggests that distillation is used not only for compression but also for structuring the student’s neighborhood geometry.

5. Empirical evaluation, baselines, and ablations

Experiments are conducted on four datasets: Clothing, Electronics, Microlens, and Music4All-Onion. The content modalities are dataset-specific: Clothing uses only text; Electronics uses image and text; Microlens uses image, text, and video; and Music4All-Onion uses audio and text. All content features are pre-extracted. The cold-start protocol withholds 20% of items as cold, split 50/50 into validation and test, while the remaining interactions are further split 80/10/10 for training, validation, and test.

Evaluation on cold items uses Recall@20, NDCG@20, MDG@20, and Gini diversity. Results are averaged over five runs at tuned hyperparameters. The principal baselines are CLCRec, ALDI, GAR, GoRec, and Heater. These are characterized as CF-aligned content models, whereas softmax SEMCo serves as a content-only baseline against the sparse variants.

The reported quantitative findings are consistent across the four datasets. SEMCo variants outperform all cold-start baselines in $\mathbf{R}\mathbf{B}.$8, $\mathbf{R}\mathbf{B}.$9, and $\mathbf{B}$0 (Meehan et al., 14 Apr 2026). The paper gives several representative gains over the best baseline GoRec: on Electronics, $\mathbf{B}$1 rises from 0.0376 to 0.0526 for SEMCo-Online Sparsemax, and $\mathbf{B}$2 rises from 0.0092 to 0.0214; on Music4All-Onion, $\mathbf{B}$3 rises from 0.0557 for CLCRec to 0.0887 for SEMCo-Online Sparsemax. All improvements except a few borderline cases are reported as statistically significant under a paired $\mathbf{B}$4-test with $\mathbf{B}$5.

The ablation results isolate several effects. Softmax SEMCo is already strong, often outperforming CF-aligned baselines. Entmax and Sparsemax generally improve over softmax. In the base SEMCo configuration, either Entmax or Sparsemax is best depending on the dataset; in SEMCo-Online, Sparsemax yields the best ranking accuracy across all datasets. Distillation also helps: both SEMCo-Offline and SEMCo-Online generally improve over base SEMCo, and SEMCo-Online Sparsemax produces the strongest overall results, including an improvement of up to 14.9% over base SEMCo Sparsemax in Music4All-Onion $\mathbf{B}$6.

The sparsity analysis is especially notable in distillation. Sparsemax produces fewer than 10% non-zero entries in the teacher and student probability matrices, whereas Entmax produces over 70% non-zero entries. A plausible implication is that the stronger pruning of active neighbors provides a more focused supervisory signal during online distillation.

6. Equity of item outcomes, practical use, and limitations

The paper emphasizes that SEMCo can produce more equitable outcomes across the item population without explicit fairness regularization. The proposed mechanism is structural rather than post hoc: because SEMCo does not align content embeddings to a popularity-skewed CF space, it avoids directly reproducing CF popularity bias in cold-item representations. The relevant fairness metrics are MDG@20, which is framed as item-oriented accuracy, and Gini diversity, which measures inequality in item exposure.

Reported results indicate substantial gains on both axes. The paper states improvements of up to 133.7% in MDG@20 on Electronics and up to 148.7% in Gini diversity on Music4All-Onion (Meehan et al., 14 Apr 2026). In the Electronics case study, GoRec, GoRec with magnitude scaling bias mitigation, and SEMCo-Online are compared through per-item MDG@20 and recommendation counts. All methods leave many items unrecommended, but GoRec exhibits sharp exposure concentration and about 10% of items are never recommended, whereas SEMCo-Online distributes predictions more evenly and nearly all items are recommended at least 10 times. The interpretation offered is that purely content-based modeling acts as an implicit bias mitigation strategy.

The practical guidance in the study is correspondingly specific. SEMCo is positioned as preferable when fair exposure for new, long-tail, or niche items matters; when the CF backbone embeds strong popularity or interaction biases that should not be imposed on cold items; or when the objective is to generalize across items by content similarity rather than to mimic behavioral embeddings. For $\mathbf{B}$7, softmax is the straightforward baseline, Entmax $\mathbf{B}$8 offers moderate sparsity, and Sparsemax $\mathbf{B}$9 is recommended in the online distillation setting. The use of in-batch negatives with large batch size, such as 2048, is presented as sufficient without additional negative-sampling heuristics.

The limitations are also explicit. Computing user embeddings as $\mathbf{R}$0 over the full catalog may be heavy at large scale; a practical remedy noted in the paper is to restrict computation to active items. Online distillation introduces additional hyperparameters and can be less stable than offline distillation. The method assumes reliable, pre-trained item representations for all modalities and does not address missing-modality scenarios. Future directions named in the paper include extending the framework to missing modalities, leveraging sparsity further in attention or similarity computation, and exploring more advanced distillation schemes in the content space.