Efficient Item-wise SAM for Long-Tail Recommendation

• The paper introduces EISAM, an optimization framework extending SAM with frequency-aware, item-specific regularization to balance head and tail recommendations.
• It reports significant tail performance gains, such as an 8.9% improvement in NDCG@10 on MovieLens-1M, while preserving overall recommendation quality.
• EISAM employs a first-order approximation and a single weighted gradient perturbation, ensuring scalability with minimal computational overhead for LLM backbones.

Searching arXiv for the specified papers and closely related SAM/LRS long-tail work. Efficient Item-wise Sharpness-Aware Minimization (EISAM) is an optimization framework for LLM-based Recommender Systems (LRSs) that targets the long-tail problem during fine-tuning by regularizing the loss landscape at the item level rather than only globally. In the formulation introduced for sequential recommendation, EISAM extends Sharpness-Aware Minimization (SAM) with frequency-aware item-wise weighting so that rare items receive stronger flattening pressure, while retaining computational scalability for LLM backbones such as BIGRec and TALLRec built on Llama2-7B (Zhang et al., 13 Mar 2026).

1. Long-tail structure in LLM-based recommendation

Real-world recommender systems often exhibit a long-tail distribution in which a small number of head items account for a large fraction of interactions, while most tail items are rarely observed. In LRSs, this imbalance appears in two distinct forms. The first is prior long-tail, inherited implicitly from the LLM pretraining corpus, because some entities or concepts occur much more frequently in general text. The second is data long-tail, originating from the skewed recommendation dataset used for fine-tuning (Zhang et al., 13 Mar 2026).

The empirical study underlying EISAM examined two representative LRS backbones, BIGRec and TALLRec, on MovieLens-1M, Steam, and Amazon Digital Music. The reported findings were that data long-tail is the dominant factor driving head-tail performance gaps; prior head items can amplify head performance when they coincide with data head; and prior head status does little to help true data-tail items. This leads directly to the design premise that any method intended to improve tail recommendation quality must address the data long-tail during fine-tuning rather than relying on the pretrained LLM prior alone (Zhang et al., 13 Mar 2026).

Two technical difficulties motivate the item-wise formulation. Tail items provide sparse training signals, so their local loss landscapes are poorly sampled, which is associated with sharp minima and poor generalization. At the same time, one-size-fits-all regularization, exemplified by vanilla SAM, flattens the global loss but cannot allocate additional flattening capacity to the rare items that need it most.

2. Formalization of the EISAM objective

EISAM is defined over an item set $I=\{i_1,\dots,i_{|I|}\}$ and training sequences $S=\{(s_j,i_j)\}_{j=1}^N$, where $s_j$ is a sequence prefix and $i_j$ is the next-item label. The standard empirical loss is

$$L_S(\theta)=\frac{1}{N}\sum_{(s,i)\in S}\ell(\theta;s,i).$$

For each item $i$, the method defines a per-item sample set $S(i)=\{(s,i)\in S\}$ and a per-item empirical loss

$$L_{S^{(i)}}(\theta)=\frac{1}{|S(i)|}\sum_{(s,i)\in S(i)}\ell(\theta;s,i).$$

If $q_i=|S(i)|/N$ denotes the empirical frequency of item $i$, then EISAM uses a decreasing weighting function $S=\{(s_j,i_j)\}_{j=1}^N$0 to emphasize rare items, with the example $S=\{(s_j,i_j)\}_{j=1}^N$1. This weighting is central: head items contribute less to the sharpness regularizer, while tail items contribute more (Zhang et al., 13 Mar 2026).

For a perturbation $S=\{(s_j,i_j)\}_{j=1}^N$2 with $S=\{(s_j,i_j)\}_{j=1}^N$3, the item-wise sharpness of item $S=\{(s_j,i_j)\}_{j=1}^N$4 is

$S=\{(s_j,i_j)\}_{j=1}^N$5

EISAM aggregates these item-wise sharpness terms as

$S=\{(s_j,i_j)\}_{j=1}^N$6

and adds the resulting penalty to the standard loss:

$S=\{(s_j,i_j)\}_{j=1}^N$7

where $S=\{(s_j,i_j)\}_{j=1}^N$8 controls the trade-off between empirical risk and item-wise sharpness regularization. In effect, the objective preserves the SAM intuition of seeking flatter minima, but the flattening pressure is redistributed according to item frequency rather than applied uniformly.

3. First-order approximation and batch update rule

The inner maximization in the EISAM objective is intractable if solved exactly. The paper therefore applies a first-order Taylor expansion to each $S=\{(s_j,i_j)\}_{j=1}^N$9 around $s_j$0, yielding

$s_j$1

Substituting this into the sharpness term gives

$s_j$2

where

$s_j$3

is the weighted sum of per-item gradients. Under an $s_j$4-ball constraint, the maximizer has the closed form

$s_j$5

Following the usual SAM practice, second-order terms in the outer gradient are dropped. The reported implementation computes the gradient of $s_j$6 with three back-propagations per batch. First, it computes $s_j$7 at $s_j$8 to form $s_j$9. Second, it computes $i_j$0 at $i_j$1. Third, it perturbs $i_j$2 by $i_j$3 and computes $i_j$4. The final update is

$i_j$5

The batch-level algorithm is presented in a directly implementable form. Given a batch $i_j$6, radius $i_j$7, weight function $i_j$8, regularization strength $i_j$9, and learning rate $$L_S(\theta)=\frac{1}{N}\sum_{(s,i)\in S}\ell(\theta;s,i).$$0, one computes the unweighted loss $$L_S(\theta)=\frac{1}{N}\sum_{(s,i)\in S}\ell(\theta;s,i).$$1 and weighted loss $$L_S(\theta)=\frac{1}{N}\sum_{(s,i)\in S}\ell(\theta;s,i).$$2, forms $$L_S(\theta)=\frac{1}{N}\sum_{(s,i)\in S}\ell(\theta;s,i).$$3, constructs $$L_S(\theta)=\frac{1}{N}\sum_{(s,i)\in S}\ell(\theta;s,i).$$4, evaluates the perturbed weighted gradient, and updates parameters using the combined gradient. This design avoids explicit per-item or per-group SAM loops even though the regularization target is item-specific (Zhang et al., 13 Mar 2026).

4. Generalization analysis

The theoretical analysis introduces $$L_S(\theta)=\frac{1}{N}\sum_{(s,i)\in S}\ell(\theta;s,i).$$5 as the original training distribution and $$L_S(\theta)=\frac{1}{N}\sum_{(s,i)\in S}\ell(\theta;s,i).$$6 as the ideal uniform test distribution in which each item has equal probability. It defines the population analogues $$L_S(\theta)=\frac{1}{N}\sum_{(s,i)\in S}\ell(\theta;s,i).$$7 and $$L_S(\theta)=\frac{1}{N}\sum_{(s,i)\in S}\ell(\theta;s,i).$$8, together with

$$L_S(\theta)=\frac{1}{N}\sum_{(s,i)\in S}\ell(\theta;s,i).$$9

Under standard boundedness and Bernstein-type arguments, the paper states that, with high probability over $i$0,

$i$1

where $i$2, $i$3, and $i$4 is the Hessian of $i$5 (Zhang et al., 13 Mar 2026).

The distinctive term is the negative curvature bonus,

$i$6

which tightens the bound beyond vanilla SAM by penalizing large curvature in directions weighted toward rare items. The stated interpretation is that, as $i$7 grows, the $i$8 terms vanish faster, while the curvature bonus yields a faster-decreasing bound than standard global SAM. This does not merely restate flat-minimum intuition; it connects item-frequency-aware curvature control to balanced-item generalization, which is the evaluation regime most relevant to long-tail recommendation.

5. Computational profile and scaling behavior

EISAM is designed to remain feasible at LLM scale. Each iteration requires three gradient computations per batch, reported as identical to standard SAM in this setting, but the weighted gradients are computed in aggregate as a single vector $i$9 rather than through per-group or per-item loops. The method therefore concentrates computational overhead in one weighted perturbation direction rather than in repeated subgroup-specific adversarial updates (Zhang et al., 13 Mar 2026).

The measured overhead relative to vanilla training is roughly $S(i)=\{(s,i)\in S\}$0–$S(i)=\{(s,i)\in S\}$1, reported as $S(i)=\{(s,i)\in S\}$2 on MovieLens-1M and approximately $S(i)=\{(s,i)\in S\}$3 on Steam and Amazon Digital Music. Relative to a naïve group-wise SAM baseline, GroupSAM, EISAM is approximately $S(i)=\{(s,i)\in S\}$4 faster. Memory overhead is described as minimal because the implementation stores only one additional $S(i)=\{(s,i)\in S\}$5 vector and reuses standard backpropagation buffers. The method is also reported to be compatible with LoRA and other parameter-efficient fine-tuning schemes.

The computational significance lies in the fact that long-tail-aware sharpness regularization is made practical without abandoning LLM backbones. A plausible implication is that the method’s main scalability advantage comes from replacing explicit subgroup decomposition with a single weighted aggregate gradient, which is precisely the design choice that separates it from group-wise SAM variants.

6. Experimental findings, interpretation, and relation to adjacent SAM variants

The empirical evaluation uses three real-world datasets:

Dataset	Scale	Additional note
MovieLens-1M	3,883 items; 1M ratings	80/20 head/tail split
Steam	32K items; 1.3M interactions	—
Amazon Digital Music	266K items; 0.8M interactions	—

The LRS backbones are TALLRec and BIGRec on Llama2-7B, and the baselines are Re-weighting (RW), standard SAM, and GroupSAM. Evaluation uses top-10 NDCG@10 and Hit Ratio HR@10 over Overall, Tail, and Head subsets (Zhang et al., 13 Mar 2026).

Averaged across datasets and backbones, the reported results are that EISAM improves Overall NDCG@10 by $S(i)=\{(s,i)\in S\}$6 over the best baseline, improves Tail NDCG@10 by $S(i)=\{(s,i)\in S\}$7, and maintains head performance within $S(i)=\{(s,i)\in S\}$8. On MovieLens-1M, the absolute tail NDCG is reported to increase from approximately $S(i)=\{(s,i)\in S\}$9 to approximately $$L_{S^{(i)}}(\theta)=\frac{1}{|S(i)|}\sum_{(s,i)\in S(i)}\ell(\theta;s,i).$$0, with similar gains on the other datasets. Standard SAM improves overall performance but yields negligible tail gains, whereas GroupSAM helps tail performance but often hurts overall performance and incurs $$L_{S^{(i)}}(\theta)=\frac{1}{|S(i)|}\sum_{(s,i)\in S(i)}\ell(\theta;s,i).$$1 greater runtime. EISAM is therefore characterized as achieving both strong overall gains and substantial tail improvements at low overhead (Zhang et al., 13 Mar 2026).

Loss-landscape visualizations, including 3D and contour plots, are reported to show markedly flatter loss surfaces around tail-item parameters under EISAM than under standard SAM. Hyperparameter studies indicate that performance peaks at moderate $$L_{S^{(i)}}(\theta)=\frac{1}{|S(i)|}\sum_{(s,i)\in S(i)}\ell(\theta;s,i).$$2–$$L_{S^{(i)}}(\theta)=\frac{1}{|S(i)|}\sum_{(s,i)\in S(i)}\ell(\theta;s,i).$$3 and degrades when $$L_{S^{(i)}}(\theta)=\frac{1}{|S(i)|}\sum_{(s,i)\in S(i)}\ell(\theta;s,i).$$4, while tail gains improve up to $$L_{S^{(i)}}(\theta)=\frac{1}{|S(i)|}\sum_{(s,i)\in S(i)}\ell(\theta;s,i).$$5–$$L_{S^{(i)}}(\theta)=\frac{1}{|S(i)|}\sum_{(s,i)\in S(i)}\ell(\theta;s,i).$$6 before over-emphasis on tail items harms head performance.

Two misconceptions are addressed directly by these results. First, long-tail behavior in LRSs is not presented as a purely pretrained-prior phenomenon; the empirical study reports that data long-tail remains the dominant factor, especially on the tail. Second, global flatness regularization is not sufficient for balanced recommendation quality; standard SAM improves overall metrics but does not materially resolve tail-item underperformance. The method is thus positioned as the first systematic solution to the long-tail problem in LRSs through item-wise sharpness regularization (Zhang et al., 13 Mar 2026).

A separate nearby line of work, “Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization” (Deng et al., 2024), addresses SAM efficiency through probabilistic subset selection based on a gradient-norm proxy derived from the Difference in Loss before and after perturbation. That work concerns speedup of SAM across classification, pose estimation, and quantization, and not the long-tail optimization problem in LLM-based recommendation. This suggests that EISAM, in the recommender setting, should be understood specifically as an item-wise frequency-aware sharpness framework rather than as a generic sampling-based SAM accelerator.