On the Memorization Behavior of LLMs in Generative Recommendation: Observations, Implications, and Training Strategies

Abstract: Generative recommendation (GR) has emerged as a promising direction for recommender systems. Recently, LLMs have been increasingly adopted for GR, as their rich pretrained knowledge is expected to help them generalize beyond common user behavior patterns that traditional memorization-oriented baselines can capture. However, existing LLM-based GR works largely ignore LLMs' well-known tendency to memorize, which, if present in LLMs fine-tuned for GR, would restrict their utilization of pretrained knowledge. In this work, we investigate this concern by examining one-hop memorization, where a model recommends items that are direct successors of items in the training data. We show that LLMs do this more than non-LLM-based GR models-in fact, the vast majority of their gains over GR baselines are actually on users whose target items can be predicted through one-hop memorization. We intuit that improving performance on the remaining users requires LLMs to learn richer item-item relations beyond one-hop transitions. To achieve this, we propose IIRG, a novel training strategy that teaches LLMs to capture: (1) collaborative relations derived from item co-occurrences across multiple hops in user sequences, and (2) semantic relations among items with similar themes, both of which can serve as useful recommendation signals. We show that IIRG significantly improves over LLMs trained solely with standard next-item prediction, with especially large gains for users whose test items are not covered by train-time one-hop transitions.

• The paper identifies that LLMs in generative recommendation primarily rely on one-hop memorization rather than semantic generalization.
• The paper introduces IIRG, a training strategy that augments next-item prediction with collaborative and semantic neighbor generation.
• The paper demonstrates that IIRG improves recommendation accuracy by up to 21% overall and 50% for non-memorization-benefiting users.

Memorization Behavior of LLMs in Generative Recommendation: Analysis and Training Methodology

Introduction

LLMs have been increasingly adopted for generative recommendation (GR) tasks, propelled by their rich, pretrained world knowledge and scalable sequence modeling capacity. This adoption presumes that LLMs will generalize beyond the repetitive, localized patterns typically leveraged by traditional recommendation models. However, the core contribution of this work is to empirically challenge this assumption: LLMs fine-tuned for GR exhibit pronounced one-hop memorization, primarily recommending items that directly succeed observed items in the training data. Furthermore, the study proposes IIRG, an item-item relation generation strategy, to explicitly expand the relational capacity of LLMs in GR beyond immediate co-occurrences.

Empirical Analysis of Memorization in LLM-based GR

One-Hop Memorization Characterization

The study rigorously formalizes and quantifies the one-hop memorization phenomenon. In the sequential GR paradigm, a model’s output is regarded as one-hop memorization if it predicts an item as a recommendation purely because that item immediately succeeds a user’s previously interacted item in the training data. Extensive experiments contrast LLM-based (Qwen-3.5 family) and non-LLM-based (TIGER) models, showing LLMs exhibit systematically higher one-hop memorization ratios in their top-k recommendation outputs, invariant to model regularization or scaling.

Notably, standard interventions such as increased regularization, model downsizing, or partial-parameter fine-tuning (e.g., LoRA) are ineffective at reducing this reliance—pointing to fundamental limitations in LLM fine-tuning when applied naively to GR tasks.

User Group Performance Disparity

By partitioning users into two cohorts—those whose test items are covered by training set one-hop transitions and those whose are not—the analysis demonstrates that virtually all gains of LLM-based GR over baselines are restricted to the former group. The implication is that LLMs are not leveraging semantic or collaborative signals via generalization, but instead their apparent improvement is a direct consequence of stronger sequence-level rote memorization. For the latter group (non-memorization-benefiting users), LLMs deliver only marginal or null improvement over conventional models.

The IIRG Training Strategy: Modeling Richer Item-Item Relations

Motivation

To transcend the limitation of one-hop transition-based modeling, the IIRG (Item-Item Relation Generation) approach augments the standard next-item prediction objective with two auxiliary sequence-level generation objectives:

• Collaborative Neighbor Generation: The LLM is trained to generate items that are frequently co-interacted (within a multi-hop window) with the anchor item across the user base, thereby learning collaborative signals beyond local transitions.
• Semantic Neighbor Generation: The LLM is trained to generate items with high semantic similarity (e.g., based on embedding or hierarchical keyword matching in metadata) to the anchor item, promoting the acquisition of theme- and intent-driven associations.

Both strategies are implemented via supervised prompting with corresponding positive sample construction, jointly optimized with the primary task.

Efficacy of Neighbor Signals

Empirical data analysis in the paper establishes that collaborative and semantic neighbors are both highly predictive of user behavior in out-of-one-hop-memorization settings. Their overlap with the one-hop neighbor set is low (low Jaccard similarity), confirming their non-redundancy and justifying their selection as auxiliary supervision. Ablation studies show that either neighbor generation strategy alone improves LLM recommendation accuracy, but the combination delivers strictly superior results across all benchmarks.

Experimental Results

Main Outcomes

Across three Amazon Review benchmarks (Sports, Toys, Beauty), IIRG-trained LLMs achieve, on average, a 21% improvement in Recall@5 compared to LLMs trained with next-item prediction alone, and consistently outperform strong non-LLM and LLM-based GR baselines—including models leveraging distilled reasoning, specialized identifier types (semantic or term-based IDs), and advanced auxiliary training methods.

The most pronounced relative improvements are observed in the user cohort whose target items are not reachable by training set one-hop transitions: a 50% gain for these "non-memorization-benefiting" users (compared to 17% for "memorization-benefiting" users) on the Sports dataset, and similarly strong trends in Toys and Beauty.

Mitigating One-Hop Memorization Reliance

IIRG demonstrably reduces the fraction of outputs attributable to one-hop memorization, as measured by explicit candidate set analysis. This suppression coincides with the highest gains precisely in user segments that were previously not well-served by rote memorization. Notably, neither model size nor regularization accounts for this reduction—only explicit multi-relation generation leads to such behavioral change.

Generalization over Identifier Types and Datasets

The effectiveness of IIRG generalizes across both semantic and term-based identifiers, as well as different SID/TID representations, and holds on datasets beyond Amazon reviews (e.g., Yelp). SID-based search for semantic neighbors, though more efficient, maintains much of the performance gain afforded by dense embedding retrieval.

Robustness and Ablation

Comprehensive ablation studies confirm that both collaborative and semantic neighbor generation components are necessary and mutually reinforcing. Training on random neighbors or merely using neighbor heuristics (without LLM training) yields inferior results, underscoring the necessity of learning to generate contextually relevant neighbor sets.

Theoretical and Practical Implications

This work provides a rigorous empirical refutation of the hypothesis that LLM-based GR systems, when fine-tuned naively, naturally capitalize on pretrained knowledge to learn abstract or semantic generalizations. Instead, memorization remains the dominant driver of their apparent gains. The IIRG framework demonstrates that only by explicitly supervising multi-hop collaborative and semantic relations can broader generalization be systematically and scalably coaxed from pretrained LLM architectures in the GR domain.

From a practical perspective, these findings expose risks associated with deploying LLM-based recommenders: a lack of careful multi-relation training may result in superficial improvements masking poor generalization, particularly in cold-start and long-tail scenarios. Explicitly diversifying relation modeling, as in IIRG, translates to robust gains across both high- and low-coverage user segments, including for cold-start items where data sparsity is most acute.

Future Research Directions

The underlying architectural or algorithmic causes of LLM one-hop memorization in sequential recommendation remain unidentified. Model size, parameter sharing, and training set redundancy are not sufficient explanations according to the presented analysis. Assessing whether these findings generalize to other LLM families (e.g., Llama, Mistral) and further probing the interaction between inductive biases and pretraining/fine-tuning objectives is an important area for ongoing study.

Additionally, extending multi-relation generation towards graph-based or hybrid symbolic-neural approaches, developing mechanisms for dynamic neighbor selection, and closing the gap between context-rich recommendation and generalized language-driven reasoning remain open challenges.

Conclusion

This work systematically characterizes and addresses the memorization behavior of LLMs in generative recommendation. It establishes that performance advantages over classical GR models derive primarily from stronger one-hop memorization. The proposed IIRG method, which explicitly supervises both collaborative and semantic neighbor generation, substantially improves out-of-memorization recommendation capacity and mitigates reliance on superficial sequence transitions. These findings have significant implications for the design, evaluation, and deployment of LLM-based recommender systems.