Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 65 tok/s
Gemini 2.5 Pro 40 tok/s Pro
GPT-5 Medium 26 tok/s Pro
GPT-5 High 24 tok/s Pro
GPT-4o 113 tok/s Pro
Kimi K2 200 tok/s Pro
GPT OSS 120B 445 tok/s Pro
Claude Sonnet 4.5 34 tok/s Pro
2000 character limit reached

Positional encoding is not the same as context: A study on positional encoding for sequential recommendation (2405.10436v2)

Published 16 May 2024 in cs.IR and cs.AI

Abstract: The rapid growth of streaming media and e-commerce has driven advancements in recommendation systems, particularly Sequential Recommendation Systems (SRS). These systems employ users' interaction histories to predict future preferences. While recent research has focused on architectural innovations like transformer blocks and feature extraction, positional encodings, crucial for capturing temporal patterns, have received less attention. These encodings are often conflated with contextual, such as the temporal footprint, which previous works tend to treat as interchangeable with positional information. This paper highlights the critical distinction between temporal footprint and positional encodings, demonstrating that the latter offers unique relational cues between items, which the temporal footprint alone cannot provide. Through extensive experimentation on eight Amazon datasets and subsets, we assess the impact of various encodings on performance metrics and training stability. We introduce new positional encodings and investigate integration strategies that improve both metrics and stability, surpassing state-of-the-art results at the time of this work's initial preprint. Importantly, we demonstrate that selecting the appropriate encoding is not only key to better performance but also essential for building robust, reliable SRS models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. A simple and effective positional encoding for transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2974–2988, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  2. On the properties of neural machine translation: Encoder-decoder approaches.
  3. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy. Association for Computational Linguistics.
  4. The youtube video recommendation system. In Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ’10, page 293–296, New York, NY, USA. Association for Computing Machinery.
  5. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255.
  6. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  7. Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations. ACM Trans. Inf. Syst., 39(1).
  8. Deepfm: A factorization-machine based neural network for ctr prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, page 1725–1731. AAAI Press.
  9. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778.
  10. Vista: A visually, socially, and temporally-aware model for artistic recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems, RecSys ’16. ACM.
  11. Translation-based recommendation. In Proceedings of the Eleventh ACM Conference on Recommender Systems, RecSys ’17, page 161–169, New York, NY, USA. Association for Computing Machinery.
  12. Ruining He and Julian McAuley. 2016a. Fusing similarity models with markov chains for sparse sequential recommendation. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 191–200.
  13. Ruining He and Julian McAuley. 2016b. Vbpr: visual bayesian personalized ranking from implicit feedback. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, page 144–150. AAAI Press.
  14. Session-based recommendations with recurrent neural networks.
  15. Explainable fashion recommendation: A semantic attribute region guided approach. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 4681–4688. International Joint Conferences on Artificial Intelligence Organization.
  16. Wang-Cheng Kang and Julian McAuley. 2018a. Self-attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM), pages 197–206.
  17. Wang-Cheng Kang and Julian McAuley. 2018b. Self-attentive sequential recommendation.
  18. Time interval aware self-attention for sequential recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining, WSDM ’20, page 322–330, New York, NY, USA. Association for Computing Machinery.
  19. Neural attentive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, page 1419–1428, New York, NY, USA. Association for Computing Machinery.
  20. Caesar: context-aware explanation based on supervised attention for service recommendations. Journal of Intelligent Information Systems, 57(1):147–170. Publisher Copyright: © 2020, Springer Science+Business Media, LLC, part of Springer Nature. Copyright: Copyright 2020 Elsevier B.V., All rights reserved.
  21. Amazon.com recommendations: item-to-item collaborative filtering. IEEE Internet Computing, 7(1):76–80.
  22. Rastislav Papso. 2023. Complementary product recommendation for long-tail products. In Proceedings of the 17th ACM Conference on Recommender Systems, RecSys ’23, page 1305–1311, New York, NY, USA. Association for Computing Machinery.
  23. David Picard. 2021. Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision.
  24. Context and attribute-aware sequential recommendation via cross-attention. In Proceedings of the 16th ACM Conference on Recommender Systems, RecSys ’22, page 71–80, New York, NY, USA. Association for Computing Machinery.
  25. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464–468, New Orleans, Louisiana. Association for Computational Linguistics.
  26. Harald Steck. 2019. Embarrassingly shallow autoencoders for sparse data. CoRR, abs/1905.03375.
  27. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.
  28. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM ’19, page 1441–1450, New York, NY, USA. Association for Computing Machinery.
  29. Personalized top-n sequential recommendation via convolutional sequence embedding. CoRR, abs/1809.07426.
  30. Trinh Xuan Tuan and Tu Minh Phuong. 2017. 3d convolutional networks for session-based recommendation with content features. In Proceedings of the Eleventh ACM Conference on Recommender Systems, RecSys ’17, page 138–146, New York, NY, USA. Association for Computing Machinery.
  31. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
  32. A survey on session-based recommender systems. ACM Computing Surveys, 2021:39.
  33. Sse-pt: Sequential recommendation via personalized transformer. In Proceedings of the 14th ACM Conference on Recommender Systems, RecSys ’20, page 328–337, New York, NY, USA. Association for Computing Machinery.
  34. Cfm: Convolutional factorization machines for context-aware recommendation. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 3926–3932. International Joint Conferences on Artificial Intelligence Organization.
  35. A dynamic recurrent model for next basket recommendation. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’16, page 729–732, New York, NY, USA. Association for Computing Machinery.
  36. A simple convolutional generative network for next item recommendation. Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining.
  37. Next item recommendation with self-attentive metric learning.
  38. Length extrapolation of transformers: A survey from the perspective of positional encoding.
  39. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM ’20, page 1893–1902, New York, NY, USA. Association for Computing Machinery.

Summary

  • The paper demonstrates that effective positional encoding can significantly enhance the stability and performance of sequential recommendation systems, especially in sparse datasets.
  • It compares various methods including absolute, relative (e.g., RMHA-4), learnable, and rotary encodings to determine which yields optimal results under different data conditions.
  • The research suggests that dynamically adapting encoding strategies to dataset sparsity and user interaction density could further improve recommendation quality.

The Ins and Outs of Positional Encodings in Sequential Recommendation Systems

What's the Deal with Sequential Recommendation Systems?

We're seeing an explosion in streaming media and e-commerce, and you might have noticed how good those recommendation systems are at showing you what you want to see next. Sequential Recommendation Systems (SRS) are at the heart of this magic. They track and predict user behaviors based on past interactions, showing you that next favorite show or must-have gadget.

Researchers have been fiddling with architectural improvements like transformer blocks, but what really caught attention is how positional encodings—the way we represent the order of items—can affect these systems. This paper dives headfirst into evaluating various positional encodings, ensuring to highlight why they’re more important than they might seem.

Types of Positional Encodings

Understanding positional encodings is key to grasping the paper's findings. Broadly, the paper discusses two primary kinds:

  1. Absolute Positional Encoding (APE): Adds fixed values to items based on their positions. Think of it like giving every item a unique but consistent badge based on its place in line.
  2. Relative Positional Encoding (RPE): Provides information on one item relative to others. This is more like telling you how far item A is from item B, no matter where they stand overall.

The authors also explored various other encoding types, including learnable and rotary encodings, adding more flavors to the mix. Here’s a quick breakdown of the encodings discussed:

  • Learnable Encoding: These are not fixed but learned during training, making them more adaptable.
  • Rotary Encoding: Adds a rotational transformation to enhance positional representation.
  • Concatenation Variants: Combine positional encodings differently within the architecture to test stability and performance.

Key Findings

  1. Stability Matters: One of the more surprising findings was how much stability varies by encoding type. For instance, relative encodings (dubbed RMHA-4 in the paper) provided much higher stability, particularly in datasets with high sparsity.
  2. Performance Highlights: Using the right positional encoding can push the performance to new heights. Some encodings not only reached new performance benchmarks but did so with consistent results over multiple runs.
  3. Sparsity Awareness: The researchers found that the dataset's sparsity heavily influences which encoding works best. Datasets with fewer interactions benefited more from highly stable encodings like RMHA-4.

Practical Insights

Imagine you're implementing an SRS for an e-commerce platform. This research suggests:

  • For sparse datasets (think niche product categories with fewer interactions), relative positional encodings can provide the stability required to generate reliable recommendations.
  • If your data is denser (like a mainstream video streaming service), rotational or learnable encodings might be more beneficial for squeezing out the best performance.

Numerical Results

Best Encodings in Different Scenarios:

  • High Stability: RMHA-4 encodings excelled with high stability for sparsely populated datasets, preventing the wild swings in model performance that can occur with more volatile methods.

Top Performance Achievers:

  • Abs + Con and Learnt encodings reached top HIT@10 values in certain datasets, but with less stability, making them less reliable without rigorous testing and tuning.

Theoretical Implications

This paper encourages a shift in how positional encodings are perceived in the field of SRS. Instead of being an afterthought, they should be considered fundamental to achieving optimal and stable performance.

The paper’s insights also pave the way for future research directions:

  • Dynamic Encoding Strategies: Could a hybrid approach that adapts encoding strategies according to real-time sparsity and user interaction density outshine static methods?
  • Bound Sensitivity: The impact of upper bounds on encoding vectors still holds untapped potential and warrants deeper exploration.

Wrapping It Up

Recommendation systems aren't a one-size-fits-all model. This paper reminds us that the minutiae—like how you encode position information—can significantly impact the overall performance and reliability of your system. Whether you're dealing with sparse datasets or dense ones, having the right positional encoding technique can make all the difference in the usefulness of your recommendations.

The journey of positional encodings in SRS is just beginning, and as we refine these methods, we can expect even smarter and more intuitive recommendation systems in the future. Until next time, happy recommending!

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 7 posts and received 101 likes.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up to View