Item Encoder Representations from Transformers

• The paper introduces a transformer framework that replaces static embeddings with context-aware item representations derived from transaction history and user behavior.
• It leverages dual pre-training objectives—Masked Item Prediction and Next Basket Prediction—to capture both local co-occurrence and global sequential patterns.
• The approach demonstrates significant gains in F1@5 and NDCG@5, highlighting the benefit of candidate-conditioned history encoding in real-time recommendations.

Searching arXiv for the target paper and closely related next-basket recommendation work. Item Encoder Representations from Transformers (IERT) is a transformer-based framework for next basket recommendation that replaces static item embeddings with context-aware item representations conditioned on transaction context and user history. In the formulation used by the model, each user $u$ has a chronologically ordered sequence of baskets $B_1^u, B_2^u, \ldots, B_t^u$, where each basket is a set of purchased items, and the task is to estimate $p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$ for every catalog item and recommend the top-$K$ items, with $K=5$ in the reported experiments. The method adapts the pre-training and fine-tuning paradigm associated with bidirectional transformers to market-basket data by treating items as tokens, baskets as sentences, and a user’s basket sequence as a document (Yang et al., 2019).

1. Problem setting and representational premise

Next basket recommendation aims to predict the set of items in a user’s next basket from that user’s historical transactions. The notation used by IERT defines users as $U = \{u_1, \ldots, u_{|U|}\}$, items as $I = \{i_1, \ldots, i_{|I|}\}$, and a user history as $B^u = \{B_1^u, \ldots, B_t^u\}$, where $$B_t^u = \{i_{(t,1)}^u, i_{(t,2)}^u, \ldots, i_{(t,j)}^u\}$$. A probabilistic view scores each item by $p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$, after which the system recommends the highest-scoring items (Yang et al., 2019).

The central representational claim is that an item does not have a single invariant semantics across all transactions. Static item lookup in earlier baselines is written as

$B_1^u, B_2^u, \ldots, B_t^u$0

but such a representation is transaction-invariant. IERT is motivated by the observation that the same item can occur in different purchasing contexts and thus carry different meanings. The examples given are concrete: earphones bought with a USB cable versus many earphones bought together, or a recipe book bought with other books versus bought with tomato and olive oil. This motivates contextualized item embeddings that condition on co-occurring items, neighboring baskets, and user identity.

A plausible implication is that next basket recommendation is not merely a sequence-modeling problem over item IDs. In the IERT formulation, it is also a contextual semantics problem in which within-basket co-purchase signals and cross-basket temporal structure jointly determine the effective representation of each candidate item.

2. Context-aware item representation with a transformer encoder

IERT adapts a bidirectional transformer encoder to market-basket data. Its input representation for each tokenized item is

$B_1^u, B_2^u, \ldots, B_t^u$1

where $B_1^u, B_2^u, \ldots, B_t^u$2 is the item embedding, $B_1^u, B_2^u, \ldots, B_t^u$3 is the segment embedding, and $B_1^u, B_2^u, \ldots, B_t^u$4 is the positional embedding (Yang et al., 2019).

The sequence construction follows two distinct regimes. During pre-training, sequences of baskets are fed for each user, with items in the same basket made contiguous, basket boundaries represented by $B_1^u, B_2^u, \ldots, B_t^u$5, and $B_1^u, B_2^u, \ldots, B_t^u$6 at the beginning. During fine-tuning, the user’s historical transactions are concatenated into a single token sequence and a candidate item is appended, separated by $B_1^u, B_2^u, \ldots, B_t^u$7. The model preserves basket order across time, while also passing positional information for items within baskets even though item order inside a basket is not strictly ordered.

The encoder uses standard transformer attention: $B_1^u, B_2^u, \ldots, B_t^u$8 with multi-head attention defined as

$B_1^u, B_2^u, \ldots, B_t^u$9

$p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$0

Each transformer block consists of multi-head attention followed by feed-forward layers, with residual connections and layer normalization in the standard transformer style.

The architectural analogy to text is explicit: items are tokens, baskets are sentences, and user histories are documents. What changes is the semantics of locality. In text, adjacency usually carries syntactic and semantic order; in IERT, within-basket adjacency primarily marks co-purchase context, and cross-basket adjacency marks temporal behavior. This suggests that the transformer is used less as a LLM in the narrow sense than as a general contextual encoder over structured transactional sequences.

3. Pre-training objectives and offline training pipeline

IERT pre-trains contextualized item embeddings with two objectives. The first is Masked Item Prediction (MIM), adapted from masked language modeling. An item $p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$1 is randomly masked within basket $p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$2, and the model predicts it from the remaining items in the basket: $p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$3 This objective is intended to model within-basket co-purchase context (Yang et al., 2019).

The second objective is Next Basket Prediction (NBP), adapted from next sentence prediction. Basket pairs are constructed with 50% consecutive and 50% non-consecutive pairs, and the model predicts whether the second basket follows the first: $p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$4 where $p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$5 indicates whether the pair is truly consecutive. The total pre-training loss is

$p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$6

The offline pipeline uses user basket sequences with basket boundaries marked, masked items for MIM, and balanced consecutive versus non-consecutive basket pairs for NBP. The reported optimization setup is Adam with learning rate $p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$7, learning-rate warmup over the first 2500 steps, linear decay afterward, batch size 32, and 40,000 training steps. The model size follows a BERTBASE configuration with hidden size $p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$8, transformer blocks $p(i \in B_{t+1}^u \mid u, B_{1:t}^u)$9, and attention heads $K$0. The maximum number of items in the same basket is capped at 100, so long baskets are truncated (Yang et al., 2019).

The pre-training design separates two forms of signal. MIM injects local basket semantics, while NBP injects cross-basket temporal continuity. A plausible implication is that IERT treats the notion of “context” as inherently two-scale: local co-occurrence and global sequence order are not collapsed into a single objective but encoded as separate pre-training pressures.

4. Fine-tuning, candidate conditioning, and recommendation scoring

In the online phase, for a given user $K$1 and candidate item $K$2, IERT packs the flattened history $K$3 and the candidate into one sequence, marked by $K$4 and $K$5, and feeds it through the pre-trained transformer. This yields hidden states

$K$6

for historical items and $K$7 for the candidate item (Yang et al., 2019).

The historical transaction representation is candidate-conditioned: $K$8 with

$K$9

Here, $K=5$0 is the element-wise product, $K=5$1, and $K=5$2. The user representation is a lookup embedding: $K=5$3

The recommendation score is

$K=5$4

and the probability over the full catalog is

$K=5$5

Fine-tuning uses a weighted cross-entropy objective: $K=5$6 where $K=5$7 if item $K=5$8 is in the true next basket and $K=5$9 otherwise, and $U = \{u_1, \ldots, u_{|U|}\}$0 alleviate class imbalance.

This scoring design is notable for conditioning the summary of user history on the candidate item itself. The historical representation $U = \{u_1, \ldots, u_{|U|}\}$1 is not a static summary of past behavior; it is reweighted by $U = \{u_1, \ldots, u_{|U|}\}$2. This suggests a compatibility view of next basket recommendation: the system evaluates how a specific candidate item reorganizes the relevance of prior purchases, rather than merely comparing a candidate against a fixed user state.

5. Dataset, evaluation protocol, baselines, and reported results

The reported experiments use the public Ta-Feng dataset with 464,118 transactions, 9,238 users, and 7,793 items. Preprocessing removes items purchased by fewer than 10 users and users who purchased fewer than 10 items in total. For each user, the last basket is used for test, the penultimate basket for validation, and all earlier baskets for training. Sequences are sorted by time, baskets are linearized with $U = \{u_1, \ldots, u_{|U|}\}$3 delimiters, and baskets longer than 100 items are truncated (Yang et al., 2019).

Evaluation uses F1@5 and NDCG@5. The baselines include TOP, NMF, FPMC, HRM, DREAM, and ANAM, along with an ablation involving IERT with and without pre-training. The reported results are:

Model	F1@5	NDCG@5
TOP	0.051	0.084
NMF	0.052	0.072
FPMC	0.059	0.087
HRM	0.062	0.089
DREAM	0.133	0.173
ANAM	0.146	0.190
IERT (w/ pre-training)	0.150	0.194
IERT	0.213	0.340

The paper reports that the transformer encoder at item level already improves over baselines, with IERT (w/ pre-training) exceeding ANAM, and that full IERT with pre-training yields large gains: $U = \{u_1, \ldots, u_{|U|}\}$4 F1 and $U = \{u_1, \ldots, u_{|U|}\}$5 NDCG versus ANAM (Yang et al., 2019).

The paper also reports an ablation comparing IERT with and without pre-training and states that pre-training is crucial. Additional ablations such as masking strategy, context window size, and transformer depth are not reported. A plausible interpretation is that the largest empirical gains arise not only from substituting a transformer for earlier sequence models, but from the separation of offline context-aware pre-training and online recommendation fine-tuning.

6. Computational profile, limitations, and broader significance

IERT inherits the quadratic sequence-length cost of transformer self-attention. The encoder complexity is reported as

$U = \{u_1, \ldots, u_{|U|}\}$6

where $U = \{u_1, \ldots, u_{|U|}\}$7 is the number of blocks, $U = \{u_1, \ldots, u_{|U|}\}$8 the token count, and $U = \{u_1, \ldots, u_{|U|}\}$9 the hidden dimension. Full-catalog scoring adds $I = \{i_1, \ldots, i_{|I|}\}$0 work per user-context because the method computes scores against all items. The paper does not mention sampled softmax or candidate pruning. In practice, basket truncation at 100 items is used to keep sequence length manageable (Yang et al., 2019).

Several limitations are explicit. Cross-dataset pre-training is difficult because item IDs do not transfer across datasets in the way words do in natural language processing. Cold-start for new users and new items, as well as data sparsity, remain intrinsic issues, and the method does not incorporate attribute side-information. Real-time recommendation may be computationally heavy because fine-tuning and scoring require transformer inference per recommendation; caching historical encodings, batched scoring, or other engineering optimizations would be needed for production deployment.

Within the paper’s own framing, IERT is most appropriate in environments with rich sequential and co-purchase signals and in catalogs where item meaning is strongly context-dependent, such as grocery or electronics accessories. Its principal contribution is to shift next basket recommendation away from static item lookup toward transaction-conditioned item semantics. This, in turn, recasts the item embedding not as a fixed latent identifier but as a context-dependent representation shaped jointly by basket-level co-occurrence and cross-basket behavioral sequence (Yang et al., 2019).