One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation

Abstract: LLMs are increasingly used for recommendation reranking, but their listwise predictions can depend on the order in which candidates are presented. This creates a mismatch between the set-based nature of recommendation and the sequence-based computation of decoder-only LLMs, where permuting an otherwise identical candidate set can change item scores and final rankings. Such order sensitivity makes LLM-based rerankers difficult to rely on, since rankings may reflect prompt serialization rather than user preference. We propose InvariRank, a permutation-invariant listwise reranking framework that addresses this dependence at the architectural level. InvariRank blocks cross-candidate attention with a structured attention mask and negates position-induced scoring changes through shared positional framing under Rotary Positional Embeddings (RoPE). Combined with a listwise learning-to-rank objective, the model scores all candidates in a single forward pass, avoiding permutation-based invariance training objectives that require multiple permutations of a candidate set. Experiments on recommendation benchmarks show that InvariRank maintains competitive ranking effectiveness while producing stable rankings across candidate permutations. The results suggest that architectural invariance is a practical route to reliable and efficient LLM-based recommendation reranking. The source code is at https://github.com/ejbito/InvariRank.

• The paper introduces InvariRank, which eliminates candidate order bias in LLM reranking using structured attention masking and shared positional framing.
• It employs a LambdaRank-style loss on candidate log-probabilities to optimize global ranking quality while maintaining single-pass efficiency.
• Empirical results on MovieLens-32M and Amazon Books demonstrate near-perfect permutation invariance with competitive recommendation performance.

One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation

Problem Statement and Motivation

Recent advances employ LLMs for listwise reranking in recommender systems, fusing semantic-rich item descriptions, user histories, and task instructions within a unified conditional scoring architecture. However, this approach inherently misaligns the set-based semantics of the recommendation candidate pool with the sequence-based computation of decoder-only transformers. This induces position sensitivity: a model's output rankings depend not solely on user or item semantics but also on the permutation in which candidates are serialized as input. This undermines reproducibility, evaluation, and deployment reliability, as rankings may reflect prompt structure rather than true relevance signals.

Empirical studies have shown that even fine-tuned LLM-based rerankers (under causal attention and standard positional encodings) remain variational with respect to candidate order, violating the set-based invariance required for recommendation stability. Existing mitigation strategies—permutation ensembling, pairwise prompting, or permutation-invariance regularization—require multiple forward passes or do not eliminate underlying leakage at the architectural level.

InvariRank: Architectural Scheme for Permutation Invariance

The InvariRank framework addresses position sensitivity via two architectural interventions, ensuring true permutation-invariant listwise reranking in a single forward pass. This is achieved by:

1. Structured Attention Masking: Cross-candidate attention is explicitly blocked via a segment mask. Each candidate can only attend to itself and the shared context (user history and instructions), never to other candidates. This eliminates interference or content leakage from other candidates, removing order-dependent information flow.
2. Shared Positional Framing via RoPE: Candidate tokens are assigned input positions within a fixed frame, such that the positional offset between context and candidate is consistent and independent of the candidate's serialization order. Under rotary positional embeddings (RoPE), this guarantees that the attention patterns and relevance computation for each candidate are invariant under input permutation.

   Figure 1: Schematic of InvariRank—cross-candidate interactions are blocked, and all candidate segments share the same positional anchor relative to the context.

Prominent prior approaches, such as permutation-aggregation at inference, regularization during training, or post-hoc calibration, do not eliminate these position-dependent computation paths and generally trade off efficiency or effectiveness. InvariRank, in contrast, enforces invariance as a property of the architectural computation graph.

Training Objective and Scoring Mechanism

While architectural invariance enforces conditional independence among candidates during the forward pass, InvariRank retains listwise supervision via a LambdaRank-style objective. Specifically, the scalar score for each candidate is a function of the context and the candidate's isolated segment, computed as the mean token-level log-probability. The LambdaRank loss is applied pairwise between candidates, weighted by the nDCG gain from swapping positions, allowing the model to optimize global ranking quality within the constraint of local candidate independence.

Experimental Evaluation

The evaluation is conducted on timestamped explicit-feedback datasets: MovieLens-32M and Amazon Books. Candidate pools (size $K=25$) are constructed from a first-stage retriever (LightGCN), ensuring the inclusion of ground-truth positives and challenging negatives.

Models compared include zero-shot LLMs, bootstrapping (permutation aggregation), STELLA (post-hoc calibration), sequential greedy selection (SGS), listwise fine-tuning (LFT, without invariance constraints), and the proposed InvariRank. Implementations use LLaMA 3.2 3B-Instruct and Mistral 7B-Instruct, fine-tuned with LoRA.

Main Numerical Results and Analysis

InvariRank achieves near-optimal robustness to candidate permutation while maintaining competitive ranking effectiveness—closing the gap to non-invariant fully fine-tuned models (LFT) with minimal degradation.

Key highlights:

• On MovieLens-32M with LLaMA-3B-Instruct: InvariRank achieves $\tau=0.9883$, $\rho=0.9984$, and top-5 agreement $=0.9906$, with nDCG@10 of $0.8166$ (compared to LFT's nDCG@10 of $0.8486$ and $\tau=0.8861$).
• On Amazon Books: Consistent robustness gains are observed, with only marginal reductions in effectiveness compared to LFT.

SGS and bootstrapping (which require multiple forward passes or permutations) improve robustness but remain inferior to InvariRank both in efficiency and residual position bias.

Position Bias and Exposure

Analysis of top-$k$ exposure by input position reveals that zero-shot and bootstrapping methods exhibit marked position bias: items earlier in the input sequence dominate the top recommendations due to inherent LLM positional priors. LFT reduces bias but fails to eliminate it. InvariRank produces a nearly flat exposure curve, confirming the elimination of serialization-induced exposure.

Figure 2: Top-5 exposure across input positions on ML-32M demonstrates uniform exposure under InvariRank, with strong left-edge bias in baseline strategies.

Ablation Study

Ablations disentangle the contributions of structured attention masking and shared positional framing. Masking alone recovers much of the permutation robustness ($\tau=0.9193$), indicating that cross-candidate isolation is critical. However, only the combination with positional framing achieves near-perfect invariance, indicating residual vulnerabilities to RoPE-induced offset drift when only masking is applied.

Practical and Theoretical Implications

Architectural permutation invariance is both necessary and sufficient for robust listwise LLM reranking in settings where candidate sets are semantically unordered. This has several direct consequences:

• Reliable Evaluation and Deployment: By removing order sensitivity, model performance can be assessed unambiguously, and online systems avoid brittle behavior arising from arbitrary candidate serialization.
• Efficiency: InvariRank achieves invariance in a single forward pass, far exceeding the inference tractability of ensembling-based or comparison-prompting methods.
• Generalization: While the approach is applied to recommender reranking, the principle generalizes to other ranking and set-based LLM tasks, including information retrieval and RAG.
• Trade-Offs: Full candidate isolation may limit the model's ability to exploit cross-candidate comparative signals—suggesting future work on hybrid architectures that enable controlled or differentiable cross-candidate interaction under a permissive invariance constraint.

Outlook and Future Work

The current study restricts evaluation to two standard datasets and moderate candidate set sizes. Directions for extension include:

• Scalability to larger candidate pools under realistic deployment constraints.
• Applicability to retrieval, RAG, and mixed-modality ranking tasks where item inter-dependencies may encode further insights.
• Hybrid architectures respecting permutation invariance but allowing selective, learnable interaction among candidates—a direction reminiscent of deep set architectures [zaheer2017deep] but within large-scale transformer designs.

Conclusion

InvariRank demonstrates that LLM-based recommender reranking can achieve strict permutation invariance through principled architectural interventions—structured attention masking and shared positional framing—while retaining high listwise effectiveness. These insights reframe best practice for LLM rankers in recommendation and related fields, suggesting a paradigm in which order invariance is an explicit architectural invariant rather than an empirical desideratum.