STAP: A Shuffle-Tokenized App Predictor with Ultra Long Context for Vocabulary-Free Mobile App Prediction

Abstract: Predicting the next mobile application a user will launch is essential for intelligent device resource management and proactive assistance. Existing models rely on fixed app vocabularies, which prevents them from generalizing across different app ecosystems. Many also depend on user-specific knowledge, which complicates deployment in cold start scenarios. We propose STAP, a Transformer-based model that eliminates the need for a fixed vocabulary. STAP replaces true app identities with randomly reassigned virtual indices via a shuffle mechanism, and compensates for discarded semantic information by processing behavioral sequences with an ultra-long context design. A theoretical analysis shows that, given a sufficiently long context, the predicted distribution converges to the correct one despite the anonymity of the mapping. Experiments on two datasets from different continents demonstrate that STAP achieves strong cross-dataset zero-shot prediction accuracy -- a setting where all existing fixed-vocabulary methods are inherently inapplicable -- while its cold start performance within each dataset remains competitive with leading models. Furthermore, we introduce a deployment strategy that enables the model to retain a sufficiently long context during continuous inference while keeping latency within acceptable bounds.

• The paper demonstrates that a shuffle-tokenized Transformer framework enables vocabulary-free mobile app prediction, overcoming cold-start and cross-dataset issues.
• It leverages ultra-long context windows and the ISWI inference strategy to achieve competitive HR@1 metrics in both in-dataset and cross-domain evaluations.
• The study provides theoretical proof that with sufficient behavioral sequences, the Bayesian predictive distribution converges to the true next-app distribution despite token anonymization.

Vocabulary-Free Mobile App Prediction via Shuffle-Tokenized Ultra-Long Context Modeling

Introduction

The paper "STAP: A Shuffle-Tokenized App Predictor with Ultra Long Context for Vocabulary-Free Mobile App Prediction" (2605.29863) introduces a Transformer-based framework for next-app prediction that eschews reliance on a fixed app vocabulary. The approach is motivated by the limitations of traditional models in handling cold-start scenarios, cross-dataset generalization, and the inability to process unseen applications post-training. Leveraging a stochastic app-ID anonymization (the shuffle mechanism) alongside ultra-long context modeling, STAP enables practical zero-shot transfer across distinct app ecosystems while achieving competitive accuracy in standard in-dataset cold-start regimes. The architecture integrates enhancements for real-time deployment and demonstrates that fixed app vocabularies are not necessary to realize effective app usage prediction.

Methodology: Shuffle Mechanism and Ultra-Long Context

Shuffle Mechanism and Vocabulary Independence

The core of STAP is a stochastic, per-user and per-epoch injective mapping $\phi$ that assigns real app IDs to a pool of virtual indices, entirely decoupling the training and inference pipeline from any physical app vocabulary. By re-sampling $\phi$ each epoch, the network is effectively prevented from associating virtual indices with semantic or behavioral priors accrued from specific app distributions. This randomization ensures invariance to app-ecosystem shifts and directly supports zero-shot adaptation to previously unobserved application inventories.

Figure 1: The shuffle mechanism enables processing of anonymized app sequences, supporting both single-sequence and cross-dataset pipelines.

Ultra-Long Contextualization

The anonymization induced by the shuffle mechanism discards static semantic information; thus, STAP compensates using ultra-long temporal contexts. With context windows extending up to 4096 events, the model is able to leverage behavioral patterns at sufficiently long temporal horizons such that the probability distribution over next-app predictions converges to the correct one, even in the absence of persistent app identities. Theoretical justification is provided: with a long enough context, the posterior over possible shuffle mappings concentrates on the ground-truth, rendering longer behavioral sequences structurally unique and informative.

Model Architecture and Feature Fusion

STAP's architecture is a multi-block Transformer backbone comprising Pre-Norm residuals, RMSNorm, SwishGLU activations, and RoPE with continuous time-stamps. The input feature space fuses categorical (shuffled app index, action type), cyclic temporal (hour-of-day), and absolute timestamp features into a unified embedding pipeline.

Figure 2: Full STAP pipeline, with shuffle-based input anonymization, multimodal fusion, and advanced Transformer details tailored for behavioral temporal reasoning.

The RoPE module is adapted to encode high-variance, minute-level temporal features by scaling the base to $10^5$, ensuring meaningful temporal differentiation.

Continuous Ultra-Long Context Inference: ISWI

Ultra-long contexts require careful tradeoffs at inference: naïve window resets induce cold start deficits, while unbounded context growth is computationally prohibitive. The Interleaved Semi-Window Inference (ISWI) strategy maintains two staggered inference instances, each providing overlapping coverage such that at least $L/2$ historical events are always available, with total compute and memory cost bounded to twice the maximal windowed instance.

Figure 3: ISWI ensures seamless long-context inference via two overlapping session windows, maintaining both latency and context continuity.

Experimental Results and Analysis

Datasets and Scenario Definition

Evaluations are conducted on the Tsinghua App Usage and LSapp datasets, with pronounced app inventory and geographic divergence, providing a stringent testbed for cross-domain transfer. The per-user distributions of app counts and sequence lengths further motivate the hyperparameter choices (virtual vocabulary size, maximum context length).

Figure 4: Distribution analytics for app count and event sequence length in Tsinghua and LSapp datasets validate modeling choices for $|\mathcal{V}|$ and context window.

Performance in Zero-Shot and Cold-Start Settings

STAP achieves HR@1 up to 68.95% for Tsinghua $\to$ LSapp cross-dataset transfer and HR@1 71.27% for in-dataset cold-start on LSapp. Baseline fixed-vocabulary models exhibit zero applicability under cross-dataset transfer, whereas STAP robustly outperforms heuristic MRU/MFU. Across all key metrics (HR@k, MRR@k), STAP demonstrates <5% degradation relative to state-of-the-art fixed-vocabulary models in the cold-start regime, establishing its efficacy in vocabulary-free prediction.

Necessity of the Shuffle Mechanism

Ablation confirms that epoch-level resampling of the shuffle mapping is necessary to prevent overfitting and accelerate generalization. The “non-shuffle” variant stagnates early and experiences significant metric degradation (e.g., HR@1 drops by 6.5 percentage points).

Figure 5: Training and validation comparisons reveal premature convergence and degraded validation metrics in ablated (non-shuffle) models.

Impact of Ultra-Long Context

Metric performance improves monotonically with context length, saturating at 4096 events — a direct reflection of the sequence statistics in the considered datasets.

Figure 6: Accuracy metrics plateau for context lengths exceeding 4096 events; saturation arises due to dataset event-length distributions.

Computational Efficiency

The ISWI deployment ensures bounded inference latency (always $<$50ms/event on single-thread C++ without SIMD) with strictly linear scaling and periodic context handoff effects visible as sawtooth latency patterns.

Figure 7: ISWI deployment manifests as bounded latency per event in extended sequences, confirming practical feasibility for on-device use.

Training Dynamics

Distinctive training dynamics are observed: an extended near-random plateau phase, abruptly followed by loss and accuracy jumps, with the epoch of this “grokking” point scaling logarithmically with the context length.

Figure 8: Training loss and HR@1 exhibit a sudden inflection following a flat regime across runs.

Figure 9: The phenomenon/grokking epoch scales linearly with the log of the context length, indicating dependence of trainability on context horizon.

Theoretical Implications

The authors formally prove that with sufficiently long anonymized behavioral sequences, the Bayesian predictive distribution over virtual tokens converges to the true next-app distribution up to permutation indistinguishability, as the posterior over shuffles collapses on the ground-truth. This guarantees that vocabulary-free architectures with randomized token assignment retain full predictive power in the infinite context regime.

Practicalities, Limitations, and Prospects

STAP's vocabulary independence, privacy properties (no app names or user identifiers needed), and strict resource control under ISWI make it directly applicable to privacy-sensitive and resource-constrained mobile deployment. The approach, however, currently fixes virtual vocabulary size ($|\mathcal{V}|=200$), excluding the small set of users with extremely large app inventories. Its performance ceiling—and grokking phenomenon—invite theoretical investigation, especially regarding transformer optimization under extreme token anonymization.

It remains to be seen whether further gains can be realized by leveraging larger behavioral corpora with even longer contexts, adaptive vocabulary windows, or integrating more nuanced temporal/contextual features, as well as the possibility of generalization to other sequence domains where vocabulary identity is inherently unstable or unknown.

Conclusion

STAP provides robust evidence that fixed app vocabularies are not required for accurate next-app prediction in mobile behavior modeling. The synergistic use of shuffle-randomized tokenization and ultra-long context transformers, alongside theoretically justified convergence properties and efficient deployment strategies, results in a model that performs competitively with state-of-the-art methods while enabling zero-shot generalization across application inventories. This paradigm advances vocabulary-agnostic behavioral modeling and provides a template for future research in privacy-preserving, ecosystem-agnostic sequence modeling.