LongNAP: Next Action Prediction Framework

• LongNAP is a user modeling framework that predicts a user's next multimodal actions by analyzing natural-language captions and screenshots from extended interaction histories.
• It employs a hybrid memory architecture that integrates vision–language modeling with in-context chain-of-thought reasoning and episodic memory retrieval.
• The framework achieves significant performance improvements over baselines through policy gradient optimization and detailed evaluation metrics including LLM and human judgments.

LongNAP (Long-horizon Next Action Prediction) is a user modeling framework designed to anticipate forthcoming user actions based on prolonged and multimodal human-computer interaction histories. It operationalizes next action prediction (NAP) as the task of predicting a user's next interactions—capturing both natural-language action captions and visual context (screenshots)—from a chronologically ordered multimodal history. LongNAP leverages a hybrid memory architecture that unifies vision–language modeling, in-context chain-of-thought reasoning, and episodic memory retrieval. The motivation arises from the need for proactive AI agents to interpret preverbal and contextually rich cues embedded in continuous user interactions, rather than sparse prompt-based input (Shaikh et al., 6 Mar 2026).

1. Model Architecture

LongNAP employs a two-phase architecture based on a vision–LLM (VLM) policy $\pi_\theta$ (Qwen-2.5-VL-7B). The input is a sequence of $k$ most recent events $E_{t–k+1:t} = \{ e_{t–k+1}, ..., e_t \}$, where each event $e_i = (a_i, I_i)$ pairs a natural-language action caption $a_i$ with an optional screenshot $I_i$.

The architecture consists of two principal components:

• Parametric component: The VLM backbone $\pi_\theta$ processes tokenized action captions and visual tokens (image patches from up to the last two screenshots). LoRA adapters (rank 8, $\alpha=32$, dropout=0.05) are employed for cost-effective fine-tuning of the model's MLP modules.
• In-context learning/memory component: A memory $\mathcal{M}_t$ maintains a set of context–reasoning-trace pairs $(C_j, z_j)$, where $k$0 is a past interaction context and $k$1 is its generated chain-of-thought. Lexical retrieval uses BM25 (parameters: $k$2, $k$3, time decay $k$4/day), with Maximal Marginal Relevance (MMR; $k$5) for diversity, reducing the top-10 to a top-5 candidate set. Retrieval dropout (10% drop, 10% reorder, 10% no retrieval) is applied for training stability.

The two-phase generation occurs at each decision point:

Phase	Input	Output
Reason-to-Retrieve	Recent context $k$6	Reasoning trace $k$7, relevant retrieved memory $k$8
Reason-to-Predict	$k$9, $E_{t–k+1:t} = \{ e_{t–k+1}, ..., e_t \}$0, $E_{t–k+1:t} = \{ e_{t–k+1}, ..., e_t \}$1	Refined trace $E_{t–k+1:t} = \{ e_{t–k+1}, ..., e_t \}$2, future action candidate $E_{t–k+1:t} = \{ e_{t–k+1}, ..., e_t \}$3

First, Reason-to-Retrieve samples an initial reasoning trace $E_{t–k+1:t} = \{ e_{t–k+1}, ..., e_t \}$4, which is used as a BM25 query to retrieve relevant past traces $E_{t–k+1:t} = \{ e_{t–k+1}, ..., e_t \}$5. Then, Reason-to-Predict samples a refined reasoning trace $E_{t–k+1:t} = \{ e_{t–k+1}, ..., e_t \}$6 conditioned on $E_{t–k+1:t} = \{ e_{t–k+1}, ..., e_t \}$7 and predicts a future action trajectory $E_{t–k+1:t} = \{ e_{t–k+1}, ..., e_t \}$8. Four candidates are generated in parallel; the one with the highest reward is selected, and its trace is appended to memory: $E_{t–k+1:t} = \{ e_{t–k+1}, ..., e_t \}$9 (Shaikh et al., 6 Mar 2026).

2. Training Objective and Optimization

Training is conducted end-to-end via policy gradient methods (REINFORCE-style) with the Generalized REINFORCE Policy Optimization (GRPO) algorithm for variance reduction. For each rollout $e_i = (a_i, I_i)$0, the log-probability is

$e_i = (a_i, I_i)$1

The reward $e_i = (a_i, I_i)$2 is computed by an LLM-judge (Gemini 3.0 Flash), scoring the semantic similarity between the predicted and ground-truth future trajectories:

$e_i = (a_i, I_i)$3

The goal is to maximize expected reward $e_i = (a_i, I_i)$4, with estimated gradient

$e_i = (a_i, I_i)$5

where $e_i = (a_i, I_i)$6 is the mean reward over $e_i = (a_i, I_i)$7 parallel rollouts. No supervised or Kullback–Leibler regularizers are applied, and temporal consistency is enforced by resetting memory at epoch boundaries (Shaikh et al., 6 Mar 2026).

3. Data Acquisition and Annotation Pipeline

The dataset comprises longitudinal, naturalistic smartphone usage from 20 users (ages 22–70, 14 F/5 M), each using their device for at least one hour daily over 28 days. Key dataset characteristics:

• Acquisition: Time-lapse smartphone screenshots sampled at one-minute intervals (1.9 million total, 1,837 hours total screen-on time), deduplicated via dHash perceptual hashing.
• Event logging: Input events—including clicks, scrolls, and keystrokes—are burst-grouped when available.
• Automatic annotation: The NAPsack pipeline uses a VLM with few-shot prompting to convert each screenshot plus I/O burst into a natural-language caption encapsulating the observed user action. The captioning context window spans 60 frames, appended with events.
• Corpus statistics: Post-processing yields 360,000 generated action captions with an average temporal resolution of 15 seconds per action.

A plausible implication is that this workflow enables scaling to massive, private, and richly multimodal user datasets using automated labeling (Shaikh et al., 6 Mar 2026).

4. Algorithmic Workflow

The procedural logic can be sketched as follows:

1. Initialize empty memory $e_i = (a_i, I_i)$8.
2. For each user $e_i = (a_i, I_i)$9 and each time $a_i$0 (in chronological order):
   • Form the recent context $a_i$1 from the last $a_i$2 events.
   • Reason-to-Retrieve: Sample $a_i$3; retrieve top-5 past traces $a_i$4 using $a_i$5.
   • Reason-to-Predict: For $a_i$6 rollouts, sample $a_i$7 and corresponding candidate $a_i$8.
   • Compute rewards $a_i$9 via the LLM-judge, identify best trace $I_i$0, and append $I_i$1 to memory.
   • Perform policy-gradient update with baseline $I_i$2.

Temporal ordering in training is enforced by resetting memory between epochs, which preserves causal structure and improves model fidelity (Shaikh et al., 6 Mar 2026).

5. Evaluation Methodology and Quantitative Results

Evaluation employs a LLM as judge (Gemini 3.0 Flash), assigning a similarity score $I_i$3 denoting semantic overlap between the predicted and ground-truth action trajectories. Human comparison of >300 sample pairs exhibits $I_i$485% agreement with this automatic judge at 95% confidence.

Core results include:

• Per-user training: LongNAP achieves an average LLM-judge similarity of 0.38, compared to 0.21 (supervised finetuning, SFT) and 0.27 (few-shot Gemini RAG), corresponding to an improvement of +79% versus SFT and +39% versus the most performant baseline.
• Cross-user generalization: LongNAP (0.26) surpasses the best baseline (0.23, Gemini RAG) by +13% relative.
• Pass@k metrics: For a similarity threshold $I_i$5 0.5, pass@1 is 17.1% and pass@20 is 36.3%. Prediction confidence correlates with accuracy: in the top 10% of high-confidence samples, pass@1 rises to 25.9% (vs. 10.3% in low-confidence).
• Human evaluation: LongNAP attains a 79% pairwise win rate over SFT, 58% over Gemini RAG, and 55% over Gemini zero-shot.
• Ablation studies: Removing reasoning traces or the retriever degrades performance (–19% and –16%, respectively). Training without chronological order reduces average score by –10%.

These findings demonstrate the utility of reasoning-trace memory and retrieval, as well as the importance of chronological consistency during training (Shaikh et al., 6 Mar 2026).

6. Limitations and Open Challenges

Limitations acknowledged in the framework include:

• Annotation noise: Action labels are generated by VLMs and may propagate errors downstream.
• User privacy: Rich contextual data is observed by the model; on-device inference or redaction is necessary for real-world deployment.
• Alignment risks: Without safeguards, LongNAP may inadvertently reinforce or facilitate undesired user behaviors.
• Computational burden: The approach assumes access to large VLMs and policy-gradient optimization; scaling to large populations or real-time adaptation may require parameter-efficient LoRA solutions.
• Context window restrictions: Only the last two screenshots are used in memory due to GPU constraints; this may limit the model's ability to fully capture temporal dependencies. A plausible implication is that longer context windows could further enhance prediction quality.

Open research challenges include robustifying labeling pipelines, controlling for privacy, improving generalization across users, and enhancing efficiency for practical, scalable deployment (Shaikh et al., 6 Mar 2026).