Learning Next Action Predictors from Human-Computer Interaction

Abstract: Truly proactive AI systems must anticipate what we will do next. This foresight demands far richer information than the sparse signals we type into our prompts -- it demands reasoning over the entire context of what we see and do. We formalize this as next action prediction (NAP): given a sequence of a user's multimodal interactions with a computer (screenshots, clicks, sensor data), predict that user's next action. Progress on this task requires both new data and modeling approaches. To scale data, we annotate longitudinal, naturalistic computer use with vision-LLMs. We release an open-source pipeline for performing this labeling on private infrastructure, and label over 360K actions across one month of continuous phone usage from 20 users, amounting to 1,800 hours of screen time. We then introduce LongNAP, a user model that combines parametric and in-context learning to reason over long interaction histories. LongNAP is trained via policy gradient methods to generate user-specific reasoning traces given some context; retrieve relevant traces from a library of past traces; and then apply retrieved traces in-context to predict future actions. Using an LLM-as-judge evaluation metric (0-1 similarity to ground truth), LongNAP significantly outperforms supervised finetuning and prompted baselines on held-out data (by 79% and 39% respectively). Additionally, LongNAP generalizes to held out users when trained across individuals. The space of next actions a user might take at any moment is unbounded, spanning thousands of possible outcomes. Despite this, 17.1% of LongNAP's predicted trajectories are well-aligned with what a user does next (LLM-judge score $\geq$ 0.5). This rises to 26% when we filter to highly confident predictions. In sum, we argue that learning from the full context of user behavior to anticipate user needs is now a viable task with substantial opportunity.

• The paper defines the next action prediction task and introduces NAPsack for scalable, rich data collection combined with the LongNAP architecture.
• It employs a two-phase reasoning process—retrieving historical context and predicting future actions using policy gradients—to achieve significant performance improvements.
• The study demonstrates practical implications for online adaptation and assistive automation, while addressing key privacy and alignment challenges in personalized AI.

Learning Next Action Predictors from Human-Computer Interaction: Technical Review

Motivation and Problem Formalization

The paper "Learning Next Action Predictors from Human-Computer Interaction" (2603.05923) addresses the challenge of proactive AI systems anticipating user behavior by introducing the next action prediction (NAP) task. Unlike current LLMs, which operate over sparse, task-specific prompts, NAP leverages the full multimodal interaction history—encompassing screenshots, clicks, and sensor data—to predict a user's next sequence of actions. The task is formally defined as modeling a temporal stream of events $\mathcal{E} = \{e_1, e_2, ... e_T\}$, with each event $e_t$ composed of action $a_t$ and optional visual context $I_t$. The goal is to predict future event trajectories $\hat{\mathcal{E}}_{t+1:t+h}$ given recent context $\mathcal{E}_{t-k:t}$.

To enable this, the authors focus on two core directions: (1) scalable collection and annotation of rich behavioral data; and (2) robust modeling approaches capable of reasoning over extended, unbounded histories.

Scalable Data Collection via NAPsack

Central to progress in NAP is an annotated longitudinal dataset reflecting authentic user interactions. The authors introduce NAPsack, an open-source, passive data collection pipeline operating by recording screenshots and I/O events, compressing via event-driven heuristics, and annotating via vision-LLMs (VLMs). This approach yields dense action labels without manual annotation effort and is validated against human-labeled ground truths using an LLM-as-a-judge similarity metric.

NAPsack applied to Screenomics data (20 users, one month) generates a dataset covering 1.9M screenshots and 360K action descriptions, spanning 1,800 hours. Compression reduces data storage by 75% while retaining caption quality, with human validation confirming alignment with LLM-judge scores.

LongNAP: Model Architecture and Training Paradigm

The LongNAP architecture exemplifies a hybrid paradigm, combining parametric and in-context learning for next action prediction. The core innovation involves a two-phase reasoning process: (1) Reasoning to Retrieve—generation of chain-of-thought traces to semantically query a memory bank of prior traces; (2) Reasoning to Predict—integrating retrieved relevant traces with current context to produce revised future trajectory predictions. Memory is continually updated with high-reward traces, facilitating adaptive modeling over an unbounded history.

Figure 1: LongNAP leverages the full multimodal context, retrieving over an unbounded history to predict user actions.

Figure 2: LongNAP’s two-phase generation: reasoning to retrieve relevant history, then reasoning to predict future actions, optimized with a temporal LLM-judge reward.

Policy gradient optimization (GRPO) is utilized to optimize discrete reasoning, retrieval, and prediction steps end-to-end, with temporal reward furnished by a validated LLM-judge metric comparing predicted and observed trajectories.

Experimental Results: Generalization and Predictability

The authors evaluate LongNAP in two setups: single-user temporal generalization and multi-user cross-generalization. In the single-user setup, LongNAP yields a 79% improvement over supervised finetuning and a 39% improvement over the strongest prompted baseline on LLM-judge similarity metrics. Cross-user generalization remains robust, with LongNAP achieving a 13% improvement over the best few-shot prompted baseline when evaluated on unseen users.

Figure 3: User-specific predictability variance; some users are markedly more predictable than others, as measured by LLM-judge scores across epochs.

Practical upper bounds are measured with pass@k accuracy: 17.1% of LongNAP’s predictions are well-aligned ($\geq 0.5$ LLM-judge) with ground truth, rising to 26% for highly-confident predictions—demonstrated via empirical calibration with intra-cluster variance.

Figure 4: Pass@k scores for LongNAP, reflecting the probability of producing well-aligned trajectories.

Figure 5: Calibration: high-confidence prompts correspond to higher accuracy (pass@1 improves from 10.3% to 25.9%).

Ablation and Mechanistic Analysis

Ablation studies underscore the criticality of reasoning and retrieval mechanisms: removal of either component degrades performance by ~15–19%. Preservation of chronological order (non-shuffled training) further enhances efficacy. Visualization reveals that LongNAP retrieves context from temporally diverse regions, drawing on user history spanning days.

Figure 6: Retrieval distribution illustrates LongNAP’s capacity to leverage temporally distant context for action prediction.

In-depth analysis of reasoning traces across users highlights substantial diversity in learned reasoning strategies, with some users exhibiting homogeneous trace patterns and others demonstrating richer behavioral variance.

Figure 7: Embedded reasoning traces show clustering by user, evidencing personalized modeling.

Reasoning traces gradually shorten during training, especially in the retrieval phase, with qualitative convergence towards retriever-query-like conciseness, while prediction phase traces maintain higher-order behavioral abstraction.

Figure 8: Training dynamics: retrieve-phase traces shrink, prediction-phase traces remain expressive.

Applications, Privacy, and Alignment Considerations

The practical implications of NAP and LongNAP are multifaceted:

• Online Learning: The powerNAP system demonstrates asynchronous online adaptation, where the model incrementally learns from ongoing interaction data without batch retraining.
• Assistive Automation: SleepWalk, an execution agent, leverages LongNAP outputs to partially automate routine user actions.
• User Privacy: By localizing retrieval and reasoning traces, privacy exposure is constrained; however, full device context remains sensitive, mandating deployment infrastructure supporting PHI and PII protection. On-device training and inference are posited as desirable future directions, backed by low-compute adaptation (e.g., LoRA, quantization).
• Alignment: Predictive modeling of user actions can potentially reinforce undesired behaviors (e.g., habitual procrastination). The authors advocate for value elicitation and steerable reward mechanisms to address the alignment problem.

Relation to Prior Work and Future Directions

LongNAP builds upon advances in world models, retrieval-augmented generation, and in-context learning, shifting focus from sim-to-real paradigms and web-scale proxies to individualized, longitudinal user modeling. The approach is conceptually related to metacognitive reasoning reuse and personalized assistant architectures. Future work is hinted at leveraging richer data types, more efficient per-user adaptation (multi-tenant LoRA serving), and improved online reward mechanisms for robust long-horizon modeling.

Conclusion

In summary, "Learning Next Action Predictors from Human-Computer Interaction" substantiates the tractability of next action prediction at the individual user level by formalizing the NAP task, releasing scalable annotation pipelines, and introducing the LongNAP architecture optimized via policy gradients and semantic similarity rewards. LongNAP achieves strong single-user and moderate cross-user generalization, with critical dependence on reasoning and retrieval mechanisms. The research implies practical advances in personalized AI systems, privacy-sensitive modeling, and generalizable user models, with prospective utility for proactive assistants and continual adaptation. Future work must address scalable online training, nuanced alignment, and robust privacy-preserving deployment.