ProMemAssist: Proactive Assistance via WM

• The paper introduces ProMemAssist, a wearable system that models working memory in real time to determine optimal moments for delivering assistance.
• It employs multimodal sensors and a dual-memory architecture (perception memory and episodic buffer) to manage information and avoid interruptive cues.
• Experimental results indicate higher user engagement and reduced frustration, demonstrating that timing based on cognitive state enhances assistance quality.

Searching arXiv for the target paper and closely related work on proactive memory assistance, wearable memory augmentation, and proactive agent memory systems. ProMemAssist is a wearable proactive-assistance system for smart glasses that models a user’s working memory in real time from multimodal sensor signals in order to decide not only what help to provide, but when help should be delivered (Pu et al., 28 Jul 2025). It is grounded in cognitive theories of working memory and represents perceived information as memory items and episodes with encoding mechanisms such as displacement and interference. The system’s central claim is that interruption timing in wearable AI cannot be reduced to task-step detection or prompt-based relevance alone; instead, assistance should be conditioned on the user’s current cognitive availability, approximated through a computational working-memory model (Pu et al., 28 Jul 2025).

1. Problem formulation and conceptual scope

ProMemAssist addresses interruption timing in proactive wearable AI. The motivating setting is everyday physical activity, where users may be busy with their hands, attention, or social situation, and where even useful assistance can arrive at the wrong moment and become disruptive (Pu et al., 28 Jul 2025). The paper characterizes this as a mixed-initiative tension between the value of helping and the cost of interrupting. Existing proactive assistants are described as relying on user initiation, task-step heuristics, or prompt-based LLM reasoning, thereby neglecting the user’s current mental state (Pu et al., 28 Jul 2025).

The system is therefore organized around a different question: what information is currently active in working memory, how salient is it, and would a new message integrate smoothly or interfere? In this framing, working memory is used as a proxy for what the user is currently attending to and how cognitively available they are for interruption. This is a narrower but more state-sensitive formulation than systems centered on explicit recall queries or archival retrieval.

That emphasis differentiates ProMemAssist from earlier wearable memory assistants such as Memoro, which was designed for “minimal disruption” during live conversation through concise retrieval, explicit Query Mode, and predictive Queryless Mode rather than through explicit working-memory modeling (Zulfikar et al., 2024). It also differs from object-retrieval assistants such as MemPal, whose evaluated use case was voice-activated object finding in older adults’ homes, even though MemPal was also positioned as a platform for proactive reminders and broader retrospective memory support (Maniar et al., 3 Feb 2025). A plausible implication is that ProMemAssist occupies a distinct point in the design space: it is primarily a timing architecture for proactive intervention, not primarily a retrieval interface.

2. Cognitive basis and working-memory formalization

ProMemAssist is explicitly inspired by the multicomponent working-memory model, especially Baddeley’s theory, including visuospatial, phonological, and episodic components (Pu et al., 28 Jul 2025). The paper does not claim to implement a full cognitive simulation; instead, it operationalizes working-memory theory as a practical wearable pipeline. The mapping from theory to mechanism is direct:

• Capacity limits map to fixed-size memory stores.
• Recency maps to decay over time unless reactivated.
• Binding / chunking maps to grouping related items into episodes.
• Displacement maps to overwriting older or lower-value items when capacity is full.
• Interference maps to competition among same-modality contents (Pu et al., 28 Jul 2025).

The architecture contains two principal memory structures. The first is perception memory, a low-level store for directly perceived information with a fixed capacity of 7 memory items, reflecting Miller’s classic capacity estimate. The second is an episodic buffer, a higher-level store with a capacity of 4 memory chunks, consistent with Cowan’s “magical mystery four” (Pu et al., 28 Jul 2025).

Each MemoryItem contains a timestamp, a type—visuospatial or phonological—content such as object label plus serialized image or transcribed speech, and a feature vector from CLIP. Each MemoryChunk contains a timestamp, a list of MemoryItem objects bound into that episode, and a short LLM-generated textual summary describing the task context (Pu et al., 28 Jul 2025). Because internal episodes are not directly observable, episodes are derived from perception memory rather than directly sensed.

The formal properties attached to each MemoryItem are recency, relevance, and importance. Recency is modeled as a linear decay function:

$\text{Recency} = 1 - \frac{t}{T}$

where $t$ is elapsed time since encoding or last rehearsal and $T = 30$ seconds (Pu et al., 28 Jul 2025). Relevance is defined as the average cosine similarity between the item embedding and the embeddings of the episodic-buffer summaries. Importance is assigned by an LLM prompt and normalized to $[0,1]$, with examples such as “a fire alarm is going off” being high importance and a background advertisement being low importance (Pu et al., 28 Jul 2025).

This formulation makes working memory an explicitly dynamic state estimator rather than a static store. In adjacent proactive-assistance theory, this aligns with the view that assistance must be beneficial, recognizable, and low-overhead under partial observability (Kulkarni et al., 2021). This suggests that ProMemAssist’s WM model functions as a practical approximation of the human-side belief and load state that earlier planning work represented more abstractly.

3. Multimodal sensing and internal representation

The system runs on prototype smart glasses equipped with an RGB camera and a 7-channel microphone array. Raw video and audio are streamed to a companion laptop at roughly 1-second intervals (Pu et al., 28 Jul 2025). The sensing pipeline then performs three primary operations: object detection on frames using YOLOv11, speech-to-text transcription using Whisper, and CLIP embeddings for both visual and textual inputs (Pu et al., 28 Jul 2025).

These outputs are converted into the internal units used by the working-memory model. Visual information is represented using the detected object image and label. Auditory information is represented using transcribed speech. Both are embedded into a shared semantic space via CLIP, permitting similarity comparison across items and with episode summaries (Pu et al., 28 Jul 2025). This shared-embedding design is central because the later binding, relevance, and interference computations all depend on cosine similarity over these representations.

The architecture is intentionally layered. Perception memory stores directly perceived items, while the episodic buffer stores integrated contextual chunks. The latter includes short LLM-generated summaries that describe task context, thereby providing the semantic anchor for relevance estimation and episode-level binding (Pu et al., 28 Jul 2025). The model thereby combines bottom-up perception with top-down contextual compression.

This multimodal pipeline should not be confused with long-term memory extraction systems such as ProMem, which emphasize iterative extraction, semantic matching, and self-questioning over dialogue history to improve memory completeness and QA accuracy (Yang et al., 8 Jan 2026). ProMemAssist instead focuses on the short-horizon dynamics of cognitive availability. A plausible implication is that its memory horizon is deliberately local, whereas long-term systems such as ProMem or T-Mem are designed for persistence, reachability, and future retrieval (Yang et al., 8 Jan 2026, Guo et al., 13 Jun 2026).

4. Memory operations: encoding, displacement, and episodic binding

When new sensor information arrives, ProMemAssist encodes it as a MemoryItem. If perception memory has free capacity, the item is inserted directly. If the store is full, the system performs displacement by replacing the least valuable item (Pu et al., 28 Jul 2025). To avoid redundant re-encoding, the system compares a detected object with existing working-memory items using CLIP; if similarity is above 0.95, the object is treated as already present and its timestamp is updated instead of creating a duplicate (Pu et al., 28 Jul 2025).

The displacement mechanism is governed by a weighted composite score:

$$\text{Score} = \alpha \cdot \text{Recency} + \beta \cdot \text{Relevance} + \gamma \cdot \text{Importance}$$

with default weights $\alpha = 0.3$, $\beta = 0.4$, and $\gamma = 0.3$. The item with the lowest score is displaced (Pu et al., 28 Jul 2025). This operationalizes both decay and overwriting under capacity constraints.

After encoding, the system decides whether the new item should join an existing episodic chunk or begin a new one. The decision uses a Binding Score:

$$\text{Binding Score} = \lambda \cdot \text{Episode Similarity} + (1 - \lambda) \cdot \text{Item Similarity}$$

where Episode Similarity is the cosine similarity between the new item and the episode summary, Item Similarity is the average similarity between the new item and the items already in the chunk, and $\lambda = 0.6$ (Pu et al., 28 Jul 2025). If the highest score exceeds the threshold $t$0, the item is bound to that chunk; otherwise, a new chunk is created and summarized by an LLM. If the episodic buffer is full, the least relevant chunk is displaced using the average composite value of its items (Pu et al., 28 Jul 2025).

These mechanisms are notable because they make the system selective before any assistance decision is made. The WM state is not a raw event log. It is a capacity-limited, dynamically pruned structure whose contents already reflect recency, semantic fit, salience, repetition suppression, and chunk formation. This differs substantially from external-memory recommenders such as MAP, where a separate user profile is stored and relevant entries are retrieved at inference time from a growing history (Chen, 3 May 2025). In ProMemAssist, memory management is not only about access efficiency; it is the basis of a timing model.

5. Timing prediction and the utility of interruption

The timing predictor is the system’s central decision module. It evaluates whether a candidate assistance message should be delivered immediately, deferred for later, or discarded (Pu et al., 28 Jul 2025). The paper formulates this as a multi-objective optimization balancing assistance value against interruption cost.

The utility function is:

$t$1

where $t$2 is the importance of the candidate assistance, $t$3 is its relevance, $t$4 is predicted displacement cost, $t$5 is predicted interference cost, and $t$6, $t$7 (Pu et al., 28 Jul 2025). The candidate assistance is treated as if it were a new MemoryItem. Recency is excluded from the value term because a delivered message would become maximally recent (Pu et al., 28 Jul 2025).

The displacement cost is estimated by finding the minimum composite score among existing working-memory items, corresponding to the item most likely to be overwritten if the assistance is introduced (Pu et al., 28 Jul 2025). The interference cost is modality-sensitive. Only items in the same modality as the candidate message are considered, consistent with the Baddeley-style assumption that phonological content competes with auditory-verbal content and visuospatial content competes with visual input (Pu et al., 28 Jul 2025). The unnormalized interference term is:

$t$8

and the normalized term is:

$t$9

(Pu et al., 28 Jul 2025). Highly similar same-modality content yields lower interference because it may be integrated or chunked; dissimilar same-modality content yields higher interference.

The decision rule is thresholded:

• if $T = 30$0, assistance is delivered immediately;
• if $T = 30$1, assistance is deferred and re-evaluated later;
• if $T = 30$2, assistance is discarded (Pu et al., 28 Jul 2025).

This policy is consequential because it makes ProMemAssist neither a pure filter nor a pure generator. Candidate voice messages are produced by an LLM-based assistance generator that runs at every WM update and receives the newest MemoryItem, a summary of episodic chunks, and a short history of previous assistance, but the paper states that the main contribution is timing rather than content generation (Pu et al., 28 Jul 2025). The content generator and the baseline share the same multimodal observations and task context; the differentiator is the explicit WM-based timing layer.

This architecture is conceptually consistent with broader proactive-agent research that separates “When to Assist” from “How to Assist” and reports that long-term memory and historical context enhance proactive prediction accuracy (Tang et al., 4 Feb 2026). ProMemAssist instantiates the “when” problem through working-memory dynamics rather than through long-range behavioral history.

6. Experimental evaluation and empirical findings

The user study used a within-subject design with 12 participants: 8 male, 3 female, 1 non-binary, with mean age 36 (Pu et al., 28 Jul 2025). Participants were research engineers or scientists familiar with wearable and mixed-reality devices, and each session lasted about 60 minutes (Pu et al., 28 Jul 2025). Two conditions were compared:

1. ProMemAssist, using WM-based timing and assistance filtering.
2. Baseline, a prompt-engineered LLM that had the same multimodal observations and task context but no explicit WM model, and that both generated assistance and decided when to deliver it (Pu et al., 28 Jul 2025).

Participants completed four cognitively demanding tabletop tasks: setting up a dining table, organizing an office desk, packing for a trip, and arranging a living room table (Pu et al., 28 Jul 2025). The experimenter inserted both relevant and irrelevant spoken prompts to simulate natural interruption pressure and changing context. After each task, participants rated timeliness, interruptiveness, relevance, helpfulness, and workload using NASA-TLX items adapted to a 1–7 Likert scale (Pu et al., 28 Jul 2025). Positive engagement was manually logged when participants visibly or verbally responded to a message or acted on it.

The system-level results indicate markedly greater selectivity in the WM-based condition. ProMemAssist processed WM 395 times, generating 218 candidate assistance messages, of which 130 were delivered immediately, 31 were deferred, and 57 were discarded (Pu et al., 28 Jul 2025). The baseline encoded new information 534 times and generated and delivered 332 messages (Pu et al., 28 Jul 2025). Per task, the ProMemAssist WM model processed on average 16.5 encoding events, added 6.92 new MemoryItems, detected 4.67 repetitions that were not re-added, and made 4.88 replacements due to capacity constraints (Pu et al., 28 Jul 2025).

The principal behavioral finding is engagement. ProMemAssist achieved 24.6% positive responses, compared with 9.34% for the baseline (Pu et al., 28 Jul 2025). This indicates fewer messages but a substantially higher proportion of messages arriving at moments when users were receptive. Task completion time showed no significant difference, with means of 175 seconds for ProMemAssist and 182 seconds for the baseline (Pu et al., 28 Jul 2025).

Among survey measures, the only statistically significant difference was frustration: ProMemAssist mean 2.32, baseline mean 3.14, with $T = 30$3 (Pu et al., 28 Jul 2025). Mental demand, physical demand, time pressure, and task difficulty did not differ significantly. Assistance was rated as less interruptive in ProMemAssist, though not significantly so, while relevance and helpfulness were also not significantly different (Pu et al., 28 Jul 2025). The paper interprets this pattern as evidence that the critical variable was timing rather than message quality.

Qualitative findings reinforce that interpretation. Participants described ProMemAssist as less disruptive, better aligned with moments when they were less busy, and more respectful because it was selective rather than “pushy” (Pu et al., 28 Jul 2025). They also explicitly connected receptivity to mental availability, task stage, and mood. These reports are consistent with adjacent HCI work arguing that proactive assistance must trade off interruption cost against need coverage and that low precision yields alert fatigue while low recall fragments workflows (Tang et al., 4 Feb 2026).

7. Position in the literature, misconceptions, and open problems

ProMemAssist belongs to a broader family of memory-augmented assistance systems, but its contribution is specific. It is not primarily a long-term conversational memory architecture, a user-profile recommender, or a memory-extraction framework. Its novelty lies in using a real-time WM model to regulate proactive interruption timing on a wearable device (Pu et al., 28 Jul 2025).

Several nearby lines of work illuminate that specificity. Memoro showed that wearable memory augmentation in social interaction benefits from minimal input and minimal output, including concise answers and a queryless predictive mode (Zulfikar et al., 2024). ProAgentBench framed proactive assistance as a hierarchical “When + How” problem and found that long-term memory and real-world behavioral traces improve proactive prediction (Tang et al., 4 Feb 2026). Proactive-assistance planning under partial observability formalized three principles: the assistant’s activity should reduce the human’s cost, the human should be able to recognize the reduction, and the assistant should optimize the human’s overall cost of achieving the goal (Kulkarni et al., 2021). Long-term memory work such as ProMem, T-Mem, and MemSifter concentrates on extraction completeness, future retrieval reachability, and proxy-mediated memory ranking rather than on moment-to-moment interruption cost (Yang et al., 8 Jan 2026, Guo et al., 13 Jun 2026, Tan et al., 3 Mar 2026).

A common misconception would be to treat ProMemAssist as a claim of cognitive realism in the strong sense. The paper explicitly rejects that reading: the WM model is described as a practical computational interpretation of working memory, not a validated cognitive simulator (Pu et al., 28 Jul 2025). Another misconception would be to view its empirical result as evidence that it generates better assistance content. The reported pattern is narrower: relevance and helpfulness were not significantly different, because both conditions used the same underlying LLM for message content; the observed benefit was higher engagement and lower frustration through better timing (Pu et al., 28 Jul 2025).

The paper is also explicit about limitations. The sensing stack includes only vision and audio; there is no gaze, gesture, head pose, or tactile input. Perspective ambiguity means the system may sometimes encode the experimenter’s actions rather than the user’s. The WM model is simplified, timing and content quality remain coupled in user experience, the tasks were limited to tabletop multistep activities, and there was no gold-standard false-negative evaluation because tasks admitted multiple valid solutions (Pu et al., 28 Jul 2025). The stated future directions include integrating long-term memory and richer user modeling, adding user feedback channels, learning personalized thresholds and preferences, exposing WM state to users, testing ablations for recency, interference, and relevance, extending to devices such as earbuds, smartwatches, or desktop companions, and adding sensors in more dynamic environments (Pu et al., 28 Jul 2025).

Taken together, these details position ProMemAssist as a proof-of-concept for cognitively informed proactive wearables. Its core proposition is not that wearables should remember more, but that they should estimate when the user can absorb help. In that sense, ProMemAssist links cognitive theory, multimodal sensing, and mixed-initiative HCI around a single operational claim: assistance quality depends materially on timing, and timing can be improved by modeling the transient contents and constraints of working memory (Pu et al., 28 Jul 2025).