Motion-Aware Memory Selection

Updated 7 July 2026

Motion-aware memory selection is a mechanism that prioritizes dynamic cues like trajectory consistency and motion change to identify and store temporally informative data.
It is applied across domains such as video object segmentation, SLAM, and audio-visual streaming to filter redundant or corrupted memory entries and enhance inference accuracy.
Empirical results demonstrate that incorporating motion-aware selection improves key performance metrics, providing a scalable solution under bounded memory and compute constraints.

Motion-aware memory selection denotes a class of mechanisms that decide which historical states should be written to memory, retained under a budget, or retrieved at inference time by using motion, temporal change, trajectory consistency, or related dynamic cues rather than recency or appearance alone. In contemporary literature, the term spans streaming audio-visual LLMs, SAM2-based visual tracking, video object segmentation, video editing, video prediction, SLAM, and even theoretical models of collective cell migration. Across these settings, the shared objective is to preserve temporally informative evidence while suppressing redundant, corrupted, or dynamically irrelevant memory entries under finite compute or memory constraints (Sun et al., 26 May 2026, Yang et al., 2024, Alansari et al., 23 Jun 2026).

1. Conceptual scope and recurrent failure modes

Motion-aware memory selection arises wherever temporal processing relies on a bounded memory bank but naive retention rules degrade long-horizon performance. The failure pattern is consistent across otherwise dissimilar systems. In streaming audio-visual LLMs, uniform KV compression ignores the fact that many video tokens are redundant across adjacent frames, while some encode important changes, events, or speech; the same setting also exhibits a severe audio-visual token imbalance, with visual tokens often 30–40× more numerous than audio tokens (Sun et al., 26 May 2026). In SAM2-based trackers, a fixed FIFO window or confidence-only memory write can store masks corrupted by occlusion, distractors, or identity switches, which then propagate error autoregressively (Yang et al., 2024, Alansari et al., 23 Jun 2026). In STM-style video segmentation, appearance-only retrieval can be noisy under fast motion, occlusion, blur, or multiple similar instances, while full-frame dense memory often stores large amounts of irrelevant background (Miao et al., 2022, Sun et al., 2021).

These systems differ in what they call “memory.” The retained units may be KV pairs, past masks, object ROIs, motion-path features, adjacent-frame key/value embeddings, submaps, long-term motion contexts, or previously solved plans. The common design question is not whether to keep history, but which parts of history remain causally useful for future inference.

Domain	Memory unit	Motion-aware principle
Audio-visual LLM streaming	KV pairs in audio and visual caches	Retain high-attention, low-redundancy states; allocate modality-specific budgets
SAM2-based tracking	Frame-mask memories	Admit writes by motion consistency or short-horizon temporal validity
VOS / VSOD	ROIs, path regions, warped neighborhoods, adjacent-frame values	Restrict storage or retrieval to motion-relevant spatial support
SLAM / planning / prediction	Submaps, motion contexts, past plans	Select by motion state, long-term context alignment, or latent retrieval

A recurring misconception is that motion-aware memory selection always means explicit optical flow. Some methods do use flow or flow-like displacement fields, but others rely on Kalman prediction, trajectory backtracking, local similarity between neighboring representations, or latent motion-context matching. OmniMem is explicit on this point: it is best understood as a motion-/change-aware memory compressor, but it does not explicitly estimate optical flow or “motion” in the classical computer-vision sense (Sun et al., 26 May 2026).

2. Selection criteria and mathematical forms

The literature implements motion-aware memory selection through several distinct scoring regimes. One important family scores a memory item by combining utility and irreducibility. OmniMem computes an importance score from accumulated attention mass,

$a_k = \sum_{q\in|Q|} A_{q,k},$

and a redundancy score from local similarity,

$s_k = [\cos(H_k,H_{k+1})+\cos(H_{k-1},H_k)]/2.$

The final retention score is

$\psi_k = a_k^{\lambda}(1-s_k),$

and the cache keeps the top- $K$ items by $\psi_k$ . High attention mass indicates that the next layer actually uses the KV pair; high local similarity indicates that neighboring positions can replace it. The retained states therefore tend to be both heavily used and locally non-redundant, which in video naturally favors temporally changing content (Sun et al., 26 May 2026).

A second family uses explicit motion consistency for memory admission. In SAMURAI, the current-frame mask is selected by combining SAM 2’s mask affinity with a Kalman-filter-derived motion score,

$\mathcal{M}^*=\arg\max_{\mathcal{M}_i}\big(\alpha_{kf}\cdot s_{kf}(\mathcal{M}_i)+(1-\alpha_{kf})\cdot s_{mask}(\mathcal{M}_i)\big),$

and a past frame enters the memory bank only if mask affinity, object occurrence, and motion score satisfy threshold criteria within a bounded look-back window (Yang et al., 2024). SENTRY generalizes this idea from box-level motion plausibility to short-horizon trajectory validity. It backtracks each candidate mask into a tracklet $\xi_i=(M_i^t,\dots,M_i^{t-\tau})$ , compares it with the reference target tracklet $\eta$ and neighbor tracklets by IoU-averaged similarity, and admits only the candidate that is most temporally and geometrically consistent (Alansari et al., 23 Jun 2026).

A third family performs motion-aware retrieval rather than motion-aware writing. MAMP warps reference features and masks toward the query frame using RAFT-estimated motion offsets, restricts matching to a local ROI around the motion-adjusted location, and then applies TopK filtering over the candidate set (Miao et al., 2021). MUNet does not explicitly prune memory entries, but it changes which memory entries are effectively selected by constructing a motion-uncertainty-aware query embedding from a correlation cost volume and a motion-aware spatial attention module; memory retrieval is therefore no longer purely appearance-driven (Sun et al., 2021). The ASTM framework for video salient object detection similarly uses soft attention over key/value embeddings from adjacent frames, so memory selection is performed as a weighted read rather than as hard admission (Zhao et al., 2022).

A fourth family combines motion with confidence and spatial support. MoSAM’s Spatial-Temporal Memory Selection first filters frames by IoU and occlusion scores and then filters pixels within selected frames by thresholding the foreground probability map. RAVOS stores only the union of motion-path regions traced between $R_{t-1}^i$ and predicted $\hat R_t^i$ , replacing full-frame memory with path-constrained memory (Yang et al., 30 Apr 2025, Miao et al., 2022). These methods treat motion not merely as a cue for object localization, but as a criterion for deciding which memory regions are trustworthy enough to store.

3. SAM2-based visual tracking: from recency to validated writes

The clearest recent concentration of work on motion-aware memory selection appears in SAM2-based zero-shot tracking. The baseline problem is sharply defined: SAM2 stores recent memories in a fixed window or writes the most confident mask, even when that mask is temporally wrong. Under occlusion, rapid motion, blur, and distractors, this causes error propagation because memory is not an archive but a conditioning signal for future decoding (Yang et al., 2024, Alansari et al., 23 Jun 2026).

SAMURAI addresses this by combining Kalman-filter-based motion modeling with rule-based memory gating. Its state vector is

$s_k = [\cos(H_k,H_{k+1})+\cos(H_{k-1},H_k)]/2.$ 0

and a frame is written into memory only if mask affinity, object occurrence, and motion consistency pass thresholds such as $s_k = [\cos(H_k,H_{k+1})+\cos(H_{k-1},H_k)]/2.$ 1, $s_k = [\cos(H_k,H_{k+1})+\cos(H_{k-1},H_k)]/2.$ 2, and $s_k = [\cos(H_k,H_{k+1})+\cos(H_{k-1},H_k)]/2.$ 3. The method is explicitly training-free and requires no retraining or fine-tuning. Empirically, the ablation isolates memory selection as a major contributor: on LaSOT, adding only motion modeling raises AUC from 68.32 to 70.81, adding only memory selection raises it to 72.67, and using both reaches 74.23 AUC, 82.69 normalized precision, and 80.21 precision. The abstract-level headline reports a 7.1% AUC gain on LaSOT $s_k = [\cos(H_k,H_{k+1})+\cos(H_{k-1},H_k)]/2.$ 4 and a 3.5% AO gain on GOT-10k (Yang et al., 2024).

SENTRY strengthens the write-time validation step. Instead of trusting a single high-confidence mask, it aggregates decoder hypotheses, AMG proposals, and a Kalman fallback; backtracks each candidate over $s_k = [\cos(H_k,H_{k+1})+\cos(H_{k-1},H_k)]/2.$ 5 frames; constructs target and neighbor tracklets; and applies neighbor-aware cycle-consistent matching with Hungarian assignment before committing a memory write. The method is therefore “refine-before-write,” and the paper’s diagnostics directly tie the gain to memory quality rather than merely better per-frame output: false memory writes are reduced by about 37.5%, severe writes by about 51%, and written false-positive area by about 31%. Despite these checks, the SAM2-L version runs at 32.8 FPS on an A100 and adds only about 0.4–0.6 GB VRAM (Alansari et al., 23 Jun 2026).

MoSAM modifies the same memory-writing problem from the video segmentation side. Its Motion-Guided Prompting injects sparse keypoint motion and dense optical-flow motion into SAM2 as prompts, while Spatial-Temporal Memory Selection filters unreliable frames and unreliable pixels. The implementation uses a memory bank length of 7 with $s_k = [\cos(H_k,H_{k+1})+\cos(H_{k-1},H_k)]/2.$ 6 and $s_k = [\cos(H_k,H_{k+1})+\cos(H_{k-1},H_k)]/2.$ 7, and the ablations show that temporal selection alone, spatial selection alone, and both together all improve over the prompted baseline, with the full temporal-plus-spatial selection giving the best LVOS-v1 result of 84.6 / 79.3 / 89.9 in the reported table (Yang et al., 30 Apr 2025).

Taken together, these trackers establish a central principle: in autoregressive visual memory, motion-aware selection is primarily a write-validation problem. The key technical dispute is not whether memory helps, but whether recency or confidence is an adequate proxy for future usefulness. The cited results indicate that it is often not.

4. Video object segmentation and saliency: motion-aware writes, reads, and regions

In video object segmentation, motion-aware memory selection appears in both storage design and matching design. RAVOS is exemplary on the storage side. Its fast object motion tracker predicts ROIs using only box positions from previous frames, with a quadratic motion template over box corners, and the paper reports about 5000 FPS on a single GPU for this tracker. Memory is then restricted to the union of motion-path regions,

$s_k = [\cos(H_k,H_{k+1})+\cos(H_{k-1},H_k)]/2.$ 8

so the bank stores only the foreground object and nearby background along the movement corridor. This materially reduces redundancy: on DAVIS 2017 validation, motion path memory alone reduces matching time from 12.2 ms to 3.2 ms and memory size from 36.3 K to 19.2 K, while the full model reaches 86.1 J&F at 42 FPS (Miao et al., 2022).

MAMP is exemplary on the retrieval side. Its memory bank contains high-resolution frame features and downsampled masks from long-term memory $s_k = [\cos(H_k,H_{k+1})+\cos(H_{k-1},H_k)]/2.$ 9 and $\psi_k = a_k^{\lambda}(1-s_k),$ 0 and short-term memory $\psi_k = a_k^{\lambda}(1-s_k),$ 1. Rather than searching all memory positions or using a fixed local window around the raw query coordinate, it warps reference features and masks with RAFT-estimated motion offsets, searches within a motion-adjusted ROI, and applies TopK correlated location selection. The reported best setting is Top 36, with Top 9 also strong. Replacing vanilla local spatio-temporal matching with motion-aware matching improves performance by +3.1% on DAVIS-2017 and +6.4% on YouTube-VOS 2018 (Miao et al., 2021).

MUNet addresses a different weakness of STM-style VOS: appearance-only matching. Its MU-Layer constructs a 4D correlation cost volume over a $\psi_k = a_k^{\lambda}(1-s_k),$ 2 search window, estimates displacement with a soft-argmin, and derives an uncertainty map from the maximum matching response. The resulting motion feature is fused with semantic features through a Motion-aware Spatial Attention Module,

$\psi_k = a_k^{\lambda}(1-s_k),$ 3

Memory selection remains dense matching, but the query that addresses memory becomes motion-aware and uncertainty-aware. On DAVIS17 validation, the full model reaches 81.1% $\psi_k = a_k^{\lambda}(1-s_k),$ 4, and the ablation attributes a +4.6% gain over baseline to the combined motion-aware design (Sun et al., 2021).

The ASTM-based motion-aware memory network for fast video salient object detection uses a smaller, explicitly local memory pool: the current frame is the query, the adjacent frames $\psi_k = a_k^{\lambda}(1-s_k),$ 5 and $\psi_k = a_k^{\lambda}(1-s_k),$ 6 are memory, and retrieval is a softmax-weighted STM read over high-level key/value embeddings. Motion enters through multitask supervision on a motion label defined by $\psi_k = a_k^{\lambda}(1-s_k),$ 7, so the representation is trained to emphasize moving boundaries and changing object parts. The paper reports nearly 100 FPS during inference and shows that expanding memory from two adjacent frames to four frames lowers FPS by about 20% without meaningful metric improvement (Zhao et al., 2022).

These segmentation systems make a useful distinction. Some methods make memory smaller and cleaner by choosing what to store; others make retrieval more reliable by choosing where and how to read. Motion-aware memory selection can therefore be a write policy, a read policy, or both.

5. Streaming multimodal and generative systems

In streaming audio-visual LLMs, motion-aware memory selection becomes a compression problem over KV caches. OmniMem addresses long-video inference, where KV memory grows linearly with video length and audio-visual token counts are highly imbalanced. The method separates audio and visual caches through Audio-Visual Budget Allocation and scores each KV pair by attention-based importance and local redundancy. The default reported value is $\psi_k = a_k^{\lambda}(1-s_k),$ 8, and the paper states that performance is fairly stable for small $\psi_k = a_k^{\lambda}(1-s_k),$ 9 but degrades when the attention term dominates and redundancy is ignored. It also allocates budgets non-uniformly across layers using a compressibility score and a softmax with temperature $K$ 0, with earlier layers generally receiving larger budgets. The ablations show that removing the perturbation-aware score hurts, but removing separate audio-visual budgets hurts more. On video-SALMONN 2+ (8B), OmniMem improves over the best baselines by 4.5 points on VideoMME Long, 3.0 on LVBench, and 2.2 on LVOmni-Bench, with another 1–2 points from budget-aware fine-tuning (Sun et al., 26 May 2026).

This formulation is important because it broadens the term “motion-aware.” OmniMem’s retained memory naturally favors temporally changing, non-repetitive content, but it does so without explicit optical flow. The mechanism is instead perturbation-aware: a token is retained when it is both used by attention and hard to replace by nearby tokens (Sun et al., 26 May 2026). This suggests that, in transformer memory compression, motion awareness can be operationalized as local novelty under actual model usage rather than as classical motion estimation.

PipeFlow uses the term in a different but related sense. Its motion-aware frame selection decides which frames are worth sending through expensive DDIM inversion and joint editing. For adjacent frames, it computes grayscale SSIM and dense optical flow with the Farnebäck algorithm; a frame is selected if $K$ 1 or mean flow magnitude $K$ 2, while the first and final frames are always kept. Skipped low-motion frames are later reconstructed with RIFE interpolation, and segment boundaries are also smoothed with RIFE. The paper attributes its speedups to the combination of skipping low-motion frames, pipelined segment processing, and memory-aware scheduling, reporting 9.6× speedup over TokenFlow and 31.7× over DMT on 240-frame videos (Munir et al., 30 Dec 2025).

The contrast between OmniMem and PipeFlow is instructive. One performs token-level perturbation-aware compression inside a model’s temporal memory; the other performs frame-level motion filtering before expensive generative processing. Both are motion-aware selection policies, but they act at different representational levels.

6. Generalizations beyond frame memory

Several works extend motion-aware memory selection beyond conventional frame or token buffers. In video prediction, the “Long-term Motion Context Memory” stores motion contexts rather than images. The method “Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning” trains a memory matrix $K$ 3 through a storing phase on long sequences and a matching phase on short sequences with frozen memory. Retrieval is based on cosine-similarity addressing over local motion queries,

$K$ 4

The paper argues that local query decomposition improves alignment because different spatial parts can retrieve different long-term motion prototypes, and its ablations report the local-context variant as the strongest model, especially on the last 10 frames of long-horizon prediction (Lee et al., 2021).

In calibration-free monocular SLAM, VGGT-Motion turns memory selection into submap construction. Its Motion-Aware Submap Construction classifies frames into Static, Turning, and Linear motion using optical-flow-derived static ratio and turning score, prunes redundant static frames, selects dynamic frames by parallax, and treats turning intervals as indivisible submaps. The default hyperparameters include $K$ 5, $K$ 6, $K$ 7, $K$ 8, and $K$ 9. In the ablations, parallax-based selection with static pruning yields ATE 26.98 and ATE* 15.36, while topology-aware turning encapsulation improves over temporal slicing, reducing ATE from 26.98 to 24.56 and drift from 1.58% to 1.41% (Xiong et al., 5 Feb 2026).

In motion planning, Motion Memory stores solved planning problem/solution pairs, hallucinates many compatible environments for each plan, learns an embedding with triplet loss, and retrieves the nearest plan cluster for a new problem by

$\psi_k$ 0

Here the memory item is not a frame but a past motion plan. The reported effect is large: planning time can be reduced by up to 89%, and the random-selection ablation shows that arbitrary reuse of past plans increases planning time, making the learned selection mechanism essential (Das et al., 2023).

At the opposite end of abstraction, “Cell Motion Alignment as Polarity Memory Effect” describes a biological memory process in which each cell’s polarity vector relaxes toward its recent velocity direction on a timescale $\psi_k$ 1. Alignment emerges when the persistent length $\psi_k$ 2 exceeds the average intercellular spacing $\psi_k$ 3, yielding the condition $\psi_k$ 4. Although this is not a machine-learning memory bank, it provides a theoretical analogue: motion-aligned collective behavior can emerge when a system retains recent directional information long enough for future interactions to reinforce it (Matsushita et al., 2019).

A plausible implication of these cross-domain results is that motion-aware memory selection is better understood as a general control principle for temporal computation than as a single architecture. The retained object may be a token, mask, region, tracklet, submap, memory slot, or plan, but the same design problem recurs: preserve dynamic evidence that will remain useful, and prevent redundant or corrupted history from steering future inference.