XMem: Unified Memory for Video Segmentation

Updated 12 July 2026

XMem is a video object segmentation architecture that decomposes memory into sensory, working, and long-term stores, each tailored for different temporal roles.
It overcomes previous GPU memory limitations by updating a rapidly changing sensory memory and consolidating features into high-resolution working and compact long-term memories.
Empirical results and extensions, like XMem++ and Lazy-XMem, demonstrate its state-of-the-art performance on long and short video benchmarks with efficient real-time processing.

XMem is a video object segmentation architecture for long videos with unified feature memory stores inspired by the Atkinson-Shiffrin memory model. It was introduced to overcome a central limitation of prior video object segmentation systems: for videos longer than a minute, a single feature memory model tightly links memory consumption and accuracy. XMem instead combines a rapidly updated sensory memory, a high-resolution working memory, and a compact sustained long-term memory, together with a memory potentiation algorithm and a new memory reading mechanism, so that it greatly exceeds state-of-the-art performance on long-video datasets while remaining on par with state-of-the-art methods on short-video datasets (Cheng et al., 2022).

1. Problem setting and conceptual basis

Video object segmentation (VOS) requires propagating object masks through time under occlusion, appearance variation, motion, and long temporal horizons. Prior work in this area typically used one type of feature memory. In that regime, storing numerous high-resolution memory features leads to prohibitive GPU memory growth and performance decay in long sequences. XMem was formulated against this background as a memory-augmented VOS system whose design is explicitly inspired by the Atkinson-Shiffrin model from cognitive psychology, which posits sensory, working, and long-term memory as distinct but connected components (Cheng et al., 2022).

The key conceptual move in XMem is to separate temporal roles rather than forcing one memory substrate to handle all time scales. Sensory memory captures rapidly changing local evidence, working memory retains high-resolution short- to medium-term context, and long-term memory stores compact prototypes for sustained retrieval. This decomposition addresses both memory consumption and temporal robustness. A plausible implication is that XMem’s contribution is not merely a larger memory bank, but a reorganization of memory according to temporal function.

Architecturally, XMem consists of a query encoder based on ResNet-50, a value encoder based on ResNet-18, and a decoder for producing segmentation masks. The decoder uses readout features from the memory stores together with the sensory state, yielding a system in which memory management is a first-class component rather than a byproduct of feature caching (Cheng et al., 2022).

2. Multi-store memory architecture and core algorithms

The three memory stores in XMem are distinct in representation, update cadence, and intended temporal horizon.

Memory store	Implementation	Update behavior
Sensory memory	Convolutional GRU hidden state $\mathbf{h}_t \in \mathbb{R}^{h \times H \times W}$ with $h=64$	Updated every frame
Working memory	Keys $\mathbf{k}^w \in \mathbb{R}^{k \times T H W}$ , values $\mathbf{v}^w \in \mathbb{R}^{v \times T H W}$ with $k=64$ , $v=512$	Inserted every $r$ frames
Long-term memory	Compact prototypical keys and values	Appended through consolidation; bounded by LFU eviction

Sensory memory is the rapidly evolving temporal representation. It is implemented as the hidden state of a Convolutional GRU and updated on every frame using multi-scale decoder features. When a new working-memory entry is created every $r$ -th frame, a deep update refreshes the sensory state through a second GRU so that redundant information can be pruned and new evidence incorporated (Cheng et al., 2022).

Working memory is a high-resolution memory bank storing keys extracted from the image and values extracted from the image and predicted mask. Its capacity is bounded between $T_{\min}$ and $T_{\max}$ ; once full, it triggers consolidation into long-term memory. Long-term memory, by contrast, is a compact set of prototypical memory features. When it nears capacity, XMem uses a Least-Frequently-Used eviction policy, with usage measured through affinity-based access, so that GPU memory remains strictly bounded even for very long sequences (Cheng et al., 2022).

The memory read mechanism is centered on an anisotropic $h=64$ 0 similarity. For query features $h=64$ 1 and memory keys $h=64$ 2, XMem computes

$h=64$ 3

$h=64$ 4

where $h=64$ 5 is a shrinkage term and $h=64$ 6 is a selection term. These quantities modulate confidence in each memory element and the relative importance of channels during retrieval (Cheng et al., 2022).

Consolidation from working to long-term memory is governed by memory potentiation. XMem first selects the top- $h=64$ 7 most-used candidate working-memory elements, then aggregates candidate values in key space:

$h=64$ 8

This procedure is described as anti-aliasing or enrichment in key space. With default parameters $h=64$ 9, the working-memory-to-long-term-memory compression ratio is 6328%, while with 10,000 long-term memory elements the maximum GPU usage is less than 1.4GB at 480p (Cheng et al., 2022).

3. Empirical behavior, benchmark results, and known limitations

On the Long-time Video dataset, XMem reports $\mathbf{k}^w \in \mathbb{R}^{k \times T H W}$ 0 and $\mathbf{k}^w \in \mathbb{R}^{k \times T H W}$ 1 at $\mathbf{k}^w \in \mathbb{R}^{k \times T H W}$ 2, and $\mathbf{k}^w \in \mathbb{R}^{k \times T H W}$ 3 and $\mathbf{k}^w \in \mathbb{R}^{k \times T H W}$ 4 at $\mathbf{k}^w \in \mathbb{R}^{k \times T H W}$ 5. On shorter-video benchmarks, it reports YouTubeVOS 2018 $\mathbf{k}^w \in \mathbb{R}^{k \times T H W}$ 6, $\mathbf{k}^w \in \mathbb{R}^{k \times T H W}$ 7, $\mathbf{k}^w \in \mathbb{R}^{k \times T H W}$ 8; DAVIS 2017 val $\mathbf{k}^w \in \mathbb{R}^{k \times T H W}$ 9, $\mathbf{v}^w \in \mathbb{R}^{v \times T H W}$ 0; and DAVIS 2016 val $\mathbf{v}^w \in \mathbb{R}^{v \times T H W}$ 1, $\mathbf{v}^w \in \mathbb{R}^{v \times T H W}$ 2. For multi-object scenarios on a V100 GPU, it processes frames at up to 22.6 FPS. With BL30K pretraining, DAVIS 2017 val improves to $\mathbf{v}^w \in \mathbb{R}^{v \times T H W}$ 3, $\mathbf{v}^w \in \mathbb{R}^{v \times T H W}$ 4 (Cheng et al., 2022).

These results establish XMem as both a long-video method and a competitive short-video method. The long-video claim is especially central: XMem maintains stable performance as video length increases, whereas alternatives such as STCN and AOT show a clearer drop. The appendix cited in the source summary further reports recovery after lengthy occlusion because long-term context is retained (Cheng et al., 2022).

At the same time, later comparative studies identify limitations in XMem’s pixel-level memory reading. Cutie characterizes XMem as a bottom-up pixel-level reader in which each query pixel independently retrieves support from stored pixel-level memory features. According to that comparison, the absence of global semantic context makes XMem more vulnerable to matching noise, especially with distractors and occlusion. On the MOSE benchmark, Cutie improves by 8.7 J&F over XMem with a similar running time, which is presented as evidence for the benefits of object-level memory reading over purely pixel-level reading (Cheng et al., 2023).

Cross-domain evaluation also exposes domain-shift sensitivity. On the Industrial Metallic Dataset, where objects are metallic, texture-less, and highly reflective, XMem obtains a mean IoU of 0.746 and recall of 0.922, compared with 0.863 and 1.000 on DAVIS-2017. SAM2 slightly outperforms it on IMD with mean IoU 0.770 and recall 0.980. The benchmark discussion attributes the degradation to reflective and texture-less surfaces, strong reflections, specular highlights, and complex arrangements, while still noting XMem’s temporal consistency and efficiency (Ma et al., 15 Sep 2025).

4. Extensions and descendants: XMem++, Lazy-XMem, and production-oriented adaptations

XMem++ extends the original model toward production-level video segmentation from few annotated frames. Its defining modification is a permanent memory module that persistently maintains all annotated frames, regardless of their recency in the temporal sequence. In the notation given in the source summary, the permanent memory bank is $\mathbf{v}^w \in \mathbb{R}^{v \times T H W}$ 5, and inference attends over all annotated-frame embeddings rather than relying only on temporally refreshed memories. XMem++ also introduces an iterative and attention-based frame suggestion mechanism that computes the next best frame for annotation, and it is described as real-time and not requiring retraining after each user input (Bekuzarov et al., 2023).

The relation between XMem and XMem++ is therefore not merely incremental. XMem was optimized around long-horizon propagation from sparse supervision with bounded memory; XMem++ explicitly targets multiple user-selected frames with varying appearances of the same object or region. This suggests a shift from first-frame-centric semi-supervised VOS toward annotation-efficient production workflows.

A different extension is Lazy-XMem, introduced in the context of Lazy Video Object Segmentation for online recorded sequences. Lazy-XMem adds on-the-fly uncertainty-based user interactions, using pixel-wise entropy

$\mathbf{v}^w \in \mathbb{R}^{v \times T H W}$ 6

and object-region aggregation

$\mathbf{v}^w \in \mathbb{R}^{v \times T H W}$ 7

When uncertainty rises, Lazy-XMem triggers either user corrections or pseudo-corrections and refines masks with SAM-HQ. It also adds Interaction-Driven Updates and Uncertainty-Driven Updates to XMem’s memory management. On LVOS, the summary reports that Lazy-XMem boosts the percentage of frames segmented above IoU thresholds by approximately 13 points over the XMem baseline while requiring about 1.05% of frames to receive user corrections, averaging one correction every about 18.4 seconds (Vujasinović et al., 2024).

5. Use as a component in broader systems

XMem and XMem++ are frequently embedded into larger pipelines in which segmentation propagation is only one stage. In VolETA, XMem++ tracks and propagates a SAM-generated reference object mask to all frames, producing $\mathbf{v}^w \in \mathbb{R}^{v \times T H W}$ 8. These masks are combined with food masks, RGBD images, and camera poses to generate RGBA inputs for NeuS2. The full pipeline reports 10.97% MAPE on the MTF dataset for food volume estimation, and the description emphasizes that segmentation quality directly affects mesh cleanliness, scaling, and final volume prediction (AlMughrabi et al., 2024).

In surgical video, a two-stage framework fine-tunes SAM with LoRA on EndoVis17 and then uses a pre-trained XMem++ model to propagate those initial masks through the rest of the video. On the ESD dataset, the reported comparison shows Track Anything at mIoU 86.75, mAcc 95.58, mDice 95.58, and fine-tuned SAM + XMem++ at mIoU 88.17, mAcc 96.16, mDice 96.16 (Yu et al., 2024). A separate surgical study instead positions XMem as a baseline and argues that point tracking plus a lightweight, fully fine-tuned SAM can surpass XMem on EndoVis 2015 while remaining comparable on other datasets; in that account, XMem is described as a strong semi-supervised VOS method that requires an accurate first-frame mask and cannot recognize new objects appearing mid-video (Wu et al., 2024).

Outside classical VOS, RGBTrack integrates XMem with a Kalman filter and a state machine for depth-free 6D pose estimation and tracking from RGB alone. In that system, XMem supplies 2D object masks that support 2D localization, lost-tracking detection, and recovery. The summary reports approximately 22 FPS on an RTX 3090 and frames the XMem mask as essential to the render-and-compare recovery loop (Guo et al., 20 Jun 2025). VideoSeg-R1 similarly uses XMem in a decoupled segmentation-propagation stage: a reasoning model generates spatial cues, SAM2 segments the target in a selected frame, and XMem propagates that seed bidirectionally through the video (Xu et al., 20 Nov 2025).

XMem has also become a point of reference in broader methodological discussions. In “Tool Documentation Enables Zero-Shot Tool-Usage with LLMs,” documentation for XMem is added to a tool-using framework so that an LLM can compose GroundingDINO, SAM, and XMem for video tracking without demonstrations. The summary states that, with only tool documentation, the LLM can “re-invent” the functionality of Track Anything (Hsieh et al., 2023). This use is significant because it treats XMem not just as a benchmark model, but as a callable module in compositional systems.

A separate naming issue concerns “xMem,” a CPU-based framework for estimating GPU memory in deep learning training workloads. Despite the similar string form, that work is unrelated in problem formulation, architecture, and domain: it performs CPU-only dynamic analysis for GPU memory estimation rather than video object segmentation (Shi et al., 23 Oct 2025). The distinction is practical because later literature contains both names.

There is also a source-level ambiguity surrounding DeformGS. The abstract in the provided material states that DeformGS leverages “existing foundation models SAM and XMEM to produce noisy masks,” whereas the accompanying detailed summary states that “XMem is not utilized in the DeformGS pipeline described in the paper” and that there is no reference to XMem in the methodology, experiments, or ablations. The most defensible interpretation is that the role of XMem in DeformGS should be treated cautiously unless checked directly against the paper text (Duisterhof et al., 2023).

Taken together, these later developments position XMem as both a foundational long-term VOS architecture and a stable baseline against which object-level readers, interactive variants, production-oriented annotation schemes, and multimodule downstream systems are evaluated. Its enduring technical identity lies in the Atkinson-Shiffrin-inspired decomposition of memory, bounded long-horizon operation, and explicit treatment of consolidation and retrieval as the core of video object segmentation rather than auxiliary engineering (Cheng et al., 2022).