Latest Object Memory (LOM) in Video Segmentation

• Latest Object Memory (LOM) is an online memory mechanism that maintains per-slot, latest-state features for consistent video instance segmentation.
• It employs a foreground-gated update rule, where only slots with confident object evidence are refreshed, ensuring both stability and appearance freshness.
• The integration with Decoupled Object Association (DOA) and occupancy-guided matching boosts segmentation accuracy, achieving state-of-the-art performance on challenging benchmarks.

Searching arXiv for the cited papers and closely related memory-based object tracking/segmentation work. Latest Object Memory (LOM) is an online memory mechanism for query-based Video Instance Segmentation (VIS) in which a dense tensor $\mathcal M_t\in\mathbb R^{N\times C}$ stores the most up-to-date feature for each object slot. Its purpose is to preserve a stable correspondence—a fixed slot or index $n$—for each physical object across frames $1..T$, while explicitly updating only foreground slots so that memory remains centered on the latest appearance of each object. In the formulation introduced with Latest Object Memory Management (LOMM), LOM operates together with Decoupled Object Association (DOA), occupancy-aware assignment, and two tracking heads, thereby coupling identity management with per-slot feature persistence (Lee et al., 26 Jul 2025).

1. Conceptual definition

In the LOM formulation, each frame $I_t$ produces a set of instance queries $\tilde Q_t\in\mathbb R^{N\times C}$, mask logits $M_t\in\mathbb R^{N\times H\times W}$, and class scores $P_t\in\mathbb R^{N\times(K+1)}$. The central problem is not merely per-frame segmentation, but temporally consistent indexing: a physical object should continue to occupy the same slot across a video, even when scene composition changes.

LOM addresses this by maintaining $\mathcal M_t$, a per-slot cache of the latest aligned object representation. The memory is updated only after object representations have been reconstructed into slot-consistent features $\dot Q_t\in\mathbb R^{N\times C}$. The design therefore separates three issues that are often conflated in VIS pipelines: producing candidate object queries, assigning those queries to persistent slots, and deciding when a slot’s memory should be overwritten.

A useful distinction within the same framework is between LOM and LOMM. LOM denotes the memory tensor and its update rule; LOMM denotes the larger system that combines LOM with DOA for temporally consistent video instance segmentation. This distinction matters because the reported gains arise from both the memory mechanism and the association strategy, not from either component alone.

2. State variables and pipeline placement

LOM is embedded in a multi-stage pipeline. The segmentation network $\mathcal S$ first produces $n$0. The previous memory $n$1 is then used to track only existing objects via $n$2, producing $n$3. Occupancy-guided Hungarian Matching aligns $n$4 with $n$5 and assigns newly appearing objects to free slots, yielding $n$6. An adaptive anchor query $n$7 is constructed from $n$8 and $n$9, and a second tracking head $1..T$0 outputs the final aligned features $1..T$1, which are then used for memory update.

Symbol	Shape	Role
$1..T$2	$1..T$3	per-slot latest memory
$1..T$4	$1..T$5	candidate, tracked, aligned, and final slot features
$1..T$6	$1..T$7	occupancy mask for slots ever used by a foreground object
$1..T$8	$1..T$9	foreground-object probability per slot
$I_t$0	$I_t$1	semantic classification outputs

The occupancy state $I_t$2 records which slots have ever been used by a foreground object. This makes slot reuse explicit: existing objects remain confined to occupied slots, while newly appeared objects can only be placed into free ones. The foreground probability is computed from the classifier outputs as

$I_t$3

This quantity is the control signal for memory overwrite. Intuitively, it converts semantic evidence into a continuous update gate.

3. Memory update rule and latest-state semantics

The defining operation of LOM is the foreground-gated memory update

$I_t$4

Here $I_t$5 is broadcast by row from $I_t$6 to $I_t$7, and $I_t$8 denotes element-wise multiplication. If slot $I_t$9 is currently classified as no-object, so that $\tilde Q_t\in\mathbb R^{N\times C}$0, the previous memory $\tilde Q_t\in\mathbb R^{N\times C}$1 is preserved. If slot $\tilde Q_t\in\mathbb R^{N\times C}$2 is a confident foreground object, so that $\tilde Q_t\in\mathbb R^{N\times C}$3, the memory is replaced by the current aligned representation $\tilde Q_t\in\mathbb R^{N\times C}$4.

This latest-state semantics distinguishes LOM from momentum-style or similarity-weighted memory schemes. The memory does not attempt to maintain a smoothed historical prototype; rather, it is explicitly centered on the latest appearance judged to be foreground. A plausible implication is that identity preservation is delegated to slot consistency and association, while appearance freshness is handled by the overwrite gate. That division of labor is reflected in the ablation studies reported for the full system.

A common confusion is to assume that memory is updated from raw per-frame queries. In fact, the update occurs only after the pipeline has reconstructed $\tilde Q_t\in\mathbb R^{N\times C}$5, the aligned feature set that already respects memory indices. LOM is therefore not merely a cache of segmentation queries; it is a cache of slot-consistent object states.

4. Association, occupancy, and adaptive anchoring

DOA splits index assignment into two phases. In the first phase, existing-object tracking uses a small transformer $\tilde Q_t\in\mathbb R^{N\times C}$6 with input $\tilde Q_t\in\mathbb R^{N\times C}$7. Cross-attention uses each memory vector $\tilde Q_t\in\mathbb R^{N\times C}$8 as a query into the current frame’s candidate set $\tilde Q_t\in\mathbb R^{N\times C}$9. The output is non-zero only for slots satisfying $M_t\in\mathbb R^{N\times H\times W}$0.

In the second phase, Occupancy-guided Hungarian Matching places remaining detections into free slots. The procedure builds occupied-slot indices $M_t\in\mathbb R^{N\times H\times W}$1 and free-slot indices $M_t\in\mathbb R^{N\times H\times W}$2, performs a first Hungarian assignment for tracked objects, identifies unmatched detections, and then performs a second Hungarian assignment that maps those unmatched detections into empty slots. The stated distance metric is typically negative cosine similarity or $M_t\in\mathbb R^{N\times H\times W}$3 distance.

This occupancy constraint prevents reassignment across slot types: an existing object is not remapped into a free slot, and a newly appearing object is not forced into a previously occupied one. The resulting aligned tensor $M_t\in\mathbb R^{N\times H\times W}$4 is then blended with old memory to produce the adaptive anchor query

$M_t\in\mathbb R^{N\times H\times W}$5

where $M_t\in\mathbb R^{N\times H\times W}$6 is per-row cosine similarity in $M_t\in\mathbb R^{N\times H\times W}$7. Final refinement is performed by

$M_t\in\mathbb R^{N\times H\times W}$8

Architecturally, the existing-object tracking head $M_t\in\mathbb R^{N\times H\times W}$9 is described as 3 transformer blocks with cross-attention from memory queries into $P_t\in\mathbb R^{N\times(K+1)}$0 keys and values. The final tracking head $P_t\in\mathbb R^{N\times(K+1)}$1 takes $P_t\in\mathbb R^{N\times(K+1)}$2 and refines both existing and new slots. LOM therefore depends on a specific interface: memory is meaningful only because the association stages make slot identity explicit.

5. Empirical behavior in video instance segmentation

On the YouTube-VIS 2022 validation set with a ResNet-50 backbone, the reported baseline without memory is AP $P_t\in\mathbb R^{N\times(K+1)}$3. Adding LOM only raises AP to $P_t\in\mathbb R^{N\times(K+1)}$4, while adding DOA only gives AP $P_t\in\mathbb R^{N\times(K+1)}$5. The full LOMM system, combining LOM and DOA, reaches AP $P_t\in\mathbb R^{N\times(K+1)}$6, corresponding to an improvement of approximately $P_t\in\mathbb R^{N\times(K+1)}$7 over the baseline (Lee et al., 26 Jul 2025).

The memory-type ablation sharpens the role of the update rule. Similarity-weighted memory yields AP $P_t\in\mathbb R^{N\times(K+1)}$8, and momentum memory with 1% update yields AP $P_t\in\mathbb R^{N\times(K+1)}$9, whereas latest-state LOM yields AP $\mathcal M_t$0. Within the reported experiments, this indicates that foreground-gated latest-state replacement outperforms both similarity and momentum schemes.

The abstract further reports that LOMM achieves a state-of-the-art AP score of $\mathcal M_t$1 on YouTube-VIS 2022 and sets new state-of-the-art on both YouTube-VIS and OVIS benchmarks. The same source states that LOMM improves Average Recall and all AP@50/AP@75 tiers. Taken together, these results position LOM as a mechanism for long-term instance tracking in challenging long videos, especially where objects frequently appear and disappear.

6. Relation to adjacent object-memory models

LOM belongs to a broader family of explicit object-memory mechanisms that maintain persistent state under partial observability, but its technical instantiation is specific to query-based VIS. In “Out of Sight, Still in Mind,” Huang et al. introduce LOOM, a Latent Occluded Object Memory for multi-object manipulation reasoning and planning. LOOM maintains a per-object latent token and, when an object becomes occluded, performs the slot update $\mathcal M_t$2, copying the predicted latent state rather than dropping the token. In simulation, LOOM achieves relational prediction F1 of $\mathcal M_t$3 versus $\mathcal M_t$4 for an implicit memory baseline across object counts, and in real-world trials it attains success $\mathcal M_t$5, while the baseline is effectively $\mathcal M_t$6 owing to occlusion failures (Huang et al., 2023).

ChronoTrack extends the explicit-memory theme to 3D Single Object Tracking by using a compact set of learnable object tokens $\mathcal M_t$7 with default configuration $\mathcal M_t$8 and $\mathcal M_t$9. Its long-term memory is trained with a temporal consistency loss and a memory cycle consistency loss, and on KITTI it reports mean Success/Precision of $\dot Q_t\in\mathbb R^{N\times C}$0, with runtime of 42 FPS on a single RTX 4090 GPU (Yoo et al., 15 Apr 2026).

These related systems differ in task, representation, and supervision. LOOM is object-oriented memory for relational reasoning and planning over partial-view point clouds; ChronoTrack is long-term token memory for 3D-SOT; LOM is a per-slot latest-state cache for VIS. Nevertheless, all three instantiate an explicit-memory design in which object persistence is handled through structured slots or tokens rather than through a monolithic sensory history alone. This suggests a common methodological pattern: temporal consistency improves when identity, occupancy, and memory update are encoded as first-class state variables rather than left implicit in raw sequence modeling.