---
title: Latest Object Memory (LOM) in Video Segmentation
url: https://www.emergentmind.com/topics/latest-object-memory-lom
type: topic
---

# Latest Object Memory (LOM) in Video Segmentation

Searching arXiv for the cited papers and closely related memory-based object tracking/segmentation work.
Latest Object Memory (LOM) is an online memory mechanism for query-based Video Instance Segmentation (VIS) in which a dense tensor \(\mathcal M_t\in\mathbb R^{N\times C}\) stores the most up-to-date feature for each object slot. Its purpose is to preserve a stable correspondence—a fixed slot or index \(n\)—for each physical object across frames \(1..T\), while explicitly updating only foreground slots so that memory remains centered on the latest appearance of each object. In the formulation introduced with Latest Object Memory Management (LOMM), LOM operates together with Decoupled Object Association (DOA), occupancy-aware assignment, and two tracking heads, thereby coupling identity management with per-slot feature persistence [2507.19754].

## 1. Conceptual definition

In the LOM formulation, each frame \(I_t\) produces a set of instance queries \(\tilde Q_t\in\mathbb R^{N\times C}\), mask logits \(M_t\in\mathbb R^{N\times H\times W}\), and class scores \(P_t\in\mathbb R^{N\times(K+1)}\). The central problem is not merely per-frame segmentation, but temporally consistent indexing: a physical object should continue to occupy the same slot across a video, even when scene composition changes.

LOM addresses this by maintaining \(\mathcal M_t\), a per-slot cache of the latest aligned object representation. The memory is updated only after object representations have been reconstructed into slot-consistent features \(\dot Q_t\in\mathbb R^{N\times C}\). The design therefore separates three issues that are often conflated in VIS pipelines: producing candidate object queries, assigning those queries to persistent slots, and deciding when a slot’s memory should be overwritten.

A useful distinction within the same framework is between LOM and LOMM. LOM denotes the memory tensor and its update rule; LOMM denotes the larger system that combines LOM with DOA for temporally consistent video instance segmentation. This distinction matters because the reported gains arise from both the memory mechanism and the association strategy, not from either component alone.

## 2. State variables and pipeline placement

LOM is embedded in a multi-stage pipeline. The segmentation network \(\mathcal S\) first produces \(\tilde Q_t\). The previous memory \(\mathcal M_{t-1}\) is then used to track only existing objects via \(\mathcal T_E\), producing \(\hat Q_t\). Occupancy-guided Hungarian Matching aligns \(\hat Q_t\) with \(\tilde Q_t\) and assigns newly appearing objects to free slots, yielding \(\hat A_t\). An adaptive anchor query \(\bar Q_t\) is constructed from \(\hat A_t\) and \(\mathcal M_{t-1}\), and a second tracking head \(\mathcal T_A\) outputs the final aligned features \(\dot Q_t\), which are then used for memory update.

| Symbol | Shape | Role |
|---|---:|---|
| \(\mathcal M_t\) | \(N\times C\) | per-slot latest memory |
| \(\tilde Q_t,\hat Q_t,\hat A_t,\dot Q_t\) | \(N\times C\) | candidate, tracked, aligned, and final slot features |
| \(\mathcal O_t\) | \(N\) | occupancy mask for slots ever used by a foreground object |
| \(p_t\) | \(N\) | foreground-object probability per slot |
| \(P_t\) | \(N\times(K+1)\) | semantic classification outputs |

The occupancy state \(\mathcal O_t\in\{0,1\}^N\) records which slots have ever been used by a foreground object. This makes slot reuse explicit: existing objects remain confined to occupied slots, while newly appeared objects can only be placed into free ones. The foreground probability is computed from the classifier outputs as
\[
p_t^n = 1 - P_t^{n,(K+1)} = \sum_{k=1}^K P_t^{n,k}.
\]
This quantity is the control signal for memory overwrite. Intuitively, it converts semantic evidence into a continuous update gate.

## 3. Memory update rule and latest-state semantics

The defining operation of LOM is the foreground-gated memory update
\[
\mathcal M_{t} = (1 - p_{t}) \odot \mathcal M_{t-1} + p_{t}\odot \dot Q_{t}.
\]
Here \(p_t\) is broadcast by row from \(\mathbb R^N\) to \(\mathbb R^{N\times C}\), and \(\odot\) denotes element-wise multiplication. If slot \(n\) is currently classified as no-object, so that \(p_t^n\approx 0\), the previous memory \(\mathcal M_{t-1}^n\) is preserved. If slot \(n\) is a confident foreground object, so that \(p_t^n\approx 1\), the memory is replaced by the current aligned representation \(\dot Q_t^n\).

This latest-state semantics distinguishes LOM from momentum-style or similarity-weighted memory schemes. The memory does not attempt to maintain a smoothed historical prototype; rather, it is explicitly centered on the latest appearance judged to be foreground. A plausible implication is that identity preservation is delegated to slot consistency and association, while appearance freshness is handled by the overwrite gate. That division of labor is reflected in the ablation studies reported for the full system.

A common confusion is to assume that memory is updated from raw per-frame queries. In fact, the update occurs only after the pipeline has reconstructed \(\dot Q_t\), the aligned feature set that already respects memory indices. LOM is therefore not merely a cache of segmentation queries; it is a cache of slot-consistent object states.

## 4. Association, occupancy, and adaptive anchoring

DOA splits index assignment into two phases. In the first phase, existing-object tracking uses a small transformer \(\mathcal T_E\) with input \((\mathcal M_{t-1}, \tilde Q_t)\mapsto \hat Q_t\). Cross-attention uses each memory vector \(\mathcal M_{t-1}^n\) as a query into the current frame’s candidate set \(\tilde Q_t\). The output is non-zero only for slots satisfying \(\mathcal O_{t-1}^n=1\).

In the second phase, Occupancy-guided Hungarian Matching places remaining detections into free slots. The procedure builds occupied-slot indices \(I_{\mathrm{occ}}\) and free-slot indices \(I_{\mathrm{free}}\), performs a first Hungarian assignment for tracked objects, identifies unmatched detections, and then performs a second Hungarian assignment that maps those unmatched detections into empty slots. The stated distance metric is typically negative cosine similarity or \(\ell_2\) distance.

This occupancy constraint prevents reassignment across slot types: an existing object is not remapped into a free slot, and a newly appearing object is not forced into a previously occupied one. The resulting aligned tensor \(\hat A_t\) is then blended with old memory to produce the adaptive anchor query
\[
\bar Q_t
=\mathrm{sim}(\hat A_t,\mathcal M_{t-1})\,\hat A_t
+\bigl(1-\mathrm{sim}(\hat A_t,\mathcal M_{t-1})\bigr)\,\mathcal M_{t-1},
\]
where \(\mathrm{sim}(\cdot,\cdot)\) is per-row cosine similarity in \([0,1]\). Final refinement is performed by
\[
\dot Q_t = \mathcal T_A(\bar Q_t,\tilde Q_t).
\]

Architecturally, the existing-object tracking head \(\mathcal T_E\) is described as 3 transformer blocks with cross-attention from memory queries into \(\tilde Q_t\) keys and values. The final tracking head \(\mathcal T_A\) takes \((\bar Q_t,\tilde Q_t)\) and refines both existing and new slots. LOM therefore depends on a specific interface: memory is meaningful only because the association stages make slot identity explicit.

## 5. Empirical behavior in video instance segmentation

On the YouTube-VIS 2022 validation set with a ResNet-50 backbone, the reported baseline without memory is AP \(=33.9\). Adding LOM only raises AP to \(39.2\), while adding DOA only gives AP \(=36.2\). The full LOMM system, combining LOM and DOA, reaches AP \(=41.1\), corresponding to an improvement of approximately \(+7.2\) over the baseline [2507.19754].

The memory-type ablation sharpens the role of the update rule. Similarity-weighted memory yields AP \(\approx 35.8\), and momentum memory with 1% update yields AP \(\approx 35.2\), whereas latest-state LOM yields AP \(\approx 39.2\). Within the reported experiments, this indicates that foreground-gated latest-state replacement outperforms both similarity and momentum schemes.

The abstract further reports that LOMM achieves a state-of-the-art AP score of \(54.0\) on YouTube-VIS 2022 and sets new state-of-the-art on both YouTube-VIS and OVIS benchmarks. The same source states that LOMM improves Average Recall and all AP@50/AP@75 tiers. Taken together, these results position LOM as a mechanism for long-term instance tracking in challenging long videos, especially where objects frequently appear and disappear.

## 6. Relation to adjacent object-memory models

LOM belongs to a broader family of explicit object-memory mechanisms that maintain persistent state under partial observability, but its technical instantiation is specific to query-based VIS. In “Out of Sight, Still in Mind,” Huang et al. introduce LOOM, a Latent Occluded Object Memory for multi-object manipulation reasoning and planning. LOOM maintains a per-object latent token and, when an object becomes occluded, performs the slot update \(z_{t+1}^k \leftarrow \hat z_{t+1}^k\), copying the predicted latent state rather than dropping the token. In simulation, LOOM achieves relational prediction F1 of \(0.974\) versus \(0.798\) for an implicit memory baseline across object counts, and in real-world trials it attains success \(32/35\), while the baseline is effectively \(0/35\) owing to occlusion failures [2309.15278].

ChronoTrack extends the explicit-memory theme to 3D Single Object Tracking by using a compact set of learnable object tokens \(X_t^{\mathrm{FG}}\in\mathbb R^{K\times D}\) with default configuration \(K=32\) and \(D=128\). Its long-term memory is trained with a temporal consistency loss and a memory cycle consistency loss, and on KITTI it reports mean Success/Precision of \(71.8\%/90.1\%\), with runtime of 42 FPS on a single RTX 4090 GPU [2604.13789].

These related systems differ in task, representation, and supervision. LOOM is object-oriented memory for relational reasoning and planning over partial-view point clouds; ChronoTrack is long-term token memory for 3D-SOT; LOM is a per-slot latest-state cache for VIS. Nevertheless, all three instantiate an explicit-memory design in which object persistence is handled through structured slots or tokens rather than through a monolithic sensory history alone. This suggests a common methodological pattern: temporal consistency improves when identity, occupancy, and memory update are encoded as first-class state variables rather than left implicit in raw sequence modeling.

Source: https://www.emergentmind.com/topics/latest-object-memory-lom