Ray-Aware Pointer Memory in 3D Reconstruction

• The paper introduces RAPM, a 3D reconstruction framework that augments persistent spatial memory with explicit viewing-ray information to jointly reason over geometry and viewpoint.
• RAPM employs a novel adaptive retain-or-replace update scheme that mitigates redundancy and distinguishes loop revisits from novel geometry by using joint spatial and angular metrics.
• Empirical evaluations show that RAPM achieves competitive accuracy in dense 3D reconstruction, depth estimation, and pose estimation while maintaining tighter control over memory usage.

Ray-Aware Pointer Memory (RAPM) is a streaming 3D reconstruction framework that augments persistent spatial memory with explicit viewing-ray information so that the system reasons jointly over geometry and viewpoint. Introduced in "Ray-Aware Pointer Memory with Adaptive Updates for Streaming 3D Reconstruction" (Li et al., 7 May 2026), RAPM is designed for dense 3D reconstruction from continuous image streams, where long-horizon stability depends not only on accurate geometric aggregation but also on memory management that remains discriminative under repeated observations and viewpoint change. Its central design choice is to represent each memory element as a pointer carrying 3D position, ray direction, feature embedding, and timestamp, and to update memory through an adaptive retain-or-replace policy rather than appearance-driven feature fusion.

1. Problem setting and motivation

RAPM addresses a limitation of recent feed-forward reconstruction frameworks that integrate observations through persistent memory representations but rely primarily on appearance-based similarity when updating memory. In long sequences, such appearance-driven integration is reported to suffer from two linked failure modes: redundant accumulation when the same surface patch is re-observed from similar views, and instability under viewpoint changes, where appearance alone is ambiguous because of occlusions, perspective, repetitive textures, or pose drift (Li et al., 7 May 2026).

The first failure mode produces bloated memory and over-smoothed features. The second undermines geometric stability because observations that are spatially related but visually altered by viewpoint can be incorrectly merged or misinterpreted. RAPM responds by introducing explicit viewing direction into memory and comparing observations with a joint spatial-plus-ray criterion. The intended effect is to distinguish three cases within a unified mechanism: local redundancy, novel geometry, and loop revisit.

This design changes the role of persistent memory. Instead of acting primarily as a fusion buffer for appearance-aligned features, memory becomes a geometric-and-viewpoint-indexed structure that supports both local updates and global consistency operations. A plausible implication is that the framework treats memory not merely as compression, but as an explicit representation of observation diversity over the scene.

2. Pointer representation and joint spatial-ray reasoning

The scene memory is defined as a set of pointers,

$M = \{m_1, m_2, \ldots, m_N\},$

with each pointer represented as

$m_k = \{x_k, r_k, f_k, t_k\}.$

The fields are specified as follows.

Field	Domain	Meaning
$x_k$	$\mathbb{R}^3$	3D position
$r_k$	$S^2$	unit ray direction from the camera to the point
$f_k$	$\mathbb{R}^d$	feature embedding
$t_k$	—	frame index when the pointer was created

Pointers are produced for each incoming frame by a pointer-image interaction module that predicts depth, camera pose, and candidate 3D points in a global coordinate system. The paper does not define a confidence or uncertainty term, and it does not specify the feature learning losses; $f_k$ is learned implicitly within the reconstruction network.

Given a new pointer $m_k = \{x_k, r_k, f_k, t_k\}.$0 and an existing pointer $m_k = \{x_k, r_k, f_k, t_k\}.$1, RAPM compares both position and viewing direction through

$m_k = \{x_k, r_k, f_k, t_k\}.$2

$m_k = \{x_k, r_k, f_k, t_k\}.$3

and

$m_k = \{x_k, r_k, f_k, t_k\}.$4

In experiments, $m_k = \{x_k, r_k, f_k, t_k\}.$5 and $m_k = \{x_k, r_k, f_k, t_k\}.$6.

This metric supports a unified interpretation of incoming observations. Local redundancy corresponds to small $m_k = \{x_k, r_k, f_k, t_k\}.$7 and small $m_k = \{x_k, r_k, f_k, t_k\}.$8; loop revisit corresponds to small $m_k = \{x_k, r_k, f_k, t_k\}.$9 and large $x_k$0; novel geometry corresponds to large $x_k$1. The paper provides an intuitive geometric justification for combining positional proximity and viewpoint discrepancy in this way, arguing that it improves discrimination between redundancy and loops and reduces drift and incorrect associations, but it does not provide formal guarantees or convergence proofs.

3. Adaptive pointer updates and streaming execution

RAPM replaces traditional fusion-based memory compression with a retain-or-replace scheme. For a new pointer $x_k$2, neighbors are first identified by a spatial radius test,

$x_k$3

If $x_k$4 is empty, $x_k$5 is appended to memory as novel geometry. Otherwise, the closest neighbor under the joint distance $x_k$6 is selected, and the framework applies a stochastic rule: either the new pointer or the selected existing pointer is discarded with equal probability, so exactly one of the two is retained (Li et al., 7 May 2026).

The paper emphasizes that redundancy is detected purely by the presence of neighbors within radius $x_k$7; it does not introduce a threshold on $x_k$8. No feature averaging is used. This is a substantive departure from merge-based approaches such as Point3R, where nearby pointers are fused via feature averaging. In RAPM, avoiding averaging is intended to preserve distinctive landmarks and bound memory growth without diluting local structure.

The per-frame pipeline is defined in six steps. A frame $x_k$9 is ingested; features are encoded, and the pointer-image interaction predicts camera pose $\mathbb{R}^3$0, dense depth or pointmaps, and candidate 3D points in the global frame; candidates are converted to pointers $\mathbb{R}^3$1; each candidate is processed through radius-based neighbor search and retain-or-replace updating; loop candidates are checked and may trigger pose graph refinement and loop-region sparsification; the updated reconstruction, depth, and pose are then output.

In complexity terms, per-frame time is dominated by neighbor search. If a frame contributes $\mathbb{R}^3$2 candidates and the memory holds $\mathbb{R}^3$3 pointers, naive linear search yields $\mathbb{R}^3$4. With a spatial radius index, such as a grid or kd-tree, which the paper notes as an example but does not specify as part of the implementation, expected complexity is $\mathbb{R}^3$5, where $\mathbb{R}^3$6 is the number of neighbors within $\mathbb{R}^3$7. This suggests that RAPM’s scalability depends materially on spatial indexing even though the data structure is left unspecified.

4. Loop candidates, pose refinement, and global consistency

Loop candidate detection in RAPM combines spatial-ray proximity with a temporal gate. Two pointers $\mathbb{R}^3$8 and $\mathbb{R}^3$9 are treated as loop candidates if

$r_k$0

$r_k$1

and

$r_k$2

The first condition enforces spatial coincidence, the second requires sufficiently different viewpoints, and the third avoids trivial short-term revisits (Li et al., 7 May 2026).

When loop candidates are detected, RAPM adds pose constraints between their source frames and runs pose graph refinement. After optimization, the pointer memory is updated in the refined global coordinate system. Within loop regions, a Fisher information-based selection strategy, attributed to Jiang et al. (2024), is applied to sparsify pointers by preserving the most informative ones.

The paper does not specify the explicit optimization objective, residual definitions, or numerical solver for pose graph refinement, and it does not introduce additional regularizers. Accordingly, RAPM’s loop-handling mechanism is conceptually clear but not fully specified at the level of optimization details. A common misunderstanding would be to treat it as a fully detailed pose-graph method; the paper instead specifies when constraints are introduced and how memory is transformed after refinement, while leaving the optimizer itself abstract.

The broader significance of this mechanism is that loop discovery occurs at the pointer level through joint spatial and angular cues. In the paper’s comparison to prior work, this is presented as a capability largely absent in feed-forward reconstruction methods, which typically lack explicit loop-triggered pose correction in a persistent 3D map.

5. Empirical performance and ablation results

The empirical evaluation covers dense 3D reconstruction on 7-Scenes and NRGBD; depth estimation on NYU-v2 (Static), Sintel, Bonn, and KITTI; and camera pose estimation on ScanNet (Static), Sintel, and TUM-dynamics. The reported metrics are accuracy (Acc, $r_k$3), completeness (Comp, $r_k$4), and normal consistency (NC, $r_k$5) for reconstruction; Abs Rel ($r_k$6) and $r_k$7 ($r_k$8) for depth; and ATE ($r_k$9), RPE translation ($S^2$0), and RPE rotation ($S^2$1) for pose (Li et al., 7 May 2026).

For 3D reconstruction mean values, the paper reports on 7-Scenes: Acc $S^2$2, Comp $S^2$3, NC $S^2$4; and on NRGBD: Acc $S^2$5, Comp $S^2$6, NC $S^2$7. Relative to Point3R as an online baseline, RAPM yields lower Acc and Comp, indicating better geometry, while NC is somewhat lower. The paper attributes the NC difference to preserving sharper structures, implying less local smoothing, and to non-uniform sampling that can affect normal estimation.

For depth estimation, the reported Abs Rel and $S^2$8 values are NYU-v2: $S^2$9, $f_k$0; Sintel: $f_k$1, $f_k$2; Bonn: $f_k$3, $f_k$4; KITTI: $f_k$5, $f_k$6. The method is described as competitive or superior to online baselines and strong among feed-forward pairwise approaches.

For camera pose estimation in the online setting, the paper reports ScanNet: ATE $f_k$7, RPE translation $f_k$8, RPE rotation $f_k$9; Sintel: ATE $\mathbb{R}^d$0, RPE translation $\mathbb{R}^d$1, RPE rotation $\mathbb{R}^d$2; and TUM-dynamics: ATE $\mathbb{R}^d$3, RPE translation $\mathbb{R}^d$4, RPE rotation $\mathbb{R}^d$5. These results are characterized as competitive without global optimization, with the paper attributing the behavior to ray-aware loops and adaptive memory.

The ablation on update modes directly targets the memory policy. Averaged across scenes, Point3R reports Acc $\mathbb{R}^d$6, Comp $\mathbb{R}^d$7, NC $\mathbb{R}^d$8; RAPM with retain gives Acc $\mathbb{R}^d$9, Comp $t_k$0, NC $t_k$1; RAPM with replace gives Acc $t_k$2, Comp $t_k$3, NC $t_k$4; and RAPM with random retain-or-replace gives Acc $t_k$5, Comp $t_k$6, NC $t_k$7. The random retain-or-replace variant yields the best Acc and Comp while maintaining NC comparable to the other RAPM variants. The paper interprets this as evidence that avoiding feature averaging preserves structure.

6. Memory behavior, implementation constraints, and relation to prior work

RAPM’s memory behavior is described as bounded in redundant regions because retain-or-replace prevents unbounded accumulation. Across evaluated sequences, the paper reports consistently lower and more stable reserved GPU memory than merge-based Point3R, typically about $t_k$8–$t_k$9 GB versus $f_k$0–$f_k$1 GB. This is presented as evidence of tighter control of redundancy (Li et al., 7 May 2026).

Several implementation details are fixed only partially. The joint-distance weights are specified as $f_k$2 and $f_k$3 in all experiments. By contrast, the redundancy radius $f_k$4, the loop thresholds $f_k$5, $f_k$6, and $f_k$7, and the data structures used for radius queries are not specified. The paper also does not detail the training scheme, supervision, or loss functions for features, depth, and pose, and it does not describe the network architecture. Code availability is not provided.

The stated practical guidance is correspondingly high level: start with $f_k$8 and $f_k$9; choose $m_k = \{x_k, r_k, f_k, t_k\}.$00 according to expected surface sampling density, since too small a value yields insufficient redundancy control and too large a value may drop complementary viewpoints; use a nontrivial $m_k = \{x_k, r_k, f_k, t_k\}.$01 and $m_k = \{x_k, r_k, f_k, t_k\}.$02 to avoid short-term false loops; and use an $m_k = \{x_k, r_k, f_k, t_k\}.$03 consistent with reconstruction noise. The paper further notes that, while not specified, a spatial radius index such as a voxel grid or kd-tree is generally advisable for efficiency.

In the comparison to prior work, RAPM is positioned against three broad alternatives. Appearance-driven fusion methods such as Point3R merge nearby pointers via feature averaging, which can over-smooth landmarks and accumulate redundancy. Implicit token memories such as CUT3R and Spann3R offer scalability but lack explicit spatial-ray reasoning and loop-triggered pose correction in a persistent 3D map. Loop closure in RAPM is discovered through joint spatial and angular cues at the pointer level and triggers pose graph refinement during streaming. This places the method at the intersection of feed-forward reconstruction and explicit memory-based geometric reasoning.

The paper also states several limitations and failure modes. RAPM depends on reasonably accurate streaming pose estimates; large pose errors can perturb memory association and updates. The stochastic update rule is simple and may discard useful observations because it does not explicitly score information content during updates. Textureless areas, strong dynamics, and severe occlusions can reduce the reliability of viewpoint cues, and non-uniform pointer distribution may affect normal consistency. These caveats are important for interpreting the method: RAPM modifies how observations are retained and related, but it does not eliminate the underlying dependence on pose quality, scene observability, and viewpoint-conditioned sensing.