Solv3R: 3D Looped Transformer as a Multi-view Solver

Abstract: Classical structure-from-motion reconstruction proceeds iteratively, refining camera poses and scene geometry from a current estimate. This allows maps to easily expand without recomputing already computed views, or geometry estimates to improve as more observations are provided. Most geometric foundation models, however, predict entire geometry in a single forward pass or update a recurrent state as frames arrive sequentially, making this expansion and refinement non-trivial. We introduce Solv3R, a looped transformer that learns to continue solving multi-view geometry from its current state. Importantly, its update operator is independent of the iteration count, allowing previously refined and newly introduced views to be jointly updated regardless of how ``solved'' they are. We train the operator to preserve accurate reconstructions under continued iteration and correct inaccurate states, enabling refinement of fixed image collections and integration of new observations. Representations of solved views can also be cached once and reused as geometric context across iterations, enabling compact and efficient key-value caching. We analyze the solver dynamics and compare it with the state-of-the-art, demonstrating that a single learned solver supports offline reconstruction, incremental reconstruction, relocalization, and streaming without task-specific training or modules. The code and models will be made publicly available at https://ubc-vision.github.io/solv3r/.

• The paper introduces Solv3R, a multi-view reconstruction solver deploying a stationary geometric update operator, which independently addresses views regardless of completeness or solver call index and outperforming tradition GFMs in stability, correction, and continuing geometry reconstruction.
• Empirical evaluations showcasing Solv3R’s success across scenarios, maintaining sub-5 degrees rotational accuracy and achieving up to 99.3% reliability in stopping criteria with over 10X speed improvement.
• Results demonstrating Solv3R'S robust and adaptable correction abilities, resolving accurate reconstruction irrespective of number of calls, presence of uncertainty, and incoming views

Problem formulation and central claim

“Solv3R: 3D Looped Transformer as a Multi-view Solver” (2611.02857) reframes learned multi-view reconstruction as an iterative geometric solving problem rather than a fixed-depth feed-forward prediction task. The distinction is substantive. Conventional geometric foundation models (GFMs) produce cameras, depths, and point or ray maps through a sequence of layer-specific transformations. Their latent states are therefore coupled to network depth: a representation produced after one computational schedule is not necessarily valid input to an arbitrary continuation of that schedule. Streaming models maintain persistent state, but typically constrain observations to sequential arrival and do not provide a general mechanism for revising already reconstructed views when new evidence appears.

Solv3R instead seeks a learned analogue of a stationary numerical solver. Its update operator is independent of the solver-call index and of the number of calls previously received by each view. The intended consequences are three solver properties:

1. Stability: an accurate reconstruction should be approximately a fixed point under further calls.
2. Correction: perturbed or partially reset states should converge back toward a geometrically consistent solution.
3. Continuation: newly introduced views should be inserted into an existing reconstruction and jointly refined with previously solved views.

The paper’s principal claim is consequently stronger than parameter sharing. A looped transformer is not a reusable solver merely because its weights are shared; the update must also be timestep-invariant and trained to operate on heterogeneous computational histories.

Figure 1: Solv3R uses a stationary geometric update operator, unlike layer-specific feed-forward GFMs and step-conditioned recurrent reconstruction models.

Stationary solver architecture

Solv3R starts from the shared transformer architecture of D, retaining frame-wise and global attention while removing the explicit refinement-step conditioning. Let $\mathbf{z}^0$ denote the image-derived initial state and $\mathbf{z}^k$ the current multi-view state. Solv3R applies

$$\mathbf{z}^{k+1}=\mathcal{T}_{\theta}(\mathbf{z}^{k};\mathbf{z}^{0}),$$

where the same operator $\mathcal{T}_{\theta}$ is used for every solver call. The initial state is injected at every call, so the current state represents the reconstruction estimate while $\mathbf{z}^0$ continually supplies the image-conditioned constraints. This prevents errors in the latent state from becoming the sole source of information during long rollouts.

Two architectural changes enforce stationarity. First, D’s step-dependent channel gates are replaced by learned constant gates initialized from the average D gates over its 16-step schedule. Second, the updated state is globally rescaled to a fixed RMS norm. The normalization acts on the entire state rather than independently on views or tokens, preserving relative feature magnitudes while preventing long-horizon norm drift. The resulting model contains 117M parameters and operates on images whose longer side is 504 pixels.

Figure 2: A stationary transformer block repeatedly updates the multi-view state, while the same operator supports continued refinement, view insertion, and decoding after any solver call.

The training procedure is designed around state distributions rather than a single prescribed rollout length. Each example receives 16–40 nondifferentiable pre-calls, followed by eight differentiable calls with supervision at the final call. With probability 0.15, the state is either corrupted with RMS-scaled Gaussian noise or partially reset by replacing a subset of views with their initial encodings. The latter explicitly simulates the mixed-progress state produced when new views enter an existing reconstruction. Supervision covers camera poses, ray maps, and depth, supplemented by distillation from D at 16 calls. The image encoder and prediction heads are frozen; only the shared block, gates, and initial-state projection are fine-tuned.

This construction has an important methodological implication: the solver behavior is not obtained by extrapolating a model trained for 16 recurrent steps. It is induced by exposing the shared operator to states at different refinement stages, corrupted states, and combinations of solved and unsolved views.

Stability, correction, and mixed-progress behavior

The solver analysis provides the clearest evidence that Solv3R differs qualitatively from D. Solv3R maintains camera accuracy while its relative state update decreases, whereas D degrades when repeatedly applying its block beyond the schedule for which it was trained. Using a stopping threshold of $\gamma=0.02$ on the relative state update, 144 of 148 evaluated windows reached the threshold within 64 calls. Of those 144 windows, 143, or 99.3%, had camera rotation error below $5^\circ$ at stopping. The stopping calls ranged from 13 to 32, with a mean of 17.5. Median rotation error was $0.44^\circ$ at the stopping call and $0.43^\circ$ after 64 calls.

Thus, the relative latent-state update is not merely a computational heuristic: in these experiments it identifies a regime in which additional computation produces negligible geometric improvement. The criterion also adapts computation to difficulty, since harder windows tend to require more calls. However, the stopping rule is empirical rather than derived from a contraction guarantee; the paper does not establish global convergence or certify that low state movement excludes a suboptimal fixed point.

Solv3R also recovers from perturbations introduced after eight calls. Gaussian perturbations and complete resets of non-reference views are followed by eight further calls. Solv3R returns to accurate camera estimates for every tested perturbation, while D fails to recover reliably. The result supports the correction property and directly motivates the use of the same operator for newly inserted views.

Figure 3: Solv3R preserves accurate geometry under continued calls and corrects perturbed or reset states, while D loses accuracy outside its trained schedule.

The mixed-progress experiments test whether different views can be updated despite having received different numbers of calls. Views are initialized from randomly selected points along independent reconstruction trajectories. After joint refinement, Solv3R produces nearly identical camera rotations across trials, with a mean difference of only $0.08^\circ$. In a pipelined configuration, views enter and leave a sliding window after every call, so active views have received between zero and fifteen previous calls. Solv3R remains accurate under this asynchronous schedule.

The practical consequence is substantial. Once an initial 16-call latency has been incurred, the pipeline can process frames with a one-call delay between successive estimates rather than re-solving an entire window for every incoming frame. The paper reports a nominal $16\times$ throughput increase relative to D in the corresponding setup and approximately $10\times$ end-to-end throughput improvement.

Figure 4: Solv3R jointly solves views with heterogeneous refinement histories, enabling pipelined reconstruction without synchronizing computational progress.

Revisable incremental reconstruction

The continuation property is evaluated by solving four views for eight calls, inserting twelve additional views, and continuing from the existing state. Solv3R jointly refines the original and newly introduced views; D fails to preserve a consistent reconstruction. Importantly, the original views are not merely kept fixed. Their camera estimates often improve after the new observations are added, demonstrating revision rather than state preservation alone.

The same behavior holds when new views are introduced at different refinement stages, including after the initial reconstruction has met the stopping criterion. When the view set expands from 8 to 64 images, continuation reaches accuracy comparable to repeated reconstruction from scratch with lower runtime. A more controlled comparison reports an AUC@10 of 92.2 after 64 views using eight calls per insertion from the current state, whereas re-solving from scratch with eight calls reaches 84.8 and requires sixteen calls to reach 91.8.

These results establish that cached geometric computation has algorithmic value. The current state is not simply a memory of prior predictions; it is a usable initialization for the enlarged constraint system. Nevertheless, the benefit is not entirely order-invariant. Across capture, reverse, and random insertion orders, continuation remains within two AUC points of the corresponding re-solving controls, but reverse-order insertion is somewhat weaker on DL3DV. This indicates that continuation is robust but not strictly path-independent.

Figure 5: After twelve views are added to a four-view reconstruction, Solv3R improves both old and new camera estimates, whereas D cannot maintain a consistent joint solution.

Reusable geometric context and relocalization

Stationarity also enables a more compact form of attention caching. In a conventional stacked GFM, key–value representations are layer-specific. In D, the shared block is reused but the step-conditioned transformations make a solved map’s cache dependent on the refinement stage. Solv3R can reuse one KV cache for a solved map across all query solver calls, reducing cache storage by $16\times$ relative to stage-specific caching in D.

The relocalization experiments freeze reconstructed map views and refine query images against retrieved map context. All methods use the same retrieved views. Solv3R uses 2.86 MB per map view and approximately 51 ms per query, while obtaining 72.8% recall within 5 cm and $5^\circ$ on 7-Scenes and 24.9% recall within 25 cm and $2^\circ$ on Cambridge. With the same small cache size, D’s single-cache configuration fails to localize, while D’s all-step cache uses 45.8 MB per view and achieves only 28.3% recall on 7-Scenes and 0.7% on Cambridge.

The implication is that Solv3R’s state representation remains geometrically interpretable across solver calls, not merely numerically stable. This is an unusually strong requirement for recurrent latent representations and is supported by the attention analysis: correspondence-sensitive heads retain more than 96% similarity to their call-16 attention maps at call 64, whereas D’s correspondence accuracy falls below 3% after its schedule ends.

Figure 6: Solv3R relocalizes queries against a frozen map using a single compact KV cache, while D requires stage-specific context to remain effective.

Offline reconstruction accuracy and efficiency

Solv3R is not presented as uniformly superior on fixed-budget offline reconstruction. Its principal contribution is solver reuse, but it also achieves competitive standard benchmark performance with a substantially smaller model. The 117M-parameter model is evaluated on DTU, ETH3D, 7-Scenes, ScanNet++, and nuScenes.

With 16 fixed solver calls, Solv3R obtains pose AUC@3/AUC@30 values of 84.5/99.0 on DTU, 65.4/94.8 on ETH3D, 13.8/81.8 on 7-Scenes, 80.7/98.2 on ScanNet++, and 41.5/85.3 on nuScenes. Adaptive stopping improves these to 85.5/99.1, 70.4/96.3, 14.2/81.9, 81.2/98.2, and 41.5/85.3, respectively, with only three to four additional calls on average. Its adaptive TFLOPs for 24 views are 91.3, compared with 190.0 for VGGT, 153.8 for $\pi^3$, and 178.7 for DA3-G.

The comparison is therefore nuanced. Larger feed-forward GFMs can achieve stronger results on particular datasets or metrics, while Solv3R provides comparable accuracy at 117M parameters and adds continuation and revision capabilities absent from fixed-depth models. The paper’s claim is not that stationarity eliminates the accuracy–compute trade-off, but that it makes additional computation and previously computed states reusable.

Streaming reconstruction

Solv3R applies the same model to streaming without streaming-specific training or a separate optimization backend. Eight anchor frames establish the coordinate frame; recent frames are actively refined; selected older frames are retained as memory and used as frozen KV context. Every four incoming frames, the model tests whether frozen states would change under another call. If the relative update exceeds $\gamma=0.02$, those frames re-enter joint refinement and their caches are rebuilt.

In the default L8 configuration, Solv3R processes a frame in 95 ms using 5.0 GB of GPU memory. Its trajectory ATE is 0.157 m on Sintel, 0.015 m on TUM dynamics, and 0.077 m on ScanNet. It obtains video-depth AbsRel values of 0.322 on Sintel, 0.064 on Bonn, and 0.078 on KITTI, together with Chamfer distances of 0.018 m on both 7-Scenes and NRGBD. These results are competitive with specialized streaming systems despite the absence of streaming-specific training.

The memory design is bounded by the fixed anchor, recent-frame, and memory-frame budgets. In a 2,000-frame TUM sequence, Solv3R’s memory saturates after the configured memory bank is filled. By contrast, offline VGGT exceeds an 80 GB H100 budget at approximately 320 frames and StreamVGGT at approximately 365 frames. The smaller S4 configuration reduces runtime to 51 ms and memory to 2.1 GB, although with degraded ScanNet trajectory accuracy.

The ablations clarify that state reuse and cache refresh are both necessary. In long sequences, removing reuse causes Solv3R’s ATE to rise from 0.077 m to 0.574 m on ScanNet, while disabling refresh increases it to 1.05 m. Hence, bounded memory alone is insufficient: the system must periodically reactivate previously frozen states when new evidence indicates inconsistency.

Figure 7: Solv3R occupies a favorable streaming accuracy–runtime–memory regime, with additional configurations exposing the trade-off between solver calls and resource use.

Ablations and learned solver dynamics

The ablations distinguish fixed-budget accuracy from actual solver behavior. Step-conditioned gates perform reasonably at their intended horizon but degrade when the planned horizon changes. With a schedule planned for 16 calls, they obtain pose accuracies of 71.5, 52.1, and 23.5 at 16, 128, and 1024 calls in the reported aggregate evaluation; with a 1024-call schedule, the corresponding values are 61.9, 43.3, and 52.2. This supports the paper’s central criticism that weight sharing with timestep conditioning does not produce a reusable stationary operator.

Input injection, state normalization, and random-depth pre-calls are individually important for long-horizon stability. Removing random-depth pre-calls is particularly damaging, reducing the 128-call score to 3.4 in the reported ablation. Removing perturbation training preserves much of the long-horizon stability but substantially degrades join and cached-context performance. This separates two properties that are often conflated: stability does not imply correction.

After the full training procedure, Solv3R reaches aggregate pose accuracies of 77.2, 79.1, and 78.8 at 16, 128, and 1024 calls, with perturbation, join, and cache errors of 0.22°, 0.54°, and 0.72°, respectively. These values indicate that the desired dynamics require explicit training pressure, not only architectural stationarity.

Limitations and open questions

The paper reports a concrete failure mode. When two view groups share little visual overlap, Solv3R can place the second group at an incorrect position and orientation by 16 calls and retain that erroneous configuration through 64 calls. Of the 144 windows that reached the stopping threshold in the main stability analysis, one stopped with rotation error above $5^\circ$. The low update norm therefore identifies a stable state, not necessarily the globally correct state.

The method also depends on several operational assumptions. It uses an external NetVLAD retrieval system to select map views for relocalization, and streaming performance depends on fixed window, anchor, memory, and refresh policies. The reported geometric metrics frequently apply per-sequence or per-view scale alignment, so they do not fully characterize absolute metric consistency. In addition, the training mixture includes substantial synthetic and curated data, while the experiments do not establish behavior under severe appearance change, very weak overlap, dynamic scene topology, or arbitrarily long-term accumulation beyond the bounded memory design.

A specific unresolved question is whether a learned stationary operator can be equipped with reliable failure detection or uncertainty estimates that distinguish convergence to a correct fixed point from convergence to the type of suboptimal fixed point shown in the failure case. A second is whether the same continuation mechanism remains effective when the scene itself changes, rather than merely when additional observations reveal previously static geometry.

Conclusion

Solv3R formulates learned multi-view reconstruction as an ongoing geometric solve. Its contribution is the combination of a timestep-invariant transformer update, initial-state injection, state normalization, and training on heterogeneous, perturbed computational states. This design yields stable long-horizon refinement, corrective behavior, mixed-progress solving, reusable KV context, and revision after view insertion.

The empirical results support the distinction between a shared recurrent block and a genuine learned solver. Solv3R matches competitive offline reconstruction accuracy with 117M parameters, achieves 99.3% sub-$5^\circ$ stopping reliability on the evaluated converged windows, reduces map-cache storage by $16\times$ relative to stage-specific caching, and provides competitive bounded-memory streaming reconstruction without task-specific streaming training. Its principal limitation is equally clear: stationary dynamics can preserve an incorrect reconstruction when visual overlap is insufficient.