---
title: 'DéjàView: Looping Transformers & Beyond'
url: https://www.emergentmind.com/topics/dejaview
type: topic
---

# DéjàView: Looping Transformers & Beyond

“DéjàView” is not a single standardized research term. In arXiv usage, it most explicitly names a looping transformer architecture for multi-view 3D reconstruction, while closely related “Déjà Vu” formulations designate unintended memorization in self-supervised and vision-language representations, a VideoLM query engine based on inter-frame computation reuse, a grid-cell-inspired place-recognition method, and a view-driven deduplication paradigm in visual analytics [2605.30215][2304.13850][2504.05651][2506.14107][1707.06393][1606.05708]. Across these lines, the recurrent theme is the exploitation of structure already present in a “view,” whether that view is a set of camera images, a background crop, a video frame sequence, a repetitive landmark pattern, or a relational visualization.

## 1. Research usages of the term

The term is used across several technically distinct literatures.

| Usage | Domain | Core object |
|---|---|---|
| “Déjà View” | Multi-view 3D reconstruction | Looped transformer block with explicit iteration |
| “Déjà vu memorization” | SSL and VLM auditing | Foreground recovery from background/context beyond correlation |
| “Déjà Vu” query engine | Video analytics | ReuseViT with GPU compaction for VideoLMs |
| “Deja vu” place recognition | Robotics | Mutually supportive feature frequencies |
| “DéjàView idea” | Visual analytics | View-driven deduplication under labeling budget |

The most direct nominal match is “Déjà View: Looping Transformers for Multi-View 3D Reconstruction,” which treats model depth as partially buying iteration and replaces a deep decoder stack with a single recurrently applied transformer block [2605.30215]. The memorization line, by contrast, uses “déjà vu” to denote the recovery of training-image-specific foreground information from background-only or context-only inputs, first for SSL encoders and then for pre-trained open-source image and vision-language models [2304.13850][2504.05651]. The video query-engine work uses the same name for a systems pipeline that accelerates ViT-based VideoLMs through learned inter-frame reuse and memory-compute joint compaction [2506.14107]. Earlier work in robotics used “Deja vu” for a place-recognition algorithm based on repetitive, mutually complementary landmark frequencies rather than unique landmarks [1707.06393]. A further related usage appears in view-driven deduplication, where the system prioritizes record cleaning according to impact on the current visualization rather than on the whole dataset [1606.05708].

## 2. DéjàView as a looping transformer for multi-view 3D reconstruction

In its most explicit contemporary usage, DéjàView is a multi-view 3D reconstruction model that takes a set of RGB images $\{I_i\}_{i=1}^V$ and predicts per-view depth maps $D_i \in \mathbb{R}^{H \times W}$, per-view ray maps $R_i \in \mathbb{R}^{H \times W \times 6}$, dense 3D pointmaps, and camera parameters in a common 3D frame [2605.30215]. The per-pixel 3D point is represented as
$$
\mathbf{X}(u,v) = R^o(u,v) + D(u,v)\, R^d(u,v),
$$
and the camera head predicts
$$
\mathbf{c}_\theta = (\mathbf{t}_\theta,\, \mathbf{q}_\theta,\, \mathbf{f}_\theta) \in \mathbb{R}^3 \times \mathbb{S}^3 \times \mathbb{R}^2.
$$
This formulation targets metric geometry directly rather than implicit fields or Gaussian splats.

The architectural claim is that contiguous transformer layers often behave like repeated applications of similar operations, and that multi-view reconstruction transformers refine their predictions progressively across decoder depth. DéjàView therefore applies a single looped transformer block recurrently for $K$ refinement steps:
$$
z_{k+1} = f_\theta(z_k, t_k, t_{k+1}),
$$
with the same block reused at every step and conditioned on a continuous time interval. During training, $K \sim \text{Beta}(2,1)$ scaled to $[K_{\min}, K_{\max}] = [8, 16]$, so the trained checkpoint exposes $K$ as an inference-time compute knob [2605.30215].

The significance of the shared-block design is not merely parameter efficiency. An ablation comparing a decoupled 16-block variant against a shared recurrent block reports that the decoupled model reaches Rel L2 $=0.056$, IR $=61.1$, and AUC@30 $=82.0$, whereas the full shared model with residual and state gating reaches Rel L2 $=0.040$, IR $=69.2$, and AUC@30 $=86.9$ [2605.30215]. The paper explicitly interprets this as evidence that explicit iteration is a stronger inductive bias for multi-view 3D reconstruction, not only a compute-efficient substitute for depth.

On efficiency and benchmarks, DéjàView is reported at 117M parameters, 75.9 TFLOPs total, 3.2 TFLOPs/image for 24 views, 4.9 GiB peak GPU memory, average IR $=80.3\%$, and average AUC@30° $=91.8\%$ [2605.30215]. It is evaluated on DTU, ETH3D, 7-Scenes, ScanNet++, and nuScenes, and is described as matching or outperforming substantially larger feed-forward baselines across these five benchmarks while using a fraction of their parameters and comparable or lower compute [2605.30215]. A common misconception is that weight tying is only compression; the shared-vs-decoupled ablation directly argues against that interpretation.

## 3. Déjà vu memorization in SSL and VLMs

A second major usage of the term concerns unintended memorization in representation learning. The original SSL work defines “déjà vu memorization” as the retention of image-specific information such that, given a trained model and a crop of a training image containing only the background, it is possible to infer the foreground object with high accuracy or even visually reconstruct it [2304.13850]. The formal setup writes a labeled image as
$$
z = (v, t), \quad v = \mathsf{crop}(x), \quad t = y,
$$
and defines a point as memorized if there exists a predictor $h$ such that
$$
h(f, v) = t \quad\text{but}\quad \arg\max_{t'} \mu(t' \mid v) \neq t.
$$
The distinction is therefore not mere background-label correlation, but performance beyond what can be explained by dataset-level correlations [2504.05651].

The original SSL study reports that, for VICReg with ResNet-101 and 300k training images, label inference from background-only crops reaches almost $95\%$ accuracy on the top $1\%$ most confident crops, and that for VICReg at $p=20\%$ confidence the déjà vu score grows from less than $10\%$ at 250 epochs to more than $30\%$ at 1000 epochs [2304.13850]. It further reports that the memorized set exceeds $15\%$ of training samples for VICReg trained 1000 epochs on 300k images, whereas a supervised model trained on the same data exhibits a much smaller déjà vu score of about $9\%$ [2304.13850]. The paper also states that this behavior cannot be detected by conventional techniques for evaluating representation quality, because linear-probe validation can improve while déjà vu memorization worsens [2304.13850].

The efficient-measurement extension removes the need to train two large shadow models. For image SSL, it proposes simple correlation estimators, including a ResNet-50 background classifier and a Naive Bayes classifier over detected objects, and defines
$$
\text{DV}(f) = \text{Acc}_f - \text{CorrAcc}.
$$
For individual samples it defines a memorization confidence as
$$
\text{MemConf}(x_i) = \text{Entropy}(\text{Correlation Classifier}) - \text{Entropy}_{\text{SSL}(\text{KNN})},
$$
so high-confidence SSL prediction paired with high-entropy correlation prediction indicates likely memorization [2504.05651]. Aggregate results are reported as very stable across measurement methods: for VICReg and Barlow Twins trained on 300k images, the two-model KNN setup gives DV $\approx 10$–$11\%$ and DV@20% $\approx 29$–$32\%$, while DINO is much lower at approximately $3$–$6\%$ [2504.05651]. For pre-trained open-source SSL models, the same paper reports lower aggregate memorization than similar architectures trained on a subset of the data; for example, VICReg OSS has DV $\approx 5.7\%$ versus $\approx 10.9\%$ on the subset, and DV@20% $\approx 20.2\%$ versus $\approx 32.4\%$ [2504.05651].

The same efficient framework extends “DéjàView-style” measurement to vision-language models by treating the caption as context $v$ and the detected objects in the paired image as attributes $t$ [2504.05651]. It uses population precision gap and population recall gap, and reports that a one-model test with a pre-trained text embedding model slightly overestimates memorization but is very close at population level: on Shutterstock models, the two-model test gives PPG $\approx 0.06$ and PRG $\approx 0.064$, while one-model tests give PPG $\approx 0.07$ and PRG $\approx 0.06$, or PPG $\approx 0.068$ and PRG $\approx 0.064$ depending on retrieval mode [2504.05651]. This directly addresses the misconception that “foreground from background” or “objects from caption” is automatically correlation; both papers insist on separating correlation from memorization.

## 4. Déjà Vu as a VideoLM query engine

In video analytics, “Déjà Vu” denotes a video-language query engine that accelerates ViT-based VideoLMs by reusing computations across consecutive frames [2506.14107]. The paper’s systems target is large-scale video query processing, where VideoLMs sample video at 2 FPS and visual embedding extraction dominates end-to-end FLOPs. Its core model, ReuseViT, is a modified ViT that learns token-level inter-frame reuse opportunities rather than relying on fixed hand-designed policies [2506.14107].

ReuseViT focuses on reusing QKV projections and FFNs rather than self-attention. The reuse decision for token $i$ is computed from a concatenation of four cues: similarity $s_i$, token importance $t_i$, reference type $r_i$, and codec metadata $c_i$,
$$
v_i = \text{concat}(s_i, t_i, r_i, c_i), \qquad d_i = \text{MLP}_{\text{decision}}(v_i),
$$
with a hard gate
$$
\mathcal{M}_i =
\begin{cases}
1 & \text{if } d_i > 0 \quad (\text{reuse})\\
0 & \text{otherwise}\quad (\text{recompute}).
\end{cases}
$$
A Restoration Layer then calibrates reused outputs via
$$
\Delta R_i = R^{\text{cur}}_i - R^{\text{ref}}_i, \qquad
\hat{R}^{\text{cur}}_i = \tilde{R}^{\text{ref}}_i + \text{MLP}_{\text{restoration}}(\Delta R_i),
$$
and training uses a Gumbel-Softmax relaxation plus a similarity-and-reuse objective [2506.14107].

A central systems result is that FLOP reduction does not directly translate into GPU speedup. Déjà Vu therefore introduces memory-compute joint compaction: layer-wise scheduling, cached memory compaction, and sparse computation compaction implemented as GPU gather-compute-scatter over active tokens [2506.14107]. In an ablation on video QA at 61% reuse, hard gating alone yields 1.25× speedup, adding sparse computation compaction yields 1.45×, and adding cached memory compaction yields 1.62× [2506.14107]. This is the paper’s explicit answer to the misconception that algorithmic sparsity automatically becomes wall-clock acceleration.

The evaluation covers three VideoLM tasks: video retrieval with CLIP4Clip on MSR-VTT, video QA with FrozenBiLM on How2QA, and video question grounding with TempCLIP on NExT-GQA [2506.14107]. Reported embedding-generation speedups reach up to 1.81× for retrieval, 2.64× for video QA, and 2.54× for grounding, all within roughly 2% accuracy degradation; the abstract summarizes this as acceleration by up to 2.64× within a 2% error bound [2506.14107]. The same work notes that embedding storage is small relative to video size and that caching and reuse make VideoLMs more practical for large-scale video analytics [2506.14107].

## 5. Deja vu as scalable place recognition

An earlier robotics usage appears in “Deja vu: Scalable Place Recognition Using Mutually Supportive Feature Frequencies,” which proposes a place-recognition algorithm inspired by mammalian place, head-direction, and grid cells [1707.06393]. The motivating observation is that most place-recognition systems attempt to extract salient features, landmarks, or signatures, with storage and computation growing as the environment increases, whereas biological spatial encoding uses a fixed-size neural encoding framework and, paradoxically, grid cells employ a highly aliased repetitive encoding in which one neuron represents an unbounded number of places in the world [1707.06393].

The algorithmic inversion is to seek out and leverage repetitive, mutually complementary landmark frequencies rather than suppress repetition [1707.06393]. The paper states that the combinatorial encoding capacity of multiple different frequencies enables efficient data storage and the potential for sub-linear storage growth in a learning and recall system. It reports experiments on both ground-based and aerial camera datasets, and discusses how the method might scale to arbitrarily large global datasets and dimensions [1707.06393].

This work is important partly because it directly reverses a standard assumption in place recognition. In conventional formulations, aliasing is a failure mode; here repetitive structure becomes the representational substrate. A plausible implication is that the “déjà vu” label is used deliberately to denote repeated structure that is informative rather than confounding.

## 6. View-driven deduplication and broader conceptual links

A related, though not identically named, “DéjàView idea” appears in view-driven deduplication for visual analytics [1606.05708]. The problem is to produce the cleanest view possible with a limited budget for data labeling, rather than to clean an entire dataset. The formal objective is to minimize
$$
\mathrm{Distance}(V_{\text{curr}}, V_{\text{clean}})
$$
subject to a small labeling budget, where $V(R)$ is the current relational view, $V_{\text{clean}}$ is the ideal deduplicated view, and the view impact score of a tuple is
$$
\mathrm{Impact}(V,t) = \mathrm{Distance}\big(V(R), V(R - t)\big).
$$
The paper uses Earth Mover’s Distance as the view distance and combines view impact with classifier uncertainty through
$$
\mathrm{HybridScore} = \alpha \cdot \mathrm{ViewImpactScore} + (1-\alpha) \cdot \mathrm{ClassifierUncertainty}.
$$
Candidate pairs are restricted via view provenance and feature-based blocking, and cleaning stops when consecutive view changes stabilize [1606.05708].

Empirically, experiments on nine different visualizations for two real-world datasets show that the approach produces significantly cleaner views for small labeling budgets than state-of-the-art alternatives and also stops the cleaning process after requesting fewer labels [1606.05708]. On the harder products dataset, all product views are cleaned within fewer than 18 batches, whereas uncertainty-based baselines fail to completely clean any product view within 40 or more batches; on the restaurants dataset, all views are fully cleaned in three batches [1606.05708].

This usage is not a nominal “DéjàView” system title in the same way as the 2026 reconstruction model, but it is presented as “exactly the ‘DéjàView’ idea” in the sense that cleaning is guided by what is on the screen rather than by global data-quality objectives [1606.05708]. This suggests a broader semantic pattern across the literature: “DéjàView” and “Déjà Vu” repeatedly name methods that exploit partial, repeated, or context-only observations to recover structure that would otherwise require full recomputation, full visibility, or full-data processing.

Source: https://www.emergentmind.com/topics/dejaview