Lookalike3D: 3D Indoor Object Similarity

• Lookalike3D is a 3D indoor scene task that distinguishes repeated and near-repeated objects by classifying pairs as identical, similar, or different based on multiview RGB crops.
• The method leverages a frozen DINOv2 backbone and a hierarchical transformer encoder to fuse intermediate and final features from multiple views, yielding significant IoU improvements over baselines.
• The 3DTwins dataset, with over 76k annotated object pairs across 1,015 classes, provides a comprehensive benchmark for fine-grained lookalike detection in indoor environments.

Searching arXiv for the primary and contextual papers to ground the article in current records. Lookalike3D denotes a 3D indoor-scene perception task and a corresponding multiview image transformer for detecting repeated and near-repeated object instances within a scene. The task is defined as classifying pairs of objects from the same semantic class as identical, similar, or different from multiview RGB crops, with the goal of exploiting repeated structure as a cue for scene-level 3D understanding rather than treating each instance independently (Yeshwanth et al., 25 Mar 2026). In this formulation, repeated and near-repeated objects are not merely semantic duplicates; they are a source of complementary geometric and appearance evidence that can support more consistent reconstruction and part reasoning. The conceptual basis also aligns with earlier work showing that resemblance is a ranking problem distinct from identity recognition, although that earlier evidence was developed for human faces rather than 3D objects (Sadovnik et al., 2018).

1. Definition of the task and label space

Lookalike3D is introduced as lookalike object detection in indoor scenes. Given an input scene, the task is to classify pairs of objects as identical, similar or different using multiview images as input (Yeshwanth et al., 25 Mar 2026). The three labels are defined as follows: identical corresponds to the same shape and appearance; similar corresponds to near-identical objects with slight differences in shape, articulation, deformation, or texture; and different corresponds to objects that are significantly different in shape or appearance (Yeshwanth et al., 25 Mar 2026).

The task is defined over object instances $o_i = (m_i, \{I_{i,j}\})$, where $m_i$ is the object mask and $I_{i,j}$ are cropped object images from multiple views (Yeshwanth et al., 25 Mar 2026). During training, RGB-D trajectories are used to associate crops across views, whereas inference can operate from a single RGB image as well (Yeshwanth et al., 25 Mar 2026). Pairs are formed within semantic classes and within the same scene, and structural elements and ambiguous tiny objects are filtered out (Yeshwanth et al., 25 Mar 2026).

The distinction between identical and similar is central rather than incidental. Exact duplicates are especially useful for joint reconstruction, because partial observations from multiple copies can be fused into one canonical object; similar-but-not-identical instances are useful for semantic transfer, including part decomposition transfer between objects with shared structure (Yeshwanth et al., 25 Mar 2026). This suggests that Lookalike3D is not only a classification problem but also a scene-structuring prior for downstream 3D pipelines.

A broader conceptual precedent comes from face similarity research. “Finding your Lookalike: Measuring Face Similarity Rather than Face Identity” argues that perceived resemblance is not equivalent to identity recognition, because once different identities are separated “far enough,” recognition embeddings need not preserve the fine-grained ranking among non-matching but visually similar instances (Sadovnik et al., 2018). A plausible implication is that Lookalike3D applies the same distinction to indoor objects: instance verification or coarse semantic recognition does not automatically solve near-duplicate ranking.

2. Architectural design of the Lookalike3D model

The Lookalike3D method is a multiview image transformer that relies on a frozen image foundation model prior, specifically DINOv2-small (Yeshwanth et al., 25 Mar 2026). Each object crop is passed independently through a shared frozen DINOv2 backbone to obtain patch tokens. The model uses not only the last-layer features but also intermediate features from layers $(1,3,5,8)$; the supplementary specifies that the intermediate features from layers $[1,3,5,8]$ and the final layer are concatenated and projected back to feature dimension 384 before entering the attention module (Yeshwanth et al., 25 Mar 2026). The stated rationale is that late DINOv2 features provide strong semantic and object-part priors, while intermediate features preserve finer structural cues (Yeshwanth et al., 25 Mar 2026).

For each candidate pair $(o_1,o_2)$, the model uses up to $k$ views per object, with $k=5$ in the main implementation (Yeshwanth et al., 25 Mar 2026). The selected crops are the top five by coverage. After DINO feature extraction, the patch tokens are refined by a transformer encoder with three hierarchical self-attention types:

$$\text{Single-view:} \quad p^{l}_{i,j} = \Phi_{\mathrm{frame},l}(p^{l-1}_{i,j})$$

$$\text{Multiview:} \quad p^{l}_{i,j} = \Phi_{\mathrm{object},l}(\{p^{l}_{i,j}\}), \quad \forall j \in \{1, \dots, k\}$$

$m_i$0

The transformer uses learned frame embeddings and object embeddings so that tokens from different views and different objects remain distinguishable (Yeshwanth et al., 25 Mar 2026). The supplementary states that the transformer has internal embedding dimension 256, 8 attention heads, and cosine positional embeddings (Yeshwanth et al., 25 Mar 2026). The final model uses one alternating attention block containing one layer each of single-view, object-level, and global attention (Yeshwanth et al., 25 Mar 2026).

After the final transformer layer, patch features are aggregated across views for each object by averaging, producing one sequence of patch features per object, denoted $m_i$1 (Yeshwanth et al., 25 Mar 2026). These features are then flattened, normalized, and compared with a dot product to obtain a scalar similarity $m_i$2 (Yeshwanth et al., 25 Mar 2026). The score is converted into a three-way label using fixed thresholds:

$m_i$3

with $m_i$4 (Yeshwanth et al., 25 Mar 2026).

The model architecture is explicitly image-based rather than native-geometry-based: it consumes object crops derived from the scene and reasons over them through multiview token fusion. This distinguishes it from cross-modal 2D–3D retrieval systems such as “Zero in on Shape: A Generic 2D-3D Instance Similarity Metric learned from Synthetic Data,” which compares RGB images and untextured 3D models using a shared embedding and cosine distance for zero-shot shape retrieval (Janik et al., 2021). The latter is instance-level metric learning across modalities, whereas Lookalike3D is intra-scene pair classification among object instances already localized in images.

3. Training objective and optimization strategy

Lookalike3D is trained with two losses: a multi-class triplet loss and a score alignment loss (Yeshwanth et al., 25 Mar 2026). The triplet formulation enforces the ordering

$m_i$5

in similarity space. Feature distance is defined as

$m_i$6

The paper specifies three margin-based triplet terms, corresponding to the constraints that identical should be closer than similar by margin $m_i$7, similar should be closer than different by margin $m_i$8, and identical should be closer than different by margin $m_i$9 (Yeshwanth et al., 25 Mar 2026). The intended form is

$I_{i,j}$0

The margins are set to

$I_{i,j}$1

(Yeshwanth et al., 25 Mar 2026). The supplementary notes that not every identical pair naturally has a corresponding similar or different pair sharing an anchor, so training is augmented by randomly sampling additional different pairs across semantic classes for anchors (Yeshwanth et al., 25 Mar 2026).

Triplet loss is supplemented by a score alignment loss intended to calibrate the absolute score scale to the fixed thresholds $I_{i,j}$2. The paper states that identical scores are pushed into the top bin and different scores into the bottom bin, while similar pairs are expected to occupy the middle bin indirectly (Yeshwanth et al., 25 Mar 2026). The full objective is

$I_{i,j}$3

The implementation details reported are a frozen DINOv2-small backbone, top 5 views per object, image size 224 \times 224, Adam with learning rate 1e-4, and batch size up to 128 pairs (Yeshwanth et al., 25 Mar 2026). Training takes about 4 hours on an NVIDIA A6000, and inference runs at 34 ms/sample with batch size 256 (Yeshwanth et al., 25 Mar 2026). The augmentations are random rotation, horizontal flip, color jitter, and crop (Yeshwanth et al., 25 Mar 2026).

The use of triplet supervision rather than instance classification follows a broader pattern in similarity learning. In the face-similarity setting, the Lookalike network of (Sadovnik et al., 2018) likewise fine-tunes a recognition embedding with triplet ranking supervision defined by human judgments of resemblance rather than shared identity. A plausible implication is that Lookalike3D adopts an analogous metric-ordering view, but generalized from subjective face similarity to object-pair structure inside real indoor scenes.

4. The 3DTwins dataset

A major contribution associated with Lookalike3D is 3DTwins, a dataset built from ScanNet++ specifically for lookalike object detection in 3D indoor scenes (Yeshwanth et al., 25 Mar 2026). The authors start from ScanNet++ 3D instance annotations and rasterize them onto undistorted DSLR images to obtain high-quality object crops (Yeshwanth et al., 25 Mar 2026). Visibility of object $I_{i,j}$4 in image $I_{i,j}$5 is computed as the fraction of the object’s vertices visible in that image, and top-coverage views are selected (Yeshwanth et al., 25 Mar 2026).

All intra-class object pairs within the same scene are generated, with exclusions for structural elements such as walls and floors, very small items such as pens and files, and objects lacking distinct 3D segmentation such as pipes (Yeshwanth et al., 25 Mar 2026). The remaining pairs are manually annotated as identical, similar, or different (Yeshwanth et al., 25 Mar 2026).

Split	Total pairs	Unique instances
Train	71,058	22,285
Validation	5,681	1,563
Full dataset	76,739	23,848

The full dataset statistics are 76,739 total pairs, comprising 34,766 identical, 6,203 similar, and 35,770 different pairs; it contains 23,848 unique object instances across 1,015 object classes (Yeshwanth et al., 25 Mar 2026). The training split contains 32,270 identical, 5,468 similar, and 33,320 different pairs, for 71,058 total pairs, 22,285 unique instances, and 867 classes (Yeshwanth et al., 25 Mar 2026). The validation split contains 2,496 identical, 735 similar, and 2,450 different pairs, for 5,681 total pairs, 1,563 unique instances, and 148 classes (Yeshwanth et al., 25 Mar 2026). The supplementary states that the dataset is based on 856 training scenes and 50 validation scenes from ScanNet++ (Yeshwanth et al., 25 Mar 2026).

The similar category is refined by similarity types: shape, articulated, mirrored, deformable, and texture/color (Yeshwanth et al., 25 Mar 2026). Similar pairs include articulated and deformed states with at least 90% shape overlap (Yeshwanth et al., 25 Mar 2026). Ambiguous or low-quality cases can be marked as unknown, and annotators may inspect multiple image views, full scene context, and the object’s 3D mesh (Yeshwanth et al., 25 Mar 2026). Annotation guidelines were drafted and refined on 50 randomly selected scenes before full annotation (Yeshwanth et al., 25 Mar 2026).

The class distribution is explicitly imbalanced, especially for the similar category, which is one reason the evaluation emphasizes IoU per class rather than raw accuracy (Yeshwanth et al., 25 Mar 2026). This suggests that 3DTwins is not simply a large corpus of repeated objects, but a deliberately structured benchmark for fine-grained intra-class differentiation.

5. Evaluation results and ablation evidence

Lookalike3D is evaluated on the 3DTwins validation set in two settings: GT instances, where object instances are ground-truth segmentations, and predicted instances, where instances come from MaskClustering (Yeshwanth et al., 25 Mar 2026). For predicted instances, the authors generate class-agnostic instance predictions with MaskClustering using Cropformer masks, associate predicted instances with ground truth using minimum IoU threshold 0.5, rasterize predicted masks into DSLR images to create crops, and count missing GT pairs as unknown (Yeshwanth et al., 25 Mar 2026).

The main metric is IoU for each pair category—Identical, Similar, Different—and an overall IoU (Yeshwanth et al., 25 Mar 2026). The reported GT-instance results are:

Method	Id IoU	Sim IoU	Diff IoU	Overall IoU
Lepard	0.06	0.13	0.47	0.22
Predator	0.52	0.06	0.08	0.22
DINOv2-feats	0.02	0.14	0.37	0.18
SuperGlue	0.03	0.07	0.44	0.18
ASpanformer	0.06	0.06	0.44	0.18
Mast3r	0.17	0.08	0.47	0.24
TLFR	0.29	0.12	0.33	0.24
Lookalike3D	0.65	0.18	0.65	0.49

For predicted instances, the results are:

Method	Id IoU	Sim IoU	Diff IoU	Overall IoU
Lepard	0.05	0.10	0.24	0.13
Predator	0.42	0.06	0.03	0.17
DINOv2-feats	0.03	0.15	0.37	0.18
SuperGlue	0.11	0.02	0.09	0.07
ASpanformer	0.10	0.00	0.10	0.07
Mast3r	0.26	0.02	0.02	0.10
TLFR	0.16	0.08	0.05	0.09
Lookalike3D	0.50	0.15	0.29	0.31

The paper states that Lookalike3D improves over the best baseline by 0.25 overall IoU in the GT setting and 0.13 overall IoU in the predicted-instance setting (Yeshwanth et al., 25 Mar 2026). The abstract’s claim of “104% IoU improvement over baselines” refers to relative improvement in overall IoU in the GT setting, from 0.24 to 0.49 (Yeshwanth et al., 25 Mar 2026).

The similar category remains the most difficult across all methods; Lookalike3D reaches 0.18 IoU on GT instances, which is still low in absolute terms but the best among the compared methods (Yeshwanth et al., 25 Mar 2026). The paper attributes baseline failures differently: DINOv2 raw features have strong semantics but weak view alignment under pose changes; 2D matchers and Lepard are overly dependent on contextual overlap; Predator over-predicts matches from coarse semantics; and most baselines do not truly fuse multiview information (Yeshwanth et al., 25 Mar 2026).

The ablation study isolates the main design choices. Replacing triplet loss with InfoNCE lowers overall IoU from 0.49 to 0.35; removing alignment loss lowers it to 0.47; removing DINO aggregated or intermediate features lowers it to 0.43 (Yeshwanth et al., 25 Mar 2026). Restricting the attention hierarchy also degrades performance: only single-view attention yields 0.43, single-view plus object-level 0.44, single-view plus global 0.45, and the full model 0.49 (Yeshwanth et al., 25 Mar 2026). Reducing input views from 5 to 1 lowers overall IoU from 0.49 to 0.43, while 3 views reaches 0.48 (Yeshwanth et al., 25 Mar 2026). These results support the interpretation that multiview evidence, intermediate structure-sensitive DINO features, and ordered triplet supervision are all material to the method’s performance.

6. Downstream applications, significance, and limitations

Lookalike3D is presented not merely as an end task but as a scene prior for downstream 3D systems (Yeshwanth et al., 25 Mar 2026). For identical objects, the method is used to improve SAM 3D Objects, a pretrained single-object 3D reconstruction model. The adaptation jointly conditions the geometry model on multiple masks from identical objects appearing in the same image: per-mask intermediate features are injected into every transformer layer, averaged after each layer, propagated forward, and used to predict a single joint $I_{i,j}$6 occupancy feature volume, which is then upsampled to a single $I_{i,j}$7 occupancy representation (Yeshwanth et al., 25 Mar 2026). The texture and refinement stage repeats the same averaging logic in the SLAT-based model, producing one canonical mesh that is placed back at each instance location using each object’s individual rotation and translation and the median of the individual scales (Yeshwanth et al., 25 Mar 2026). The reported gains are qualitative rather than numerical (Yeshwanth et al., 25 Mar 2026).

For similar pairs, the method improves part decomposition with P3-SAM (Yeshwanth et al., 25 Mar 2026). The described pipeline reconstructs both similar objects independently with SAM 3D, aligns them with point-to-point ICP, optionally stabilizes alignment with binned vertical-axis rotations, transfers part labels from the better-segmented source to the under-segmented target, and assigns target part IDs by majority vote over the $I_{i,j}$8-nearest neighbors from the source (Yeshwanth et al., 25 Mar 2026). This, too, is presented primarily qualitatively (Yeshwanth et al., 25 Mar 2026).

In relation to prior work, Lookalike3D is positioned against 2D re-identification, 3D re-identification or re-localization, 2D image matching, and 3D point cloud registration (Yeshwanth et al., 25 Mar 2026). Its novelty is threefold: the task of lookalike object detection in indoor 3D scenes with explicit identical / similar / different distinctions; the model, a multiview image transformer with frozen DINOv2 priors and alternating attention; and the 3DTwins dataset with 76k manually annotated object pairs over 1,015 classes (Yeshwanth et al., 25 Mar 2026).

The paper’s stated limitations are that performance degrades on small or far-away objects because DINO patch resolution is coarse, performance depends on camera coverage across views, subtle fine-grained differences may be missed with limited visibility, and the long-tail nature of indoor objects remains challenging (Yeshwanth et al., 25 Mar 2026). These constraints are consistent with its image-token design: the method leverages strong visual priors, but its representation is still bounded by crop quality, view availability, and the granularity of ViT patch features.

A wider implication emerges when Lookalike3D is read alongside neighboring research. The face-similarity literature shows that resemblance requires metric ordering beyond identity recognition (Sadovnik et al., 2018), while 2D–3D retrieval work shows that shared embeddings can be trained to compare photographs and 3D models by shape alone under zero-shot conditions (Janik et al., 2021). This suggests that Lookalike3D occupies a specific point in a broader research program: it operationalizes lookalikeness as a structured similarity relation inside real 3D scenes, using multiview image evidence and foundation-model priors to expose repeated and near-repeated objects as a usable source of geometric and semantic consistency (Yeshwanth et al., 25 Mar 2026).