SketchVCL Multi-View: 3D Sketch Understanding

• The paper demonstrates effective disentanglement of content and viewpoint to enable robust view-specific retrieval and 3D volumetric reconstruction using multi-view sketches.
• It employs dual architectures—a disentangled 2D encoder for cross-modal matching and a volumetric U-net for iterative 3D synthesis—yielding improved IoU scores and fine-grained pixel correspondence.
• The approach integrates synthetic multi-view data generation with advanced loss functions and iterative feature fusion, enhancing applications like interactive 3D modeling and precise sketch-based retrieval.

SketchVCL Multi-View refers to a class of neural architectures and training protocols for multi-view sketch understanding, retrieval, and 3D reconstruction, in which multiple sketches or projections from distinct viewpoints are leveraged for robust volumetric reasoning, fine-grained cross-modal retrieval, and pixel-level correspondence. The core research in this area advances disentanglement of content and viewpoint, synthetic multi-view data generation, iterative volumetric fusion, and multi-scale descriptor learning, enabling both user-controlled and data-driven synthesis and querying over sketch and photo domains.

1. Multi-View Sketch Supervision and Data Synthesis

Multi-view learning in sketch-based systems critically relies on the ability to generate supervisory data across varying viewpoints, circumventing the scarcity of such data in real freehand sketch datasets. Systems such as SketchVCL Multi-View utilize two principal forms of data (Sain et al., 2024):

• DCM (Drawing–Photo Pairs): Standard fine-grained datasets consisting of (sketch, photo) pairs, each sketch drawn from a single canonical viewpoint.
• D2D (3D Model Projections): Unpaired 3D objects from repositories (e.g., ShapeNet). For each model $\gamma_i$, synthetic 2D projections $p_{ij} = R(\gamma_i,v_j)$ are rendered at discrete yaw angles $v_j \in \{0^\circ, 15^\circ, \ldots, 345^\circ\}$ via orthographic projection, producing silhouette or line-drawing style images devoid of lighting or shading effects. The camera extrinsic (rotation $R_j$ and translation $t$) and a fixed intrinsic $K$ (identity for orthographic) ensure that each projection encodes both object geometry and viewpoint.

No explicit view-class labels are required; distinguishing view is a task left for the model given projection diversity.

2. Network Architectures: Disentanglement and Volumetric Prediction

SketchVCL Multi-View and its related approaches incorporate architectures that support either 2D cross-modal matching or 3D volumetric prediction. Two main lines emerge: disentangled 2D encoding for retrieval and correspondence, and volumetric U-net architectures for 3D synthesis.

• Disentangled Embedding (Retrieval/Matching) (Sain et al., 2024):
  • View-agnostic retrieval: use $f_c(I)$.
  • View-specific retrieval: use $p_{ij} = R(\gamma_i,v_j)$0.
• 3D Volumetric U-net (Reconstruction) (Delanoy et al., 2017):

Single-view and multi-view modules predict a 256×256×64 occupancy volume from bitmap sketches. Both use an 8-layer encoder–decoder U-net with skip connections and per-voxel softmax for $p_{ij} = R(\gamma_i,v_j)$1. The Updater network additionally ingests the current estimate $p_{ij} = R(\gamma_i,v_j)$2 (reprojected into the camera frame) midway in the encoder, merging new sketch gradients with prior volumetric prediction.

• Local Descriptor Learning (Pixel Correspondence) (Yu et al., 2020):

SketchDesc-Net employs a fully-convolutional, multi-branch network with four parallel branches processing concentric patches (32×32 to 256×256) around each pixel, all resized to 32×32. Features (128-dim per branch) are concatenated and mapped via a shared FC layer to a 128-dim embedding, producing scale-invariant, locality-aware descriptors for correspondence.

3. Training Objectives and Loss Functions

Multi-view sketch systems employ an array of losses to enforce cross-modal discrimination, view disentanglement, and correspondence:

Loss Name	Functional Purpose	Notation/Equation
View-agnostic triplet loss	Content-invariant retrieval	$p_{ij} = R(\gamma_i,v_j)$3
View-specific triplet loss	View-aware matching	$p_{ij} = R(\gamma_i,v_j)$4
View-consistency loss	Aligns view codes in sketch/photo	$p_{ij} = R(\gamma_i,v_j)$5
Instance-consistency (projs.)	Unifies content across projections	$p_{ij} = R(\gamma_i,v_j)$6
Cross-view reconstruction loss	Enforces composable disentanglement	$p_{ij} = R(\gamma_i,v_j)$7
Triplet loss (SketchDesc)	Pulls positives, pushes negatives	$p_{ij} = R(\gamma_i,v_j)$8

The total objective in SketchVCL Multi-View combines these terms:

$p_{ij} = R(\gamma_i,v_j)$9

with empirically set weights $v_j \in \{0^\circ, 15^\circ, \ldots, 345^\circ\}$0, $v_j \in \{0^\circ, 15^\circ, \ldots, 345^\circ\}$1.

For 3D volumetric networks, the principal loss is a per-voxel binary cross-entropy between predicted occupancy probability and ground truth.

4. Iterative Multi-View Reasoning and Inference Protocols

Multi-view integration is realized through both iterative volumetric update and flexible feature selection.

• 3D Volumetric Fusion (Updater):

The reconstruction process applies the Updater CNN in cycle: given $v_j \in \{0^\circ, 15^\circ, \ldots, 345^\circ\}$2 input sketches $v_j \in \{0^\circ, 15^\circ, \ldots, 345^\circ\}$3 from known viewpoints, initialize $v_j \in \{0^\circ, 15^\circ, \ldots, 345^\circ\}$4, then iterate

$v_j \in \{0^\circ, 15^\circ, \ldots, 345^\circ\}$5

for $v_j \in \{0^\circ, 15^\circ, \ldots, 345^\circ\}$6 ($v_j \in \{0^\circ, 15^\circ, \ldots, 345^\circ\}$7 in practice). Each pass integrates another viewpoint, correcting errors and refining occluded geometry, with rapid empirical convergence ($v_j \in \{0^\circ, 15^\circ, \ldots, 345^\circ\}$8 drops sharply).

• Feature Customization for Retrieval:

Test-time selection between $v_j \in \{0^\circ, 15^\circ, \ldots, 345^\circ\}$9 (view-agnostic) and $R_j$0 (view-specific) is performed without retraining, granting explicit user control over the granularity of retrieval.

• Pixel-wise Correspondence:

SketchDesc locates semantically corresponding pixels by comparing 128-dim descriptors via nearest-neighbor search, establishing cross-view semantic maps even under large viewpoint disparity.

5. Evaluation and Empirical Outcomes

Quantitative evaluation of SketchVCL Multi-View-type systems spans volumetric reconstruction, retrieval metrics, and correspondences:

• 3D Reconstruction (Delanoy et al., 2017):
  • Single-view mean IoU: chairs $R_j$1, vases $R_j$2, procedural shapes $R_j$3.
  • Multi-view (4 views, 5 iterations): IoU $R_j$4 (procedural).
  • CNNs outperform silhouette carving, particularly in capturing concavities and nontrivial topology.
• Cross-modal Retrieval (Sain et al., 2024):
  • Chairs, VGG-16 backbone: View-agnostic mAP 0.615; view-specific top-1 accuracy 60.7%.
  • Switching to PVT yields mAP 0.689, top-1 accuracy 67.1%.
  • SketchVCL Multi-View consistently outperforms prior art (e.g., StrongPVT).
• Pixel-level Correspondence (Yu et al., 2020):
  • On synthetic and hand-drawn sketches, multi-view pixel-wise retrieval mAPs: Structure-Recovery 0.82 (vs. AlexNet-VP 0.67), PSB 0.73, ShapeNet 0.66.
  • Robust under wide viewpoint changes and stylized/skewed sketches.

Empirical runtimes for 3D reconstruction are 140 ms (single-view) and $R_j$5350 ms (4 views, 5 iterations) on a modern GPU.

6. Limitations, Scalability, and Extensions

Major constraints and scalable advantages include:

• Data Dependence: Access to 3D CAD models for rendering multi-view supervisory data is required; categories lacking extensive 3D collections cannot be easily adapted (Sain et al., 2024).
• Discrete Viewpoints: Supporting arbitrary continuous view angles demands either denser projections or explicit pose regression.
• Input Robustness: While volumetric models are robust to moderate drawing errors, severe misalignment, excessive line thickness, or occlusive views degrade quality (Delanoy et al., 2017).
• Sparse Geometry: Objects with extremely thin/sparse topology challenge silhouette-based renderings (Sain et al., 2024).
• Model Design: Pixel-wise SketchDesc descriptors are independent per pixel, neglecting spatial coherence, though future directions suggest incorporating graph-based or deformation-aware modules (Yu et al., 2020).

Potential expansions involve joint learning of view/perspective estimation, extension to non-rigid objects, or integration with photometric edge cues.

7. Applications and Implications

SketchVCL Multi-View frameworks enable a spectrum of practical and research applications:

• Interactive 3D Modeling: Real-time, iterative modeling from hand-drawn sketches via direct volumetric prediction, facilitating rapid, user-guided object creation (Delanoy et al., 2017).
• Fine-grained Cross-modal Retrieval: User-driven sketch-based search of photo collections or object databases with the option for view-specific or invariant results (Sain et al., 2024).
• Multi-view Correspondence Estimation: Dense, semantic mapping of points or parts across disparate sketch views, supporting tasks in annotation transfer, mesh analysis, and sketch-based design (Yu et al., 2020).

This synergistic approach—melding view-disentangling representations, synthetic multi-view data, and both volumetric and fine-grained correspondence learning—defines best practices for advanced sketch-based multi-view systems.