XFactor: Geometry-Free Latent Pose for NVS

• XFactor is a geometry-free self-supervised novel view synthesis model that leverages transformer modules to extract and transfer 256-dimensional latent pose representations.
• It employs a dual-augmentation scheme and a transferability loss to ensure that the latent vector captures only geometric information while discarding scene content.
• Experimental results show significant improvements in metrics like AUC and RRA over baselines, confirming effective cross-scene viewpoint transfer.

XFactor is a geometry-free self-supervised model for novel view synthesis (NVS) that enforces true viewpoint transferability without explicit multi-view geometry priors or SE(3) parameterizations. Unlike prior pose-free NVS approaches, XFactor's key criterion is that its latent pose representations, extracted from one scene, control the viewpoint when used to synthesize images in completely different scenes, thus supporting transferable camera motion without reliance on explicit 3D structure (Mitchel et al., 15 Oct 2025).

1. Architectural Design and Pair-wise Pose Estimation

XFactor factorizes NVS into two transformer-based modules: POSEENC and RENDER. POSEENC is a stereo-monocular pose encoder that processes exactly two images—a context image $I_c$ and a target image $I_t$—and outputs an unconstrained 256-dimensional latent vector $z \in \mathbb{R}^{256}$ describing their relative camera motion. RENDER is a monocular renderer that accepts a single context image $I_c$ and latent pose $z$ to synthesize the target view $\hat{I}_t$. Both modules share a Vision-Transformer (ViT) backbone employing local–global attention and rotary position embeddings. POSEENC interleaves per-image self-attention with cross-view global tokens to promote extraction of pair-wise geometric information. During multi-view fine-tuning, POSEENC encodes pairs between a fixed reference view and other sequence frames; RENDER uses all context views plus the target latent for synthesis. Restriction to pairwise input in core training ensures that latent interpolations cannot "cheat" by exploiting information from more than two views (Mitchel et al., 15 Oct 2025).

2. Latent Pose Representation

The latent pose $z$ serves as the exclusive representation of relative camera motion, deliberately avoiding any SE(3) parameterization or inductive biases for rotation or translation. $z$ is initialized by a small MLP "head" attached to a shared global transformer token between context and target; the context’s global token is fixed at zero, isolating the relative motion signal. POSEENC and RENDER are trained end-to-end via an objective that enforces transferability (defined below), making it optimal for $z$ to encode only geometric information about relative pose. No explicit regularization (e.g., KL-divergence or norm penalties) is applied to $z$ (Mitchel et al., 15 Oct 2025).

3. Augmentation and Content–Pose Disentanglement

To guarantee that $I_t$0 encodes pose and not scene content, XFactor applies a dual-augmentation scheme for each input pair. Two independent augmentations, $I_t$1 and $I_t$2, are generated for each $I_t$3 pair, preserving the underlying camera transform while minimizing pixel overlap. Specifically, a random binary mask $I_t$4 divides the 16×16 patch grid into two disjoint groups. Each group is processed separately:

• $I_t$5
• $I_t$6
• $I_t$7
• $I_t$8

Here, $I_t$9 denotes elementwise masking. This results in two views, $z \in \mathbb{R}^{256}$0 and $z \in \mathbb{R}^{256}$1, with nearly non-overlapping pixels but identical relative pose. The construction ensures $z \in \mathbb{R}^{256}$2, thus enforcing that latent $z \in \mathbb{R}^{256}$3 must generalize the geometric transformation across content (Mitchel et al., 15 Oct 2025).

4. Objective Function and Training Protocol

Training is driven by a single self-supervised "transferability" loss. POSEENC processes $z \in \mathbb{R}^{256}$4 to produce $z \in \mathbb{R}^{256}$5. RENDER is then tasked with synthesizing $z \in \mathbb{R}^{256}$6 given $z \in \mathbb{R}^{256}$7. The objective minimizes: $z \in \mathbb{R}^{256}$8 where

$z \in \mathbb{R}^{256}$9

The $I_c$0 term measures perceptual similarity. As $I_c$1 and $I_c$2 share only the geometric relationship, minimization of this loss incentivizes $I_c$3 to contain purely pose information. During multi-view fine-tuning, some unmasked pairs are presented to retain autoencoding capacity; the same loss reduces to a two-view autoencoder objective with these pairs. No explicit regularizer constrains $I_c$4 (Mitchel et al., 15 Oct 2025).

5. Transferability Metric: True Pose Similarity (TPS)

To quantify cross-scene transferability, XFactor proposes the True Pose Similarity (TPS) metric. Given two sequences $I_c$5 and $I_c$6 of length $I_c$7 with ground-truth SE(3) poses $I_c$8 and $I_c$9 (from oracles such as COLMAP), TPS is defined as: $z$0 where $z$1 is a relative-pose distance, such as Relative Rotation Accuracy (RRA) or AUC over angular thresholds. At test time, latent sequence $z$2 is produced by POSEENC, RENDER synthesizes $z$3 using sequence $z$4 as context, and TPS is computed as $z$5. High TPS indicates effective viewpoint transfer across scenes (Mitchel et al., 15 Oct 2025).

6. Experimental Protocols and Quantitative Results

Experiments are conducted on a range of real-world video datasets at multiple scales: RE10K, DL3DV, MVImgNet, and CO3Dv2, cropped to 256×256 resolution. Training follows a two-stage protocol:

1. Stereo-monocular pre-training on random frame pairs, with dataset-specific tuning.
2. Multi-view fine-tuning on sequences of 5–6 frames, designating the temporal midpoint as reference.

Competing baselines include RayZer [ICCV ’25] and RUST [CVPR ’23], both predicting SE(3) or partial-view latents and decoding via transformer backbones. For transferability, 4,000 random sequence pairs per dataset are evaluated—each with five target frames—using TPS measured by RRA, RTA (Relative Translation Accuracy), AUC @ {10°, 20°, 30°}, and FID.

Key findings are as follows:

• On AUC@20°, XFactor reports 55–60%, exceeding RayZer and RUST (∼5%) by more than 10×.
• RayZer and RUST may produce plausible interpolations but their TPS is near zero, indicating no real viewpoint transfer.
• XFactor’s FID is comparable to, or better than, baselines, demonstrating no compromise in photometric quality despite transferability enforcement (Mitchel et al., 15 Oct 2025).

7. Pose Probing and Latent–Pose Correspondence

To assess the geometric content of $z$6, a probing experiment regresses ground-truth relative SE(3) poses from frozen POSEENC latents via a 3-layer MLP. Evaluation with RRA, RTA, and AUC shows:

• RRA @ 10°: 98–99% for XFactor; 85–90% for RayZer/RUST
• RTA @ 20°: 85–95% for XFactor; 20–40% for RayZer/RUST
• AUC @ 20°: 50–70% for XFactor; 3–6% for RayZer/RUST

This correlation between $z$7 and accurate pose estimates confirms that XFactor encodes transferable, quantitatively meaningful pose information without explicit geometric priors (Mitchel et al., 15 Oct 2025).

In summary, XFactor demonstrates that enforcing a transferability-based self-supervised objective, coupled with content-destroying, pose-preserving augmentations and avoidance of rigid SE(3) parameterization, enables geometry-free learning of latent pose under true cross-scene viewpoint control.