ArbiViewGen: Arbitrary View Synthesis

• ArbiViewGen is a diffusion-based framework for controllable arbitrary viewpoint synthesis, generating plausible images for unseen camera poses in autonomous driving scenes.
• It integrates Feature-Aware Adaptive View Stitching (FAVS) with Cross-View Consistency Self-Supervised Learning to generate pseudo views when ground-truth images are unavailable.
• The method supports data augmentation, sensor prototyping, and simulation, while addressing challenges like extreme viewpoint extrapolations and dynamic scene details.

Searching arXiv for the specified paper and closely related arbitrary-view autonomous-driving generation work. arXiv.search query: "ArbiViewGen controllable arbitrary viewpoint camera data generation autonomous driving stable diffusion" arXiv.search query: "\"ArbiViewGen\" OR \"Arbiviewgen\" OR \"controllable arbitrary viewpoint\" autonomous driving diffusion" arXiv.search query: "\"DriveSuprim\" autonomous driving arbitrary view" ArbiViewGen is a diffusion-based framework for controllable camera image generation from arbitrary viewpoints in autonomous driving scenes. It addresses the regime in which a target camera pose or sensor layout was never physically collected and no ground-truth RGB image exists at that extrapolated viewpoint. The framework uses only standard multi-camera surround-view images and their associated poses, without LiDAR, depth, or additional sensors, and combines Feature-Aware Adaptive View Stitching (FAVS) with Cross-View Consistency Self-Supervised Learning (CVC-SSL) to construct pseudo novel views and train a latent diffusion model without direct supervision at the target pose (Lan et al., 7 Aug 2025).

1. Problem definition and scope

In ArbiViewGen, an arbitrary viewpoint is a camera pose in $SE(3)$ that may differ from the original camera extrinsics, may correspond to a different vehicle configuration, and typically lies off the original data-collection trajectory. In the autonomous-driving setting assumed by the method, a vehicle has a ring of rigidly mounted cameras with known intrinsics and extrinsics; the task is to synthesize what a new camera at a user-specified pose and intrinsics would observe, even when that pose was never realized during data collection (Lan et al., 7 Aug 2025).

This problem is motivated by data augmentation and sensor redesign, simulation and safety testing, and cross-platform model reuse. The intended use cases include reusing an existing fleet’s data for new camera layouts, simulating unusual or more informative viewpoints, and reducing collection and annotation cost when transferring perception systems across vehicle platforms. A central obstacle is that real driving datasets provide images only from the physically mounted cameras and only along the realized ego trajectory. As a consequence, large viewpoint extrapolations have no ground-truth target image. The paper further identifies sparse 3D coverage, occlusion, dynamic scenes, and the need to preserve multi-camera geometric consistency as primary sources of difficulty.

The research is positioned against two families of prior methods. Diffusion-based novel view synthesis methods such as Zero-1-to-3, StreetCrafter, DiST-4D, DriveX, and Drive-1-to-3 are described as typically requiring paired input-target supervision, often being object-centric or limited to small viewpoint changes, and in some cases depending on extra sensing modalities; StreetCrafter is noted as using LiDAR and DiST-4D as using metric depth. Reconstruction-based methods such as NeRF and 3D Gaussian Splatting are characterized as two-stage pipelines that can be sensitive to sparse views and dynamic urban scenes, especially under large extrapolations. ArbiViewGen is presented as an alternative that replaces unavailable novel-view supervision with pseudo-ground-truth generation and self-supervised cyclic reconstruction. The paper states that, to its knowledge, it is the first method capable of controllable arbitrary view camera image generation in multiple vehicle configurations (Lan et al., 7 Aug 2025).

2. System-level formulation

The framework operates on six RGB images $\{I_1,\dots,I_6\}$ and their poses $\{P_1,\dots,P_6\}$, corresponding in nuScenes to front, front-left, front-right, back-left, back-right, and back cameras. A target arbitrary viewpoint is represented by intrinsics $K$ and pose $(R,t)$; for some components of FAVS, rotation is parameterized by azimuth $\theta$ and elevation $\phi$ (Lan et al., 7 Aug 2025).

The pipeline has two principal components. First, FAVS takes the real multi-camera images and poses together with a target pose $P_{\text{target}}$ and warps and fuses the available observations into a stitched pseudo image at that pose. This stitched image is intended to be geometrically plausible, but it is not assumed to be photorealistic in unobserved or weakly constrained regions. Second, CVC-SSL uses those pseudo views as conditions for a latent diffusion model and trains the model to reconstruct original real camera images. The reconstruction objective induces a closed-loop constraint: real multi-view observations are converted into pseudo novel views, and those pseudo views are then required to predict back the original camera views (Lan et al., 7 Aug 2025).

Training and inference are organized differently. During training, for each real camera image $I_i$ at pose $P_i$, the method samples pseudo poses to the left and right of $\{I_1,\dots,I_6\}$0, synthesizes pseudo views at those poses with FAVS, and trains the diffusion model to reconstruct $\{I_1,\dots,I_6\}$1. During inference, arbitrary user-specified poses are provided, FAVS generates corresponding pseudo target views, and the trained diffusion model produces high-quality images at those poses. This design implies that viewpoint controllability is implemented as conditioning on pose and intrinsics rather than as selection from a discrete set of training viewpoints.

A common misunderstanding is to treat the stitched FAVS output as the final rendering. In the framework as described, FAVS provides a pseudo-view condition; the final generated image is produced by the latent diffusion model conditioned on that pseudo view and pose information. This distinction is central to the method’s separation between geometric plausibility and photorealistic completion.

3. Feature-Aware Adaptive View Stitching

FAVS constructs a pseudo image at an arbitrary viewpoint through a four-stage hierarchy: geometric transformation foundation, feature matching optimization, object alignment fine-tuning, and adaptive fusion generation (Lan et al., 7 Aug 2025).

The first stage uses a far-field approximation under which inter-camera mapping is approximated by a rotation-only homography. With source intrinsics and rotation $\{I_1,\dots,I_6\}$2 and target intrinsics and rotation $\{I_1,\dots,I_6\}$3, the method uses

$\{I_1,\dots,I_6\}$4

The paper explicitly contrasts this with a full planar homography formulation and states that ArbiViewGen does not rely on planar assumptions. Instead, it validates the coarse homography by transforming image corners and checking for excessive distortion through a homography-validity test.

Because pure geometry is imperfect under pose noise, parallax, and non-pure-rotation effects, the second stage refines alignment by feature matching. The method extracts SIFT features, matches them with FLANN, filters matches using Lowe’s ratio test with threshold $\{I_1,\dots,I_6\}$5, and estimates a refined homography with RANSAC. Reliability is then assessed using three criteria: sufficient matches, homography validity, and consistency with the geometric homography. Depending on that consistency, the base transformation is chosen as the feature homography, the geometric homography, or a dynamically weighted interpolation between them:

$\{I_1,\dots,I_6\}$6

This stage is therefore neither purely pose-driven nor purely appearance-driven; it fuses both sources of evidence.

The third stage addresses residual object-level misalignment. Keypoints are clustered with DBSCAN, and each cluster is treated as an object with center, bounding box, confidence, and type. For each cluster, source and target centroids are computed, and a small 2D translation $\{I_1,\dots,I_6\}$7 is estimated to minimize weighted centroid discrepancies:

$\{I_1,\dots,I_6\}$8

where the weights combine confidence, type weight, and density factor. The final aligned homography is then obtained by composing $\{I_1,\dots,I_6\}$9 with a translation matrix scaled by $\{P_1,\dots,P_6\}$0.

The fourth stage warps every source image into the target frame and blends them using spatially varying weights. For each source view,

$\{P_1,\dots,P_6\}$1

and the fusion weight is factored into distance, gradient, quality, and primary-camera terms:

$\{P_1,\dots,P_6\}$2

The resulting stitched pseudo target image is

$\{P_1,\dots,P_6\}$3

Distance weighting favors central regions and discourages edges and holes, gradient weighting suppresses seam artifacts, quality weighting reflects match quality, and a primary-camera weight boosts the main contributing camera. This suggests that FAVS is designed less as a conventional panorama stitcher than as a pseudo-ground-truth constructor optimized for downstream generative conditioning.

4. Cross-view self-supervision and diffusion architecture

CVC-SSL is the training paradigm that removes the need for ground-truth images at extrapolated viewpoints. For each real view $\{P_1,\dots,P_6\}$4 at pose $\{P_1,\dots,P_6\}$5, the method samples pseudo target poses to the left and right, generates pseudo images at those poses with FAVS using the full six-camera context, and conditions a diffusion model on the pseudo views and their poses in order to reconstruct the original real view $\{P_1,\dots,P_6\}$6 (Lan et al., 7 Aug 2025).

The paper characterizes this as a cross-view cycle:

$\{P_1,\dots,P_6\}$7

Because the supervision is applied on original real images rather than unavailable extrapolated targets, the model is trained self-supervisedly while still being forced to learn mappings across viewpoints. The total loss is

$\{P_1,\dots,P_6\}$8

with $\{P_1,\dots,P_6\}$9 and $K$0. The main loss is the standard latent-diffusion noise prediction objective, the geometric consistency loss constrains predicted cross-view attention maps to match geometry-derived targets through a Frobenius norm, and the perceptual loss is VGG-based.

The generative backbone is a Stable Diffusion-style latent diffusion model comprising a VAE encoder $K$1, a latent-space U-Net denoiser $K$2, and a decoder $K$3. ArbiViewGen extends this backbone with a multi-branch U-Net in which there is one branch per view but all branches share exactly the same weights. Noise prediction is performed for all views in parallel, which avoids sequential error accumulation.

Cross-view consistency is further enforced by geometry-guided cross-view attention through Correspondence-Aware Attention (CAA), described as based on MVDiffusion-style modules. For a token at position $K$4 in view $K$5, a corresponding position in view $K$6 is computed through homography:

$K$7

with multi-level attention aggregated over layers and masked to overlap regions predicted by geometry. Geometry-aware positional encoding uses the deviation between geometric prediction and actual position,

$K$8

encoded through sinusoidal functions. In operational terms, these modules inject explicit camera-pose structure into feature interaction, so the model’s consistency mechanism is not purely learned from appearance statistics.

At inference time, controllability is direct. The user specifies desired camera poses and intrinsics for a new configuration; for each desired view, FAVS computes an initial stitched pseudo view, and the reverse diffusion process is run with classifier-free guidance strength $K$9 and, as reported, $(R,t)$0 denoising steps. The paper’s formulation implies that new vehicle layouts are handled by changing the conditioning geometry rather than retraining a layout-specific network.

5. Data, training protocol, and evaluation

The reported experiments use nuScenes, which contains 1,000 scenes, each with approximately 40 keyframes at 2 Hz, together with 6 cameras, 5 radars, 1 LiDAR, and precise calibration. ArbiViewGen uses only the multi-camera RGB images and camera poses for training the generative model. The data split is 60% of scenes for training, approximately 24,000 frames; 20% for validation; and 20% for extrapolated viewpoint generation and evaluation, approximately 34,000 annotated frames after LiDAR-based projection (Lan et al., 7 Aug 2025).

The implementation uses a Stable Diffusion-style latent diffusion model with U-Net and VAE. Cross-view attention modules are inserted at encoder levels with 320, 640, 1280, and 1280 channels, at the middle with 1280 channels, and at decoder levels with 1280, 1280, 640, and 320 channels. Training uses a learning rate of $(R,t)$1 with cosine annealing, batch size 8 per GPU, and 1000 diffusion steps during training. Inference uses 50 diffusion steps.

Evaluation is sparse because no dense ground-truth RGB exists for arbitrary extrapolated viewpoints. The paper therefore colors LiDAR point clouds with the original six camera images and projects those colored points into novel viewpoints to obtain sparse reference pixels. PSNR, SSIM, MAE, and RMSE are then computed on those sparse samples. This protocol is significant because it preserves physically grounded reference data while explicitly acknowledging that the evaluation target is only partially observed.

Against the main reference baseline, DriveSuprim, ArbiViewGen reports PSNR $(R,t)$2 versus $(R,t)$3, SSIM $(R,t)$4 versus $(R,t)$5, MAE $(R,t)$6 versus $(R,t)$7, and RMSE $(R,t)$8 versus $(R,t)$9. The ablation study reports a stepwise progression from Geometric to FAVS to the full model. Geometric alone obtains PSNR $\theta$0, SSIM $\theta$1, MAE $\theta$2, and RMSE $\theta$3; FAVS alone improves these to PSNR $\theta$4, SSIM $\theta$5, MAE $\theta$6, and RMSE $\theta$7; the full model reaches the previously stated ArbiViewGen scores. The qualitative analysis described in the paper further reports that DriveSuprim suffers from geometric distortions and object misalignment under large rotations, Geometric-only preserves rough alignment but exhibits tearing, black borders, and missing content, FAVS improves alignment and texture continuity but still leaves discontinuities, and the full system generates more complete and structurally consistent images under substantial viewpoint changes such as $\theta$8, $\theta$9, $\phi$0, $\phi$1, and $\phi$2, $\phi$3 rotations (Lan et al., 7 Aug 2025).

6. Practical roles in autonomous-driving research

ArbiViewGen is explicitly framed as a tool for data augmentation and cross-platform reuse, sensor-layout prototyping, scenario and safety analysis, and support for multiple vehicle configurations (Lan et al., 7 Aug 2025).

For data augmentation and cross-platform reuse, the method can generate images for hypothetical camera layouts on new vehicle models from existing multi-camera datasets. The stated objective is to train perception systems for new sensor configurations without collecting large new corpora. For sensor-layout prototyping, the user can specify new camera positions, orientations, and fields of view, then inspect generated observations under those proposed layouts. This suggests a workflow in which camera placement can be evaluated before hardware deployment.

For simulation and safety analysis, the method can render dangerous or rare scenes from more informative viewpoints, such as slightly higher or angled cameras, in order to stress-test downstream perception systems. Because conditioning is based on camera pose and intrinsics rather than a fixed sensor graph, the system is described as agnostic to the exact number and placement of cameras so long as poses are available and FAVS can fuse them. A plausible implication is that ArbiViewGen is intended not merely as a novel-view synthesis method but as a bridge between archival fleet data and prospective sensor designs.

The method should not, however, be conflated with a full 3D scene reconstruction pipeline. Its central mechanism is pseudo-view generation plus self-supervised diffusion, not explicit scene representation learning in the NeRF or Gaussian-splatting sense. That distinction matters for interpreting both its strengths and its failure modes.

7. Limitations, open issues, and future directions

The reported limitations concern extrapolation range, dynamic-scene fidelity, calibration dependence, and the quality ceiling imposed by pseudo-view conditioning (Lan et al., 7 Aug 2025).

Extreme extrapolations remain difficult, especially for poses far off trajectory or with very high pitch. Because the training signal is still mediated by sparse geometric support and cyclic reconstruction, the model may hallucinate unrealistic content in heavily extrapolated regions. Dynamic scenes and fine details are also difficult: thin poles, detailed pedestrian limbs, and other fine-grained structures are explicitly identified as challenging, and FAVS can misalign dynamic objects before diffusion-based correction.

The framework is also dependent on pose and calibration accuracy. Since FAVS relies on known camera intrinsics and extrinsics, errors in those quantities degrade pseudo-view quality and thereby degrade the self-supervised signal used by CVC-SSL. In addition, the stitched FAVS outputs are acknowledged to be visibly non-photorealistic in some regions, with tearing and ghosts in poorly observed areas. The diffusion model is intended to correct such defects, but if the pseudo views are too poor, the supervisory cycle becomes noisy.

Future directions mentioned in the paper include sparse-to-dense supervision, more direct use of LiDAR-based depth priors during training rather than only for evaluation, semantic consistency constraints such as segmentation-level consistency across views, and improved handling of dynamic objects and fine structures potentially through explicit motion or depth modeling. These directions indicate that the present system is best understood as a geometry-aware, self-supervised generative framework that reduces the dependence on ground-truth novel views, rather than as a complete solution to arbitrary-view scene synthesis in all driving regimes (Lan et al., 7 Aug 2025).