--- title: Multi-View Diffusion Trajectories url: https://www.emergentmind.com/topics/multi-view-diffusion-trajectories-mdts-b8efd0d2-2d28-49f7-be46-9dd66c784e11 type: topic --- # Multi-View Diffusion Trajectories to=arxiv_search.search 天天彩票与你同行json {"query":"all:multi-view diffusion trajectories OR all:multi-view diffusion OR all:camera-controllable multi-view video diffusion OR all:autoregressive multi-view diffusion novel view synthesis", "max_results": 10, "sort_by": "relevance"} to=arxiv_search.search 大发快三官网 微信上的天天中彩票json {"query":"2412.02631 OR 2410.10774 OR 2404.17419 OR 2405.03894 OR 2404.02148 OR 2405.20674 OR 2505.09315 OR 1508.05550 OR 2411.07449", "max_results": 20, "sort_by": "relevance"} to=arxiv_search.search аанацҳауеит 天天彩票提现json {"query":"ti:\"Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation\"", "max_results": 5, "sort_by": "relevance"} Multi-View Diffusion Trajectories (MDTs) is not yet a standardized formal object in the arXiv literature. The phrase is best used as an umbrella description for diffusion processes whose denoising path is jointly structured across multiple viewpoints, synchronized video streams, or camera-conditioned trajectory queries, rather than over isolated images. Under that reading, MDT-related systems range from fixed-grid multi-view latent refiners such as Sharp-It to camera-controllable multi-view video models such as Cavia, autoregressive novel-view systems such as CausNVS, and dynamic 4D generators such as Diffusion\(^2\) and 4Diffusion; what unifies them is that the evolving variable during reverse diffusion is a coupled multi-view state, and not a collection of independent per-view samples [2412.02631][2410.10774][2509.06579][2404.02148][2405.20674]. ## 1. Conceptual scope and defining properties In current usage, MDTs are best understood as a family resemblance rather than a single formalism. Some systems denoise a fixed ordered set of rendered views packed into one latent tensor; some denoise a view-time lattice; some roll out camera-conditioned views autoregressively; and some use diffusion to generate physical or control trajectories that then condition multi-view rendering. A recurring distinction is whether the coupled object is a **set of views**, a **video tensor with a view dimension**, or a **trajectory/control representation** that later anchors image generation. A second distinction concerns how cross-view consistency is obtained. In several image-first systems, consistency is largely **implicit, learned, and architectural**: all views are processed jointly, and attention over the packed representation serves as cross-view communication. In other systems, camera geometry is injected explicitly through Plücker coordinates, pairwise-relative pose encodings, or epipolar weighting. A third distinction is whether denoising is **joint and parallel** over a fixed bundle of views, or **causal/autoregressive** over a camera trajectory. These differences matter because they determine whether an MDT can support arbitrary camera queries, streaming inference, or only a fixed camera layout. | MDT regime | Representative systems | Core coupling mechanism | |---|---|---| | Fixed-view joint latent denoising | Sharp-It, MultiImageDream, MVDiff | Shared latent tensor, self-attention or dense 3D attention, sometimes epipolar weighting | | Multi-view video denoising | Cavia, Diffusion\(^2\), 4Diffusion | Joint view-time tensor or row/column score composition | | Autoregressive camera-trajectory rollout | CausNVS, Virtually Being | Causal attention with camera encodings and sequential view generation | | Trajectory-conditioned control and rendering | RiskMV-DPO, TransDiffuser, MBD | Diffusion over planned trajectories or trajectory-conditioned controls | A common misconception is that any multi-view diffusion model is automatically an MDT. That is too broad. A fixed \(3\times2\) grid with no arbitrary pose support and no explicit trajectory variable is MDT-like only in the weaker sense that its reverse diffusion path is jointly defined over multiple views. Conversely, a planner such as TransDiffuser is not a canonical multi-view image model, but it is relevant because diffusion is used to generate a trajectory distribution conditioned on rich multi-sensor context, and that trajectory view of diffusion transfers directly to MDT design [2505.09315]. ## 2. Joint latent denoising over fixed view sets A major MDT branch operates on a fixed, ordered set of target views and treats the entire set as one denoised object. "Sharp-It" is exemplary. It starts from a coarse but 3D-consistent object produced by Shap-E, renders it into six predefined camera views arranged as a \(3\times 2\) grid, and applies a multi-view-to-multi-view latent diffusion refiner before reconstructing with a feed-forward sparse-view model such as InstantMesh. The model is built on Zero123++, uses Stable Diffusion’s VAE latent space, expands the UNet input to 8 channels—4 for noisy latents and 4 for VAE-encoded degraded renderings—and relies on global self-attention over a packed \(960\times640\) six-view grid. The training objective is standard latent diffusion with v-prediction, \[ \mathcal{L} = \mathbb{E}_{t,\epsilon \sim \mathcal{N}(0,1)} \left[ \| v - v_\theta(x_t, x_{\text{Shap-E}}, c_{\text{text}}) \|^2 \right], \] with a CFG drop probability of 0.1. The method preserves consistency through the coarse 3D prior, joint processing, and conditioning on the degraded view set, but it does not introduce explicit epipolar attention, triplanes, pose embeddings, differentiable reprojection, or geometric consistency losses. On its Objaverse-derived paired dataset, Sharp-It reports **FID 6.60**, **CLIP 0.90**, and **DINO 0.92**, versus **FID 19.13**, **CLIP 0.87**, and **DINO 0.89** for the strongest reported Zero123++ with SDEdit baseline, with runtime around **10 seconds** [2412.02631]. MultiImageDream shows the same fixed-bundle logic under image prompting rather than 3D refinement. It extends ImageDream, itself derived from MVDream, from one prompt image to multiple prompt images without fine-tuning. MVDream jointly denoises four orthogonal target views with densely connected 3D attention over a stacked feature map of shape \((b,4,c,h_l,w_l)\). ImageDream adds a local controller based on resampled CLIP features and a pixel controller that appends a prompt-image latent, changing the attention tensor to \((b,4+1,c,h_l,w_l)\). MultiImageDream generalizes this by concatenating multiple local-token banks and stacking multiple prompt-image latents, giving \((b,N+4,c,h_l,w_l)\) for \(N\) prompt images. The generated four-view latents therefore evolve under a shared denoising trajectory that is continuously anchored by multiple observed-view conditions. Quantitatively, the strongest gains appear on synthesized multi-view imagery: for example, “2-ImageDream - pixel(f) + local(fb)” improves the single-image baseline from **QIS \(27.10 \pm 12.8\)**, **CLIP(TX) \(32.39 \pm 2.78\)**, **CLIP(IM) \(85.69 \pm 3.77\)** to **QIS \(28.95 \pm 10.2\)**, **CLIP(TX) \(32.54 \pm 2.65\)**, **CLIP(IM) \(87.10 \pm 3.09\)**, while 3D gains after SDS-based NeRF optimization are present but more limited [2404.17419]. MVDiff occupies a related but more geometry-explicit position. It builds a Scene Representation Transformer that aggregates one or more source views into a latent scene representation \(\boldsymbol{z}\), predicts a coarse \(32\times32\) target latent \(\tilde{\boldsymbol{x}}_{\mathrm{T}}\), and feeds multiple target views jointly into a latent diffusion UNet. Its most explicit geometric device is epipolar attention: for each pair of views, it builds a weighted affinity correction from inverse epipolar distance and modifies attention affinities as \[ A'_{i,j}=A_{i,j}+W_{i,j}. \] The paper states that target views are predicted simultaneously rather than sequentially. In ablation, removing epipolar attention reduces PSNR/SSIM/LPIPS from **20.24/0.884/0.095** to **19.14/0.864/0.118**, while removing multi-view attention gives **19.92/0.871/0.113**. On GSO novel-view synthesis with one reference view, MVDiff reports **PSNR 20.24**, **SSIM 0.884**, **LPIPS 0.095**, improving over Zero123-XL’s **18.93/0.856/0.124**; for downstream GSO reconstruction it reports **Chamfer Distance 0.0411** and **Volume IoU 0.4357** with one input view, and improves further with more reference views [2405.03894]. These fixed-view systems illustrate a narrow but important MDT regime. Their trajectory is the reverse diffusion path of a packed multi-view latent, usually under a fixed camera layout. This suggests that synchronized denoising alone can be a strong source of multi-view coherence, but also that flexibility in camera graphs, trajectory length, and online querying is limited unless the formulation is widened beyond a fixed bundle. ## 3. Multi-view video denoising on view-time lattices A stronger MDT interpretation appears when diffusion is defined over both **view** and **time**. Cavia does this explicitly. It extends Stable Video Diffusion to camera-controllable multi-view video generation and represents the latent state as a tensor of shape \[ (B\;V\;F\;C\;H\;W), \] where \(V\) is the number of views and \(F\) is frames per view. Camera control is encoded by Plücker ray coordinates derived from extrinsics and intrinsics, concatenated channel-wise with latent inputs. The architectural core is View-Integrated Attention: cross-frame attention rearranges features so that attention spans all spatiotemporal tokens within a view, and cross-view attention rearranges them so that synchronized timesteps from all views attend jointly. This gives direct communication between different camera trajectories of the same scene. Cavia is trained with EDM-style denoising score matching, \[ \mathbb{E}\left[\left\|D_{\boldsymbol{\theta}}(\mathbf{x}_0+\mathbf{n}; \sigma, \text{cond})-\mathbf{x}_0\right\|_2^2\right], \] on a mixture of static scene/object multi-view videos, synthetic multi-view dynamic videos, and monocular dynamic videos with estimated poses. Its ablations are unusually MDT-relevant: removing cross-view attention causes different object motions to appear in different views, and removing cross-frame attention causes severe distortions. On RealEstate10K monocular camera control, Cavia reports **FID 11.43**, **FVD 55.10**, and **COLMAP error 14.4%**; on two-view generation it reports **FID 8.82**, **FVD 94.86**, and **Prec. \(29.39\)**, **MS. \(15.22\)** on Real10K, outperforming CameraCtrl in the reported table [2410.10774]. Diffusion\(^2\) addresses the same joint structure from a different angle. Rather than training a native multi-view video model, it composes a pretrained video diffusion prior and a pretrained multi-view diffusion prior over a dense image array \[ \mathcal{I} = \left\{ I_{i,j} \in \mathbb{R}^{H \times W \times 3} \right\}_{i=1,j=1}^{V,F}. \] Its key theorem assumes conditional independence between the same-frame multi-view context and same-view temporal context given a center cell, yielding the score identity \[ \nabla_{\mathrm{x}} \log p\left( \hat{\mathcal{I}} \right) = \nabla_{\mathrm{x}} \log p\left( \hat{\mathcal{I}}_{\{1:V\},j} \right) + \nabla_{\mathrm{x}} \log p\left( \hat{\mathcal{I}}_{i,\{1:F\}} \right) - \nabla_{\mathrm{x}} \log p\left( \hat{I}_{i,j} \right). \] In practice the unknown single-image marginal score is approximated by a convex combination of the row and column scores, with a logistic schedule \[ s = 1-\frac{1}{1+e^{k(i/N-s_0)}}, \] and the paper reports \(s_0=0.5\), \(k=20\) as the best setting. This is presented as a way to decouple geometry-consistent generation and temporally smooth appearance during denoising. The output is a dense multi-view, multi-frame lattice used to optimize 4D Gaussian Splatting. The paper reports around **10 minutes** end-to-end runtime and gives user-study evidence that Diffusion\(^2\) improves geometric consistency and overall model quality over Animate124 and DreamGaussian4D, while CLIP similarity on video-to-4D is **0.94** versus **0.92** for Efficient4D [2404.02148]. 4Diffusion replaces score composition with a learned unified denoiser. It starts from ImageDream, inserts zero-initialized motion modules into a frozen 3D-aware UViT, and trains a multi-view video diffusion model, 4DM, on a curated set of **926** animated Objaverse assets rendered as synchronized multi-view videos. The latent tensor has shape \[ \mathbf{Z} \in \mathbb{R}^{B \times F \times N \times C \times H \times W}, \] with \(N=5\) viewpoints: one source monocular view and four target views. Spatial modules reuse ImageDream’s 3D self-attention across views, while motion modules reshape the tensor to apply temporal self-attention across frames. The resulting denoiser is used in a 4D-aware SDS objective, \[ \nabla_{\theta_{\text{4D}}} \mathcal{L}_{\text{4D-SDS}} \approx \mathbb{E}_{(\mathbf{c}_{\text{mv}},\epsilon,t)} \left[ 2 (V_r-\hat{V}_0)\frac{\partial V_r}{\partial \theta_{\text{4D}}} \right], \] to optimize a hash-encoded dynamic NeRF, together with an anchor loss and regularizers. In direct multi-view video evaluation against ImageDream, 4DM reports **CLIP-I 0.8630**, **LPIPS 0.1644**, **CLIP-C 0.9640**, and **FVD 523.90**, compared with ImageDream’s **0.8776**, **0.1704**, **0.9412**, and **591.25**. On full 4D generation, the final system reports **CLIP-I 0.8803**, **CLIP-C 0.9654**, and **FVD 1196.8**, outperforming Consistent4D, DreamGaussian4D, and 4D-fy in the reported benchmark [2405.20674]. Together, these systems show two distinct routes to MDTs over view-time structure. One route composes orthogonal score fields over rows and columns; the other trains a single denoiser whose latent trajectory already spans views and frames. This suggests that “trajectory” can refer both to the reverse diffusion path and to the induced coupling topology over a view-time grid. ## 4. Camera-controlled and autoregressive trajectory formulations CausNVS pushes MDTs toward open-ended novel-view synthesis. It addresses the limits of non-autoregressive multi-view diffusion by generating target views sequentially, conditioned on accumulated context and target poses. Given input views \[ \{(\bm{x}_1,\bm{p}_1),\dots,(\bm{x}_N,\bm{p}_N)\} \] and target poses \[ \{\bm{p}_{N+1},\dots,\bm{p}_{N+M}\}, \] it represents each frame as \[ \bm{v}_i = (\bm{z}_i^{t_i}, \bm{p}_i), \] uses causal masking in frame-wise attention, and trains with independent per-frame noise levels under \[ \mathcal{L}_{\text{causal}} = \mathbb{E}_{\{(\bm{x}_i, \bm{p}_i), t_i, \bm{\epsilon}_i\}_{i=1}^{F}} \sum_{i=1}^{F} \left\| \bm{\hat\epsilon}_{\theta}\!\left(\bm{v}_i \mid \bm{v}_{