Multi-view TSI: Textual-Scene Fusion

• Multi-TSI is a paradigm that fuses textual signals with multi-view scene representations to achieve stable semantic and geometric consistency.
• It leverages advanced mechanisms like cross-attention, diffusion, and graph-based fusion to integrate language cues with diverse spatial data.
• Recent advances demonstrate improved motion synthesis, scene editing, and reconstruction by coherently aligning multi-view textual and geometric information.

Multi-view Textual-Scene Interaction (Multi-TSI) denotes a family of formulations in which textual signals interact with scene representations across multiple viewpoints, spatial descriptors, or local scene observations so that semantics and geometry remain mutually consistent. In the cited literature, the term spans language-conditioned human–scene motion synthesis with dual local voxels, text-to-3D hand–object interaction via generated multi-view observations, view-specific text-to-3D generation, multi-view 3D scene editing, structured indoor reconstruction from multi-view captures, controllable indoor novel view synthesis, sparse-view instruction-based editing, 3D visual grounding, and scene-to-language parsing (Jiang et al., 2024, Hao et al., 10 Jun 2026, Yan et al., 2024, Wu et al., 2024, Xu et al., 26 Jun 2026, Fang et al., 3 Dec 2025, Gilo et al., 18 Nov 2025, Huang et al., 15 Jul 2025, Li et al., 20 Sep 2025). Across these formulations, the recurring objective is not merely to attach text to images, but to bind text to a scene model whose interpretation remains stable when evidence is distributed over multiple views.

1. Problem formulations and research scope

The literature uses Multi-TSI to solve several distinct but structurally related problems. In motion synthesis, the task is to generate multi-stage human–scene interaction motions in 3D indoor environments from a single text instruction and a goal location, without user-provided waypoints or phase labels; the “multi-view flavor” arises from local scene perception at multiple locations per segment, specifically a Current Scene Voxel centered at the character’s current pelvis and a Predictive Scene Voxel near the forthcoming sub-goal or target object (Jiang et al., 2024). In text-to-3D hand–object interaction, the task is to generate coherent multi-view observations from text and then recover a unified 3D hand mesh and object mesh that preserve language semantics, cross-view consistency, object geometry, articulated hand shape, and physically plausible contact (Hao et al., 10 Jun 2026). In 3D visual grounding, Multi-TSI becomes a cross-modal fusion problem in which viewpoint semantics must be injected into both language and scene streams to resolve perspective shifts and multi-anchor ambiguity (Huang et al., 15 Jul 2025).

A second cluster of work treats Multi-TSI as multi-view generation or editing. DreamView controls appearance differently from multiple viewpoints via an overall prompt $T$ and a set of view-specific prompts $\{T_v\}$ while maintaining global coherence (Yan et al., 2024). GaussCtrl edits a 3D Gaussian Splatting scene from a single text prompt by jointly editing multiple rendered views under depth conditioning and latent alignment, instead of iteratively editing one view at a time (Wu et al., 2024). MVRoom grounds user intent in a coarse 3D layout and propagates it across multiple camera viewpoints through a multi-view diffusion pipeline with layout-aware epipolar attention (Fang et al., 3 Dec 2025). InstructMix2Mix addresses sparse-input multi-view editing by distilling a frozen 2D instruction editor into a pretrained multi-view diffusion model that already embodies a data-driven 3D prior (Gilo et al., 18 Nov 2025).

A third cluster uses Multi-TSI for reconstruction and scene understanding. ReScene threads multi-view geometry throughout reconstruction and lets language-derived relational cues from a vision-LLM steer a confidence-weighted assembly process (Xu et al., 26 Jun 2026). Text-Scene converts multi-view 3D observations into textual scene graphs and summaries that preserve object attributes and spatial relations for downstream reasoning by multimodal LLMs (Li et al., 20 Sep 2025). Earlier work such as SceneScape already combined text-conditioned image synthesis with geometry accumulation across long camera trajectories, using a unified mesh as the interaction substrate between text and views (Fridman et al., 2023).

Paper	Setting	Multi-TSI mechanism
(Jiang et al., 2024)	Human–scene motion synthesis	Current + Predictive scene voxels
(Hao et al., 10 Jun 2026)	Text-to-3D hand–object interaction	Stacked multi-view VQ tokens
(Yan et al., 2024)	View-specific text-to-3D	Overall + view-specific prompts
(Wu et al., 2024)	3DGS scene editing	Depth-conditioned joint multi-view editing
(Xu et al., 26 Jun 2026)	Structured scene reconstruction	Multi-frame VLM relation fusion
(Fang et al., 3 Dec 2025)	Indoor NVS / text-to-scene	Layout-aware epipolar attention
(Gilo et al., 18 Nov 2025)	Sparse-view instruction editing	Multi-view diffusion personalization
(Huang et al., 15 Jul 2025)	3D visual grounding	Cross-modal Consistent View Tokens

2. Representations of views, scenes, and textual context

A defining feature of Multi-TSI is that “view” is not always literal camera imagery. In autonomous character–scene interaction synthesis, the scene is represented by two occupancy voxels, $$V_{\text{cur}}, V_{\text{pred}} \in \{0,1\}^{32\times32\times32}$$, extracted around the pelvis and around a predictive location; these are concatenated along channels to form a $64\times32\times32$ tensor, processed by a 6-layer ViT with 16 heads into a 512-d scene token. The text stream uses a CLIP text embedding projected from 768-d to 512-d, summed with a 512-d sinusoidal frame embedding, and then passed through an MLP to produce the final text token $V_{\text{emb}}$ (Jiang et al., 2024). This representation makes temporal progress an explicit part of language conditioning and treats local scene context at the current and near-future locations as two spatially distinct viewpoints.

Several systems instead encode all views jointly. TextHOI-3D stacks fixed-camera RGB observations as

$$X=\mathrm{Concat}(I_1,\ldots,I_N)\in\mathbb{R}^{H\times W\times 3N},$$

then learns a VQ-VAE over the stacked tensor so that each token represents a multi-view scene unit rather than an independent per-view patch. Its reported configuration uses $N=6$ views, $256\times256$ resolution, a $16\times16$ latent grid, codebook size $K=4096$, and code dimension 32 (Hao et al., 10 Jun 2026). MVRoom likewise uses explicitly view-aligned conditioning planes $\{T_v\}$0 rendered from a coarse 3D layout for each camera, with multi-layer semantics and depth capturing occluded structure, local spatial embeddings encoding object-level orientation and surface identity, and global spatial embeddings encoding absolute 3D positions (Fang et al., 3 Dec 2025).

View-specific textual conditioning is handled differently across models. DreamView represents the object with an overall prompt $\{T_v\}$1 and a set of view-specific prompts $\{T_v\}$2, and supplies both token embeddings and CLS embeddings from CLIP for each (Yan et al., 2024). ViewSRD introduces a shared set of learnable Cross-modal Consistent View Tokens (CCVTs),

$\{T_v\}$3

which are injected into both the text stream and the scene stream (Huang et al., 15 Jul 2025). SeMv-3D replaces image- or voxel-centric scene encoding with three orthogonal feature planes, $\{T_v\}$4, so that any 3D point can be projected to all three planes and fused into a triplane feature (Cai et al., 2024). ReScene and Text-Scene use yet another representational choice: confidence-weighted scene graphs whose nodes are reconstructed instances and whose edges are relation predictions fused across frames or expressed as textual relation triplets (Xu et al., 26 Jun 2026, Li et al., 20 Sep 2025).

These representation choices indicate that Multi-TSI is not tied to one data modality. It can be instantiated with occupancy voxels, stacked RGB tensors, triplanes, object tokens, scene graphs, or serialized textual descriptions, provided that the representation preserves correspondences across views and exposes a pathway for textual control.

3. Architectural mechanisms for textual–scene fusion

Most Multi-TSI systems use attention or diffusion as the main interaction mechanism, but they differ in how views are coupled. In the motion-synthesis setting, the next motion segment $\{T_v\}$5 is generated by a Transformer-based DDPM conditioned on language, scene, and stage-specific goals, while an autonomous scheduler predicts stage boundaries from motion and text:

$\{T_v\}$6

A simple threshold rule $\{T_v\}$7 triggers a stage switch, enabling end-to-end synthesis of sequences such as “walk to sofa → sit down → pick up remote → turn on TV” (Jiang et al., 2024). TeSMo, by contrast, is explicitly view-agnostic: it uses a scene-aware branch conditioned on a 2D floor map for navigation and an object-aware branch conditioned on Basis Point Sets for sitting and standing interactions, while its extension path to Multi-TSI is described through multi-view reconstruction, reprojection guidance, and cross-view feature fusion rather than native multi-view conditioning (Yi et al., 2024).

DreamView operationalizes Multi-TSI through collaborative text guidance injection inside a diffusion UNet. At each block, image features are compared to overall and view-specific CLS tokens; the module injects either overall text tokens or view-specific text tokens depending on the margin test $\{T_v\}$8. Cross-attention is then computed with the selected token set, so the model can favor global consistency in some layers and view-specific customization in others (Yan et al., 2024). MVRoom embeds a different inductive bias: cross-view attention is restricted to layout-supported epipolar segments. For a pixel $\{T_v\}$9 in view $$V_{\text{cur}}, V_{\text{pred}} \in \{0,1\}^{32\times32\times32}$$0, only keys and values from positions lying on valid epipolar segments in reference view $$V_{\text{cur}}, V_{\text{pred}} \in \{0,1\}^{32\times32\times32}$$1 are attended, producing layout-aware epipolar attention rather than unconstrained cross-view mixing (Fang et al., 3 Dec 2025).

Other systems separate semantic generation from geometric recovery. TextHOI-3D first predicts a multi-scale pyramid of multi-view VQ tokens with a CLIP-conditioned visual autoregressive model that combines global AdaLN modulation and local cross-attention, then recovers a unified 3D hand–object mesh by minimizing

$$V_{\text{cur}}, V_{\text{pred}} \in \{0,1\}^{32\times32\times32}$$2

with explicit contact and SDF-based anti-penetration terms (Hao et al., 10 Jun 2026). GaussCtrl similarly uses a two-step strategy: depth-conditioned diffusion edits multiple rendered views, and a shared 3DGS scene is optimized to fit all edited views jointly, while latent codes across views are aligned by self- and cross-view attention (Wu et al., 2024).

ViewSRD makes viewpoint semantics explicit in both modalities. The textual module reweights CCVTs by averaged cosine similarity between sentence features and view tokens, then applies cross-attention

$$V_{\text{cur}}, V_{\text{pred}} \in \{0,1\}^{32\times32\times32}$$3

while the scene module concatenates the corresponding view token $$V_{\text{cur}}, V_{\text{pred}} \in \{0,1\}^{32\times32\times32}$$4 to object features in each view and refines them with per-view Transformer layers (Huang et al., 15 Jul 2025). InstructMix2Mix couples views differently again: its teacher-side Random Cross-View Attention forces all views to attend to a randomly chosen key frame,

$$V_{\text{cur}}, V_{\text{pred}} \in \{0,1\}^{32\times32\times32}$$5

so that multi-view guidance during SDS personalization is anchored to a common reference without the cost of full extended attention (Gilo et al., 18 Nov 2025). ReScene’s fusion mechanism is graph-based rather than token-based: candidate relation edges are scored by a weighted combination of multi-frame votes, geometric priors, and category priors,

$$V_{\text{cur}}, V_{\text{pred}} \in \{0,1\}^{32\times32\times32}$$6

before being compiled into physically constrained attachment operations (Xu et al., 26 Jun 2026).

4. Training data, supervision, and optimization regimes

The data bottleneck is repeatedly emphasized. TeSMo states that previous text-to-motion methods focus on characters in isolation “due to the limited availability of datasets that include motion, text descriptions, and interactive scenes,” and addresses this by embedding annotated navigation and interaction motions within scenes (Yi et al., 2024). Text-Scene identifies the absence of large-scale 3D vision-language datasets as a central obstacle and responds by automatically parsing scenes into language rather than relying on human-authored 3D scene captions (Li et al., 20 Sep 2025).

Dedicated datasets and synthetic pipelines are therefore central to Multi-TSI research. The LINGO dataset provides 16 hours of MoCap in 120 indoor scenes across 40 motion types, with 20 everyday object categories, VICON capture at 30 FPS, precise language descriptions, GPT-4 augmentation, and a scene split by room type in a 4:1 ratio without overlap (Jiang et al., 2024). TextHOI-3D trains its stacked multi-view representation on HO3D-derived renderings, using 16,291 frames split into 14,662 train and 1,629 validation examples, with a fixed six-camera orbit (Hao et al., 10 Jun 2026). DreamView constructs a rendered dataset from Objaverse with approximately 435k assets and approximately 14M images, uses BLIP-2 to produce view captions, and merges 32 view-specific texts into an overall text via GPT-4 (Yan et al., 2024). MVRoom is trained on 3D-FRONT/3D-FUTURE rooms filtered to 6,287 scenes and more than 4 million rendered views at $$V_{\text{cur}}, V_{\text{pred}} \in \{0,1\}^{32\times32\times32}$$7 resolution (Fang et al., 3 Dec 2025).

Optimization regimes are correspondingly heterogeneous. The motion diffusion module in autonomous character–scene interaction synthesis uses $$V_{\text{cur}}, V_{\text{pred}} \in \{0,1\}^{32\times32\times32}$$8 diffusion steps, Adam, learning rate $$V_{\text{cur}}, V_{\text{pred}} \in \{0,1\}^{32\times32\times32}$$9, 500 epochs, batch size 1024, and 4 NVIDIA A100 GPUs, while its scheduler is trained for 5 epochs with a 3-layer, 8-head Transformer encoder (Jiang et al., 2024). TextHOI-3D trains the VQ-VAE with AdamW at $64\times32\times32$0 for 100 epochs on 3 Quadro RTX 6000 GPUs and then trains the CLIP-conditioned VAR with teacher forcing over scales (Hao et al., 10 Jun 2026). SeMv-3D uses a three-stage regimen built on Stable Diffusion 2.1: Object Retention for 150k steps at $64\times32\times32$1, Triplane Orthogonalization for 60k steps at $64\times32\times32$2, and Semantic-aligned View Synthesizer training for 100k steps at $64\times32\times32$3, all with AdamW on eight NVIDIA A6000 GPUs (Cai et al., 2024). SceneScape instead relies on online test-time training, finetuning MiDaS-DPT Large for 300 epochs per frame and the latent diffusion decoder for 100 epochs per frame, then resetting both to their pretrained weights after each frame (Fridman et al., 2023).

The supervision signals used in these works are also notably varied. They include DDPM noise-prediction losses for motion or image diffusion (Jiang et al., 2024, Yan et al., 2024, Fang et al., 3 Dec 2025); VQ reconstruction and commitment losses (Hao et al., 10 Jun 2026); rendering reconstruction losses with RGB, mask, depth, and LPIPS terms (Cai et al., 2024); relation-aware assembly energies and depenetration passes (Xu et al., 26 Jun 2026); CLIP-based selection or ranking rather than direct supervision (Yan et al., 2024, Li et al., 20 Sep 2025); and CLIP-based evaluation metrics for sparse-view editing rather than explicit geometric losses (Gilo et al., 18 Nov 2025). This diversity reflects a broader pattern: Multi-TSI is supervised at whatever level—tokens, rendered views, motion segments, meshes, or scene graphs—most directly exposes cross-view consistency.

5. Evaluation protocols and empirical findings

Empirical evaluation in Multi-TSI is task-specific, but nearly all papers measure some combination of semantic alignment, geometric consistency, and physical plausibility. In multi-stage motion synthesis, the autonomous character–scene interaction framework reports improvements over TRUMANS in interactive motion quality, with FID $64\times32\times32$4 versus $64\times32\times32$5, Precision $64\times32\times32$6 versus $64\times32\times32$7, Recall $64\times32\times32$8 versus $64\times32\times32$9, and F1 $V_{\text{emb}}$0 versus $V_{\text{emb}}$1; in cluttered locomotion it reports lower scene penetration and foot sliding, and in object reaching it reports error distance $V_{\text{emb}}$2 versus $V_{\text{emb}}$3 for GOAL (Jiang et al., 2024). TeSMo likewise reports strong scene-aware performance: navigation root position error $V_{\text{emb}}$4 m, orientation error $V_{\text{emb}}$5 rad, height error $V_{\text{emb}}$6 m, collision approximately $V_{\text{emb}}$7, and for interaction versus DIMOS, position error $V_{\text{emb}}$8 versus $V_{\text{emb}}$9 m and penetration value $$X=\mathrm{Concat}(I_1,\ldots,I_N)\in\mathbb{R}^{H\times W\times 3N},$$0 versus $$X=\mathrm{Concat}(I_1,\ldots,I_N)\in\mathbb{R}^{H\times W\times 3N},$$1 m (Yi et al., 2024).

In explicit multi-view generation and recovery, the quantitative effect of multiple views is often large. TextHOI-3D reports that moving from single-view to six-view recovery reduces object Chamfer Distance from $$X=\mathrm{Concat}(I_1,\ldots,I_N)\in\mathbb{R}^{H\times W\times 3N},$$2 mm to $$X=\mathrm{Concat}(I_1,\ldots,I_N)\in\mathbb{R}^{H\times W\times 3N},$$3 mm, increases F@5 from $$X=\mathrm{Concat}(I_1,\ldots,I_N)\in\mathbb{R}^{H\times W\times 3N},$$4 to $$X=\mathrm{Concat}(I_1,\ldots,I_N)\in\mathbb{R}^{H\times W\times 3N},$$5, reduces MPJPE from $$X=\mathrm{Concat}(I_1,\ldots,I_N)\in\mathbb{R}^{H\times W\times 3N},$$6 mm to $$X=\mathrm{Concat}(I_1,\ldots,I_N)\in\mathbb{R}^{H\times W\times 3N},$$7 mm, and lowers penetration volume from $$X=\mathrm{Concat}(I_1,\ldots,I_N)\in\mathbb{R}^{H\times W\times 3N},$$8 to $$X=\mathrm{Concat}(I_1,\ldots,I_N)\in\mathbb{R}^{H\times W\times 3N},$$9 (Hao et al., 10 Jun 2026). DreamView-2D improves the balance between consistency and customization relative to SD-v2.1 and MVDream, with reported validation averages of CLIP overall score $N=6$0, CLIP view-specific score $N=6$1, CLIP image-image score $N=6$2, and Inception Score $N=6$3 (Yan et al., 2024). SeMv-3D reports CLIPScore $N=6$4, Aesthetic Score $N=6$5, arbitrary-view synthesis in one step, and user-study preferences of $N=6$6 overall, $N=6$7 for semantic consistency, and $N=6$8 for multi-view consistency (Cai et al., 2024).

In scene-centric settings, geometric and perceptual metrics dominate. ReScene reports on a ScanNet subset that it achieves CD $N=6$9, F@5 $256\times256$0, NC $256\times256$1, PSNR $256\times256$2, SSIM $256\times256$3, and LPIPS $256\times256$4; the paper explicitly states a 17% reduction in Chamfer Distance and 26% in LPIPS over the strongest prior baseline, while runtime is 36.3 minutes versus 375.5 minutes for DRAWER (Xu et al., 26 Jun 2026). MVRoom reports that layout-aware epipolar attention reaches PSNR $256\times256$5 and SSIM $256\times256$6, outperforming MVDiffusion at PSNR $256\times256$7 and SSIM $256\times256$8, and its user study reports perceptual quality $256\times256$9, 3D structure consistency $16\times16$0, and layout plausibility $16\times16$1 (Fang et al., 3 Dec 2025). SceneScape evaluates long-horizon text-driven scene generation with COLMAP-based depth consistency, reprojection error, pose accuracy, AMT 2AFC, and CLIP aesthetic score, and reports that its unified mesh and online finetuning substantially improve 3D-consistency metrics relative to VideoFusion and GEN-1 (Fridman et al., 2023).

In sparse-view editing and grounding, the evaluation emphasizes cross-view alignment under textual control. InstructMix2Mix reports the best CLIP Directional Consistency among compared methods at $16\times16$2, versus $16\times16$3 for Instruct-NeRF2NeRF, $16\times16$4 for Instruct-GS2GS, $16\times16$5 for Text2Video-Zero, and $16\times16$6 for DGE; it also reports a human study favoring I-Mix2Mix with a 75% scene win rate over DGE (Gilo et al., 18 Nov 2025). ViewSRD reports 69.9 overall accuracy on Nr3D versus 64.4 for CoT3DRef, with a +6.7 improvement on the view-dependent split, and its ablations show that removing CCVTs causes the largest drop, from 69.9 to 62.2 (Huang et al., 15 Jul 2025). Text-Scene reports ScanRefer [email protected]/0.5 of 64.5/59.4, Multi3DRefer [email protected] of 60.7, ScanQA CIDEr/EM of 93.7/23.4, SQA3D EM of 61.2, and InPlan3D planning performance of GAcc 47.23% and TAcc 65.91% (Li et al., 20 Sep 2025).

6. Misconceptions, limitations, and open directions

A common misconception is that Multi-TSI always requires explicit multi-camera RGB input. The literature does not support that narrow definition. Autonomous character–scene interaction synthesis is described as having a “multi-view flavor” because it fuses two spatially distinct local voxels per segment rather than two literal cameras (Jiang et al., 2024). TeSMo is explicitly “view-agnostic” in its native form, yet is presented as a clear foundation for Multi-TSI extensions based on synchronized multi-view sensing, reprojection consistency, and 3D scene fields (Yi et al., 2024). Conversely, several later systems are genuinely multi-camera or sparse-view by construction, including TextHOI-3D, MVRoom, ReScene, and InstructMix2Mix (Hao et al., 10 Jun 2026, Fang et al., 3 Dec 2025, Xu et al., 26 Jun 2026, Gilo et al., 18 Nov 2025).

A second misconception is that cross-view consistency is always enforced by an explicit consistency loss. Some works do so through depth constraints, multi-view joint optimization, or epipolar masking (Hao et al., 10 Jun 2026, Wu et al., 2024, Fang et al., 3 Dec 2025). Others rely more on shared latent structures or architectural priors: DreamView uses shared overall text and expanded attention across views, while ViewSRD explicitly states that it does not introduce a separate cross-view consistency loss, instead inducing consistency through shared CCVTs and cross-modal interaction (Yan et al., 2024, Huang et al., 15 Jul 2025). This suggests that Multi-TSI is as much an architectural design problem as a loss-design problem.

The limitations reported across papers are consistent. Fine-grained manipulation remains difficult: the motion-synthesis framework focuses on body-level motions and does not model fine-grained hand manipulation or facial expressions (Jiang et al., 2024). TextHOI-3D notes open problems in semantic fidelity for complex prompts, robustness of occlusion handling and inpainting, dependence on fixed camera rigs, and scalability to more views and scene complexity (Hao et al., 10 Jun 2026). ReScene is currently limited to support and attachment relations and assumes dominant planar floors and walls (Xu et al., 26 Jun 2026). ViewSRD reports diminishing returns when the number of views becomes large, with four views offering a better trade-off than eight in the reported setting (Huang et al., 15 Jul 2025). SceneScape struggles with extreme depth discontinuities, long-horizon error accumulation, and dynamic content (Fridman et al., 2023). Text-Scene remains sensitive to reconstruction and instance segmentation quality and does not explicitly model support, containment, or affordances (Li et al., 20 Sep 2025).

Future work in the cited literature converges on a small set of directions. One is richer multi-view fusion: attention-weighted pooling of multiple scene descriptors, cross-view feature alignment, and learned view-invariant tokens are all proposed extensions (Jiang et al., 2024, Huang et al., 15 Jul 2025). Another is stronger geometry-aware priors, including 3D neural fields, occupancy networks, and physics-aware contact constraints (Jiang et al., 2024, Hao et al., 10 Jun 2026). A third is temporal or dynamic Multi-TSI: DreamView explicitly lists extension to 4D with consistent per-view textual constraints over time, MVRoom points to video diffusion backbones, and SceneScape already demonstrates that persistent 3D memory can stabilize long trajectories (Yan et al., 2024, Fang et al., 3 Dec 2025, Fridman et al., 2023). A plausible implication is that future Multi-TSI systems will increasingly combine three ingredients already present in separate lines of work: explicit 3D scene memory, text-conditioned multi-view generation, and physically grounded cross-view verification.