GS-VLA: Plug-and-Play Viewpoint Canonicalization for Frozen VLA Policies via Gaussian Splatting

Abstract: This paper proposes a lightweight, plug-and-play framework that improves robustness to viewpoint shifts in Vision-Language-Action (VLA) policies without policy retraining. To our knowledge, this is the first approach to directly leverage 3D Gaussian-based novel-view synthesis for observation-space adaptation in VLA policies. Current VLA performance relies on the implicit assumption that training and deployment camera configurations are identical. Our experiments show that even a small displacement of the camera mount can reduce the success rate on the LIBERO benchmark from about 90% to about 10% in the worst case. Prior approaches, such as large-scale fine-tuning or generative data augmentation, are computationally expensive and risk catastrophic forgetting. To address this, viewpoint shifts are reformulated as a localized novel-view synthesis problem. Under a Locality assumption, that camera perturbations remain within a small bounded region relative to the workspace, viewpoint normalization reduces to a scene- and policy-independent disocclusion task. Our work implements this idea with a 4M-parameter 3D-Gaussian canonicalizer prepended to a frozen VLA policy. Without modifying policy weights, GS-VLA improves performance across three orthogonal axes: (1) Policy architectures, (2) Unseen task suites, and (3) Perturbation scales. These results show that a lightweight visual module can recover a large fraction of the performance lost under viewpoint shift, without policy retraining.

• The paper introduces a 4-million-parameter Gaussian-splatting canonicalizer that transforms shifted camera observations into a policy’s training viewpoint, improving success by up to 79.6 percentage points without retraining the VLA policy.
• GS-VLA recovers a four-suite mean gain of 39.8 percentage points at approximately 13 milliseconds per frame, while remaining effective across different policies, task suites, and camera translations up to 200 centimeters.
• The method depends on bounded viewpoint changes and calibrated metric depth, tolerates roughly 3 centimeters and 3 degrees of extrinsic noise, and remains unvalidated on real robots beyond LIBERO simulation.

Problem and motivation

Vision-Language-Action (VLA) models achieve high success rates on manipulation benchmarks such as LIBERO, but this performance rests on the implicit assumption that the deployment camera configuration matches the training configuration. The authors quantify the severity of this fragility: on their worst LIBERO configuration, a camera displacement reduces the success rate of $\pi_{0.5}$ from $92.4\%$ to $9.3\%$ — a collapse of more than 80 percentage points from a perturbation that, in practice, amounts to a few centimeters of mount drift.

The standard remedies — large-scale data collection, fine-tuning, or generative data augmentation — are computationally expensive and carry the risk of catastrophic forgetting. The paper argues instead that viewpoint-induced failure does not reflect a deficit in high-level reasoning or control, but rather an observation-space mismatch. This motivates a reframing of the question: rather than encoding viewpoint robustness in policy parameters, can it be restored by adapting the observation space itself, leaving the policy frozen?

The Locality reduction

The central conceptual contribution is the Locality assumption: deployment-time camera perturbations remain within a bounded neighborhood of the canonical pose (formally, a ball in SE(3) with translation tolerance $\varepsilon_t$ and rotation tolerance $\varepsilon_R$, with $\varepsilon_R \approx \varepsilon_t / d_{\min}$). Under this assumption, the newly exposed (disoccluded) region of the canonical view has fractional area $\eta(\rho)$ that grows only linearly to leading order in the normalized perturbation magnitude $\rho$, with separate coefficients for the parallax (translation) and frame (rotation) contributions. The consequence is that viewpoint canonicalization is not a general novel-view synthesis problem: most pixels can be mapped to the canonical frame by closed-form depth warping, and learning is required only for an $O(\rho)$-thin band along depth-discontinuity curves. This reduces the problem to a scene- and policy-independent disocclusion task, which is the theoretical basis for the module's small capacity requirement.

Architecture and training

The canonicalizer $f_\phi$ maps a source RGB image $92.4\%$0, a per-pixel metric depth map $92.4\%$1, and the camera parameters $92.4\%$2 to a canonical-view image $92.4\%$3, which is then passed to the frozen policy. It is implemented as a symmetric U-Net (base width $92.4\%$4) with only 4,047,824 parameters, FiLM-modulated by a pose-pair embedding. At each source pixel, the decoder predicts a 14-dimensional Gaussian descriptor — position residual, anisotropic log-scale, orientation quaternion, opacity, and color residual. Gaussian centers are anchored at the world-space back-projection of each source pixel, with displacement clamped to 10% of local depth. The primitives are composed at the canonical pose via gsplat's differentiable $92.4\%$5-blender, which resolves occlusion and fills the disocclusion band without a separate inpainting head. All residual heads are zero-initialized so the module starts exactly at the geometric warp.

Training uses only the standard 3DGS loss (0.8 : 0.2 $92.4\%$6/SSIM) on 50k source-to-canonical pairs from a single suite (libero_spatial) at $92.4\%$7\,cm, converging in roughly 5 hours on a single RTX A5000. Inference costs approximately 13 ms per frame (~75 Hz). The module is 42× smaller than the concurrent AnyCamVLA and ~840× smaller than fine-tuning $92.4\%$8 itself.

Main results

A single checkpoint, trained once on libero_spatial at $92.4\%$9\,cm, is evaluated without any axis-specific retraining across three orthogonal axes. The headline numbers are substantial:

Axis	Setting	No canon	GS-VLA	$9.3\%$0 (pp)
Cross-policy	XVLA, spatial	1.4	81.0	+79.6
Cross-policy	OpenVLA-OFT (7B), object	19.8	81.6	+61.8
Cross-suite	$9.3\%$1, libero_10	9.3	72.1	+62.8
Cross-suite	$9.3\%$2, spatial	42.6	86.8	+44.2
Scale	$9.3\%$3, $9.3\%$4\,cm	35.7	78.5	+42.8
Scale	$9.3\%$5, $9.3\%$6\,cm	85.2	88.0	+2.8

The four-suite mean gain is $9.3\%$7\,pp. Two patterns are notable. First, the magnitude of recovery anti-correlates with the policy's intrinsic robustness — the most fragile policies (XVLA, OpenVLA-OFT on object) benefit most, which is consistent with the Locality account that more out-of-distribution pixels leave more room for canonicalization. Second, the largest cross-suite gain occurs on the long-horizon libero_10 suite, which the authors attribute to the locality property limiting per-frame inpainted area and thereby suppressing frame-to-frame error accumulation over long trajectories.

In the perturbation sweep over $9.3\%$8\,cm, the unprotected baseline drops 49.5 pp while GS-VLA drops only 9.5 pp, with $9.3\%$9 growing approximately linearly within the Locality bound and matching the predicted $\varepsilon_t$0 ratios to within 11%. Gains taper at 200 cm as $\varepsilon_t$1 approaches the Locality boundary. Under joint translation–rotation perturbations ($\varepsilon_t$2\,cm, yaw $\varepsilon_t$3), $\varepsilon_t$4 remains in the $\varepsilon_t$5\,pp range; larger rotations narrow the gain, which the authors attribute to undersampling of large rotations in training — a data-side issue rather than an architectural one.

Deployment realism: calibration noise

Because the canonicalizer requires the deployment pose as input, the paper stress-tests sensitivity to extrinsic calibration error. The module degrades gracefully up to roughly 3 cm and 3° of pose noise (mean drop ≤ 5 pp), but at 5 cm / 5° the long-horizon libero_10 suite suffers a $\varepsilon_t$6\,pp drop, marking the practical calibration tolerance. Intrinsics errors are far less damaging: a full focal-length sweep of $\varepsilon_t$7 produces at most 2.8 pp degradation, roughly 4× smaller than the extrinsic budget. This asymmetry is a useful deployment guideline: extrinsic calibration, not intrinsics, is the binding constraint.

Mechanism validation

The appendix provides direct empirical support for the Locality proposition rather than relying only on end-task numbers. The disoccluded fraction $\varepsilon_t$8 is linear in $\varepsilon_t$9 with $\varepsilon_R$0 up to $\varepsilon_R$1; the rotation-induced slope is essentially scene-invariant (1.084 vs. 1.089 across two suites, a 0.4% discrepancy) while the translation slope varies by ~60%, matching Corollary 1's prediction that only the parallax term depends on silhouette geometry. Disoccluded pixels lie 5–7 px from the nearest depth edge versus 27–33 px for uniform-random pixels — a ~5× tighter localization consistent with the silhouette-hugging geometry. The full $\varepsilon_R$2 chain closes numerically with $\varepsilon_R$3, with the $\varepsilon_R$4 link the loosest.

Ablations

Three ablations support the design choices. Capacity: doubling the base width to 15M parameters raises reconstruction PSNR from 26.84 to 28.4 dB but lowers SR to 85.2%, demonstrating that beyond ~4M parameters reconstruction quality and policy success decouple — the policy does not consume the extra texture detail. Data: training-set size saturates by roughly 10k–25k pairs, consistent with the low intrinsic learning complexity that Locality predicts. Depth substitution: replacing ground-truth metric depth with zero-shot DepthAnything V2 collapses SR to 44.5%, because per-image analytical calibration cannot recover the absolute scale required by the depth-bounded center clamp; a short fine-tune of the depth backbone (3 epochs, 50k frames) recovers to 71.7%. Notably, larger depth encoders, a residual refinement head (which collapsed to zero), and end-to-end depth gradients all failed to close the remaining ~15 pp gap, which the authors read as a structural distribution mismatch and splat-module information bottleneck rather than a depth-accuracy limitation.

Limitations

The authors are explicit about three boundaries. First, the canonicalizer consumes a single view with calibrated metric depth, so scene content entirely unobserved by the source camera cannot be recovered; a multi-view extension (e.g., fusing a wrist camera) is proposed but not implemented. Second, the rotation budget could not be cleanly swept beyond the joint cells reported, because large rotations combined with large translations frequently rotate the camera off the workspace entirely — a property of the perturbation distribution rather than the canonicalizer. Third, and most significantly, all experiments are conducted in the LIBERO simulator; no real-robot validation is reported, and the method additionally requires an external metric-depth source that real rigs may not provide. The simulator results should accordingly be read as preliminary evidence rather than a demonstration of real-world robustness. The method also presupposes that deployment perturbations fall within the Locality ball; behavior at or beyond the boundary ($\varepsilon_R$5) degrades, as seen at $\varepsilon_R$6\,cm.

Conclusion

GS-VLA demonstrates that viewpoint robustness for frozen VLA policies can be achieved in the observation space with a 4M-parameter, 13-ms Gaussian-splatting canonicalizer, recovering a four-suite mean of $\varepsilon_R$7\,pp across four policies, four suites, and perturbations from 5 to 200 cm without any policy retraining. The paper's analytical contribution — the Locality reduction, validated at each link of the causal chain — explains why such a small module suffices and predicts where it will and will not help. The principal open questions left by the work are whether the gains transfer to real robots with imperfect depth sensing, whether a multi-view input can lift the single-view disocclusion ceiling, and whether the same locality-based template extends to other perturbation axes such as lighting and embodiment.