Plücker Ray Conditioning in Vision

• Plücker ray conditioning is a method that encodes per-pixel 3D viewing rays using 6D Plücker coordinates, capturing both direction and moment from camera parameters.
• Its integration in feed-forward view synthesis and view-invariant policy models enhances geometric awareness, leading to improved robustness under complex viewpoint variations.
• However, its dependence on the world-coordinate gauge results in fragility under global transformations, motivating the development of invariant alternatives like projective conditioning.

Plücker ray conditioning is a method for encoding camera parameters and per-pixel viewing geometry in computer vision models, particularly within view synthesis and vision-based policy learning systems. By representing each image pixel’s corresponding 3D viewing ray in Plücker coordinates—a 6-dimensional embedding capturing both direction and moment—this technique explicitly incorporates camera intrinsics and extrinsics at a fine spatial granularity. Plücker ray conditioning offers a principled means of imparting geometric awareness to neural architectures, facilitating improved robustness and generalization under complex viewpoint changes. However, its dependence on the world-coordinate gauge introduces fragility to certain global transformations, motivating alternative input parameterizations in recent research (Wu et al., 8 Jan 2026, Jiang et al., 2 Oct 2025).

1. Mathematical Foundation of Plücker Ray Representation

A ray in 3D space can be uniquely encoded as a pair $(d, m)$, where $d \in \mathbb{R}^3$ is the unit direction and $m \in \mathbb{R}^3$ the Plücker moment, given by $m = o \times d$ with $o$ any point on the line. All valid $(m,d)$ tuples inhabit the Klein quadric in projective space $\mathbb{P}^5$, representing the Grassmannian $G(1,3)$ of lines in $\mathbb{R}^3$.

Camera-centric construction proceeds by:

• Computing, for each pixel with coordinates $[u, v, 1]^\top$, a direction vector in the camera frame as $d \in \mathbb{R}^3$0;
• Normalizing $d \in \mathbb{R}^3$1 to unit length;
• Transforming to world frame: $d \in \mathbb{R}^3$2 for extrinsic rotation $d \in \mathbb{R}^3$3;
• Computing the camera center in world frame: $d \in \mathbb{R}^3$4 for translation $d \in \mathbb{R}^3$5;
• Defining the Plücker moment $d \in \mathbb{R}^3$6;
• Concatenating $d \in \mathbb{R}^3$7 and $d \in \mathbb{R}^3$8 into a per-pixel 6D embedding.

Stacking across an image forms a dense ray-map $d \in \mathbb{R}^3$9, aligned spatially with the corresponding RGB image (Jiang et al., 2 Oct 2025).

2. Integration into Modern Model Architectures

Plücker ray conditioning has been instantiated in two principal domains:

a. Feed-Forward View Synthesis Models

Large View Synthesis Models (LVSMs) and variants never pass target camera pose as a simple 6-DOF vector but instead encode it as an $m \in \mathbb{R}^3$0 Plücker ray-map. This map, patchified into $m \in \mathbb{R}^3$1 tokens, is linearly projected and concatenated with image tokens, forming the geometric inputs to a transformer-based model. The resultant tensor, $m \in \mathbb{R}^3$2, serves as the only geometric cue apart from raw images (Wu et al., 8 Jan 2026).

b. View-Invariant Policy Learning

Policy networks in imitation learning systems incorporate Plücker ray-maps using either early fusion (where the vision encoder’s first layer ingests image and ray-map channels simultaneously) or late fusion (where the image is encoded separately and ray features are merged deeper in the network). Both approaches grant the policy explicit access to pixelwise camera extrinsics, critical for modulating attention to relevant scene regions and improving viewpoint generalization (Jiang et al., 2 Oct 2025).

3. Gauge Sensitivity and Fragility of Plücker Conditioning

A fundamental desideratum for view synthesis is gauge invariance—robustness to arbitrary reparametrizations of world coordinates under the SE(3) group (rotations and translations). However, under a global transformation $m \in \mathbb{R}^3$3, Plücker rays transform as

$m \in \mathbb{R}^3$4

introducing inhomogeneous, spatially varying perturbations in the ray-map that depend on both camera motion and pixel location (Wu et al., 8 Jan 2026).

Consequences include:

• Violation of geometric consistency under small SE(3) or Sim(3) perturbations (e.g., tiny translations, world scaling, focal length changes);
• Out-of-distribution behavior in model predictions under such perturbations;
• Learning inefficiency, as networks must allocate capacity to correct for nonuniform shifts in input distributions;
• Inability to generalize to unseen camera perturbations without extensive retraining or data augmentation.

Empirical results confirm that small gauge changes can substantially degrade rendering fidelity and policy success rates (Wu et al., 8 Jan 2026, Jiang et al., 2 Oct 2025).

4. Implementation Workflows and Algorithmic Details

The runtime construction and use of Plücker ray conditioning involve:

• For a given camera/intrinsic setup:
  • Derive camera-frame direction $m \in \mathbb{R}^3$7;
  • Normalize $m \in \mathbb{R}^3$8;
  • Transform to world frame $m \in \mathbb{R}^3$9;
  • Camera center $m = o \times d$0;
  • Moment $m = o \times d$1.
• Form $m = o \times d$2 as input.

Depending on architecture:

• Early fusion: Concatenate $m = o \times d$3 with image, pass as a $m = o \times d$4-channel tensor to the vision backbone.
• Late fusion: Pass $m = o \times d$5 and the image through parallel feature encoders, concatenate deep features, and process in the policy head.

All matrix operations—cross-product, inversion, and back-projection—are computationally negligible compared to deep model forward passes (Jiang et al., 2 Oct 2025).

5. Empirical Impact and Evaluation

Explicit Plücker ray conditioning provides significant, quantifiable improvements in performance across vision-driven tasks exposed to viewpoint or camera-parameter variation.

In view-invariant imitation learning, conditioning policies with per-pixel Plücker ray-maps yields improvements of +10–30 percentage points in task success rates compared to unconditioned baselines across multiple policy classes and manipulation tasks (e.g., “lift,” “push”; Table 1 (Jiang et al., 2 Oct 2025)). Performance gains are most notable in randomized, non-stationary scenes where inference of camera pose from static background cues fails. Additionally, random cropping of both images and ray-maps acts as effective data augmentation. Late fusion architectures outperform early fusion with pretrained backbones.

For feed-forward view synthesis, while Plücker conditioning enables minimal 3D bias models to exploit geometric information, its gauge-sensitivity results in notable degradation under global frame perturbations. For example, PSNR drops from 25.43 (projective cue) to 14.56 (Plücker-based LVSM) under world-scale shifts (Wu et al., 8 Jan 2026). This suggests that while Plücker conditioning is powerful for encoding pixelwise geometry, its utility is undermined by lack of gauge invariance.

6. Alternatives and Developments: Projective Conditioning

Recent research has addressed the fragility of Plücker ray conditioning by introducing projective conditioning (Wu et al., 8 Jan 2026). In this method, context-view images, combined with their estimated depth, are unprojected into a 3D point cloud and reprojected into the target view by a differentiable “UnProj” + “Rast” pipeline. The result, $m = o \times d$6, is a single $m = o \times d$7 image encapsulating relative geometry in a gauge-free quotient representation.

Empirically, projective conditioning yields:

• Robustness to all SE(3) and Sim(3) frame transforms by design;
• Superior fidelity and perceptual quality on modern view synthesis benchmarks;
• Dramatically improved consistency across camera intrinsic and extrinsic perturbations compared to Plücker-based conditioning.

Ablations confirm the core pipeline and use of masked autoencoder pretraining are central to these gains (Tables 4–5, Fig. 8 (Wu et al., 8 Jan 2026)). A plausible implication is that future neural geometric systems will increasingly favor quotient representations that are invariant by construction rather than attempting to learn invariance through data-driven correction.

7. Summary Table: Plücker Ray Conditioning in Context

Approach	Representation	Gauge Invariance	Empirical Robustness
Plücker Ray Conditioning	$m = o \times d$8	No	Sensitive to SE(3)/Sim(3)
Projective Conditioning	$m = o \times d$9	Yes	High under perturbations

Plücker ray conditioning constitutes a pivotal development in geometric input encoding for modern vision networks, providing precise per-pixel access to camera pose and orientation. Though highly expressive and lightweight, its practical limitations under gauge transformations have stimulated the adoption of intrinsically invariant alternatives in state-of-the-art architectures (Wu et al., 8 Jan 2026, Jiang et al., 2 Oct 2025).