---
title: 'Rays as Pixels: Rethinking Imaging Primitives'
url: https://www.emergentmind.com/topics/rays-as-pixels
type: topic
---

# Rays as Pixels: Rethinking Imaging Primitives

Searching arXiv for the cited papers to ground the article and confirm bibliographic details.
arXiv search query: "Rays as Pixels camera trajectories video diffusion 2604.09429"
“Rays as Pixels” denotes a family of formulations in which the primary discrete element is not a fixed screen pixel or a compact camera-parameter vector, but a ray, a ray bundle, or a direction-indexed sample. In this view, a detector sample may be treated as a line integral along a ray, a ray-traced image element, a per-patch Plücker ray, a cubemap texel indexed by world-space direction, a surface texel storing directional radiance, or a dense ray image (“raxel”). Across imaging, graphics, camera estimation, and reciprocal optics, the common move is to make ray geometry or directional response the native carrier of signal and then reconstruct images, poses, or wavefronts from that representation [1402.2426][1010.5534][2402.14817][2604.09429].

## 1. Conceptual scope and recurring representations

The literature uses the phrase in several technically distinct but structurally related ways. In tomography and multiview imaging, each detector sample is a projection on a ray. In adaptive ray tracing, each pixel is a ray-sampled region that can be recursively subdivided. In camera modeling, each image patch predicts a ray, so a camera becomes a dense field of rays rather than a single pose vector. In rasterization-oriented methods, ray responses are stored in textures or cubemaps and later read back like pixels. In optical hardware, a photon event or a pixel-sized diffractive element becomes the effective unit of spatial or wavefront sampling [1402.2426][1010.5534][2301.01719][1501.06825][2601.09022].

| Domain | Ray-like unit | Function |
|---|---|---|
| Tomography and multiview imaging | detector sample on a ray | line integral or attenuated projection |
| Adaptive image ray-tracing | adaptive ray/pixel cell | local refinement around prominent features |
| Radiance textures | \(n \times n\) bucket per texel | directional radiance lookup |
| Camera pose and video diffusion | per-patch ray or raxel | distributed camera representation |
| Texel splatting and Gaussian ray tracing | cubemap texel or camera ray | stable world sample or Gaussian traversal |
| X-ray and Fourier-optical devices | photon event or Fourier pixel | sub-pixel localization or reciprocal wavefront control |

A persistent misconception is that “rays as pixels” refers only to replacing screen pixels with traced rays at render time. The published usage is broader. It includes inverse problems in which a pixel is reinterpreted as a ray measurement, dense geometric representations in which each image patch stores a ray, and optical devices in which a local element senses or emits an entire directional or vectorial field rather than a scalar intensity [1402.2426][2402.14817][2601.09022].

## 2. Ray measurements as the basis of imaging and inverse problems

“Imaging with Rays: Microscopy, Medical Imaging, and Computer Vision” formulates a wide range of imaging systems as ray-based projection operators [1402.2426]. For a simple projection along the \(z\)-axis, it writes
\[
p(x,y) = \int f(x,z,y)\,dz,
\]
with Fourier-slice form
\[
\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).
\]
In 2D, the same idea appears as a Radon-style projection
\[
g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.
\]

Within this framework, CT is modeled by x-rays as rays through the object, transmission tomography becomes log-linear after taking logarithms of
\[
p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},
\]
PET/SPECT are written as attenuated emission projections, computational confocal microscopy is treated as a sheared ray-space sampling process, and a pinhole camera is represented as a collection of cone-beam-like projections. After discretization, these systems reduce to a matrix equation \(A x = b\), where each row of \(A\) corresponds to a ray measurement [1402.2426].

The same paper extends the ray framework to self-occluding opaque objects by jointly estimating brightness and attenuation. Its continuous forward model factorizes the object-dependent PSF as
\[
H(\mathbf r,\mathbf r') = P(\mathbf r,\mathbf r')\,Q(\mathbf r,\mathbf r'),
\]
and its discrete nonlinear model is
\[
\mathbf b = \mathbf C \,\exp\!\bigl(\mathbf E \log \mathbf x\bigr),
\]
with \(\mathbf x = (\mathbf a,\mathbf s)^T\) concatenating attenuation and brightness. Reconstruction is posed as a regularized nonlinear optimization problem with an explicit empty-space prior and a positivity floor \(\mathbf x \ge \epsilon\) [1402.2426].

This establishes a strict interpretation of the phrase: a pixel is not fundamentally a square on a detector plane, but a sample associated with a ray path and its accumulated interaction with the scene. A plausible implication is that once ray measurements are treated as the primitive, distinctions among medical imaging, microscopy, and multiview computer vision become largely distinctions in forward operators and regularization rather than in measurement ontology.

## 3. Adaptive and ray-centric rendering

In “Adaptive image ray-tracing for astrophysical simulations,” the image plane is no longer a uniformly fixed pixel grid; it is an adaptive hierarchy of ray-sampled cells [1010.5534]. Conventional ray tracing “first discretize[s] the plane of the sky into a uniform array of pixels and then follow[s] the path of a ray for each respective pixel.” AIR instead begins from a coarse image, traces rays for current pixels, computes a truncation-error estimate, refines flagged pixels, and iterates until the desired image resolution is reached. The refinement criterion is a modified second-derivative interpolation error estimate from Löhner (1987), with \(0 \le \xi < 1\), and refinement occurs when
\[
\xi_{ij} \ge \xi_{\rm crit}.
\]
For the test problem, \(\xi_{\rm crit} > 0.01\) works well, and for more complex images a starting point of \(\xi_{\rm crit} \simeq 0.5\) is suggested [1010.5534].

Its test case used a \(512 \times 512\) fixed image versus AIR with a \(32^2\) base image, 4 nested image refinement levels, and the same effective final resolution \(512^2\). The fixed image required 9604 s and 262144 pixels. AIR at \(\xi_{\rm crit}=0.8\) used 1339 s and 22748 pixels, with \(1\times 10^{-4}\) fractional error, summarized as about 1/7 the time, about 1/12 the number of pixels, and only 0.01% error in the integrated flux. The conclusion gives a typical speed-up factor of \(\sim 4\)–\(7\) and a pixel reduction factor of \(\sim 6\)–\(11\), with savings roughly the inverse of the image filling factor [1010.5534].

A different ray-to-pixel conversion appears in “Radiance Textures for Rasterizing Ray-Traced Data” [2301.01719]. There, “Each pixel of the model’s texture contains discrete radiance hemispherical map of size \(n \times n\), called ‘bucket’.” The object texture has size \(w \times h\), each texel is replaced by an \(n \times n\) radiance bucket, and the stored representation behaves like a larger texture of size \(w\cdot n \times h\cdot n\). The radiance is parameterized as \(L(u,v,\theta,\phi)\) or \(L'(u,v,x,y,z)\), incidence direction is mapped into a 2D bucket coordinate, and the fragment shader reconstructs color by directional lookup. The method explicitly trades storage for runtime cost: the computational footprint is that of simple diffuse-only graphics, while fidelity depends on bucket resolution and memory [2301.01719].

“Texel Splatting: Perspective-Stable 3D Pixel Art” turns each cubemap texel into a stable scene sample indexed by direction from a fixed probe origin [2603.14587]. Its premise is that orthographic camera snapping works because projection is linear and depth-independent, whereas perspective projection causes depth-dependent screen drift that no single snap can correct. The method therefore renders geometry into a cubemap from a fixed world-space origin and splats each texel back to the screen as a world-space quad. World position is reconstructed as
\[
\mathbf{p} = \mathbf{o} + \frac{d}{\|\mathbf{r}\|_\infty}\,\mathbf{r}.
\]
Cubemap indexing gives rotation invariance, and grid-snapping the origin gives translation invariance. The primary limitation is equally explicit: a fixed origin cannot see all geometry, so disocclusion at probe boundaries remains the open tradeoff [2603.14587].

“RaySplats: Ray Tracing based Gaussian Splatting” replaces projection-based 3D Gaussian Splatting with ray tracing through Gaussian confidence ellipsoids [2501.19196]. A ray \(r(t)=o+td\) is intersected with
\[
E_{\mu,\Sigma,Q} = \{x \in \mathbb{R}^3 : (x-\mu)^T \Sigma^{-1} (x-\mu)=Q\},
\]
and ray-wise color is composited as
\[
C = \sum_{i=1}^{N} c_i \alpha_i \prod_{j=1}^{i-1}(1-\alpha_j).
\]
Because Gaussian primitives become ray-intersectable objects, shadows, reflections, transparency, refraction, and mesh integration become natural parts of the same traversal process [2501.19196].

## 4. Cameras as dense ray fields

“Cameras as Rays: Pose Estimation via Ray Diffusion” argues against direct regression of a low-dimensional global camera parametrization \((R,T,K)\) and instead represents a camera as a set of per-patch rays \(\mathcal{R}=\{r_1,\ldots,r_m\}\), with each ray \(r_i \in \mathbb{R}^6\) tied to a known pixel location \(u_i\) [2402.14817]. Rays are represented in Plücker coordinates
\[
r = \langle d, m \rangle \in \mathbb{R}^6,
\]
with \(m = c \times d\). Given a camera,
\[
d = R^\top K^{-1} u, \qquad m = (-R^\top T) \times d.
\]
The reverse conversion recovers the camera center by least squares and then uses a homography, DLT, RQ decomposition, and \(T=-R^\top c\) to recover the conventional camera parameters [2402.14817].

Its regression model uses a pretrained frozen DINOv2 backbone, concatenates patch features with normalized pixel coordinates, and processes all \(N \cdot p^2\) tokens with a set-level transformer. A diffusion version denoises ray bundles directly in ray space with the same \(L_2\) reconstruction loss. On CO3Dv2, the ray representation outperforms direct \(R+T\) regression and pose-space diffusion. At 2 images on seen categories, the reported relative-rotation-within-\(15^\circ\) values are 49.1 for R+T Regression, 75.7 for PoseDiffusion, 81.8 for RelPose++, 88.8 for Ray Regression, and 91.8 for Ray Diffusion; on unseen categories they are 42.7, 63.2, 69.8, 79.0, and 83.5. An ablation further reports that \(16\times 16\) rays is best among the tested settings, while more rays increase compute [2402.14817].

“Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories” generalizes the same principle from pose estimation to joint generative modeling [2604.09429]. It learns the joint density
\[
p(z,r)
\]
over video latents \(z\) and ray or trajectory latents \(r\), rather than only \(p(z\mid r)\) or \(p(r\mid z)\). Cameras are encoded as dense ray pixels, or raxels. After canonicalization by
\[
P^{\text{rel}} = P_s^{-1} P_j,
\]
each pixel \(u\) yields a direction and origin
\[
d = R^{\text{rel}} \frac{K_j^{-1}\tilde{u}}{\|K_j^{-1}\tilde{u}\|_2}, \qquad o = T^{\text{rel}},
\]
and the raxel is defined as \(\text{raxel}(u)=d+o\). The point of this construction is architectural: raxels are dense, 3-channel, VAE-compatible, and spatially aligned with video tensors [2604.09429].

The model builds on a pretrained Wan 2.1 14B video diffusion transformer, uses Flow Matching with
\[
\mathcal{L}(\theta) = \mathbb{E}_{t,x_0,x_1} \left[ \left\| v_\theta(x_t,t) - u_t \right\|_2^2 + \lambda \left( 1 - \frac{\langle v_\theta(x_t,t), u_t \rangle}{\|v_\theta(x_t,t)\| \, \|u_t\|} \right) \right],
\]
and \(\lambda=0.5\), and couples video and ray branches through Decoupled Self-Cross Attention. A single trained model supports three tasks: predicting camera trajectories from video, jointly generating video and trajectory from sparse images, and generating video from sparse images along a target trajectory. It is evaluated by a closed-loop self-consistency test, and it reports that trajectory prediction requires far fewer denoising steps than video generation; best pose results occur at 2 diffusion steps, with mean Relative Rotation Accuracy @30 around 95.91 on RealEstate10K, 88.37 on DL3DV-140, and 93.51 on Tanks and Temples [2604.09429].

## 5. Event-based and reciprocal optical realizations

In “Sub-pixel resolution with color X-ray camera SLcam(R),” the effective imaging unit is a single X-ray photon event rather than a fixed CCD pixel [1501.06825]. The system combines a pnCCD detector with 48 \(\mu\)m pixel size and polycapillary optics. Because the detector is single-photon and energy-resolving, each event carries spatial and energy information. The absorbed photon creates a charge cloud,
\[
S(x,y) = \frac{Q}{2\pi \sigma_x \sigma_y} \exp\left( -\frac{(x-x_0)^2}{2\sigma_x^2} -\frac{(y-y_0)^2}{2\sigma_y^2} \right),
\]
whose distribution over neighboring pixels encodes sub-pixel position [1501.06825].

The algorithm uses intensity ratios
\[
d(I_1,I_2)=\frac{I_2-I_1}{I_2+I_1},
\]
defines \(d_x\) and \(d_y\) over a \(2\times 2\) box, and then maps ratio histograms to positions by
\[
x_0 = p\left(P_{d_x}(d_x)-\frac{1}{2}\right), \qquad
y_0 = p\left(P_{d_y}(d_y)-\frac{1}{2}\right).
\]
A central contribution is that the method uses all photon events, including pixel-center events, rather than only corner events. With 8:1 optics, one CCD pixel corresponds to a \(6 \times 6~\mu\text{m}^2\) area on the sample; \(2\times 2\) sub-pixel division resolves one additional stripe in the Au bar pattern, tilted structures can resolve \(5~\mu\text{m}\) or even \(4~\mu\text{m}\) bars, and overly fine grids such as \(9\times 9\) create artifacts, so the best useful division for a \(48\times 48~\mu\text{m}^2\) pnCCD pixel is approximately \(5\times 5\) [1501.06825].

“Fourier pixels for reciprocal light control” extends the concept from event localization to reciprocal control of optical wavefronts [2601.09022]. A Fourier pixel is a miniaturized diffractive element based on Fourier optics that can both generate and sense optical fields with control over amplitude, phase, and polarization. The implementation uses coherent surface plasmon polariton waves on silver, launched by source gratings and diffracted by shallow wavy microstructures. In the scalar case, the reference wave is
\[
r(x,y) = e^{i k_{\mathrm{spp}} x},
\]
the surface response is \(f(x,y)=r(x,y)t(x,y)\), and shallow-profile linearization yields
\[
h_p(x,y) \propto \operatorname{Im}\!\left\{ f(x,y)e^{-i k_{\mathrm{spp}} x} \right\}.
\]
For vectorial operation, the pixel uses a two-component field representation and a diagonal transparency matrix, allowing polarization-dependent outputs and full Stokes sensing through
\[
S_0 = 2(I_x+I_y), \quad
S_1 = 2(I_x-I_y), \quad
S_2 = I(0)-I(\pi), \quad
S_3 = I(3\pi/2)-I(\pi/2).
\]
Reported demonstrations include vortex beams with \(q=+1,+3,+5\), vector beams of order \(Q=+1\) and \(Q=+2\), a diffraction-limited focus with measured spot size around \(0.27~\mu\text{m}\), focusing efficiencies exceeding 40% power in to power out for visible wavelengths in the green/red range, and phase readout fluctuations limited by the setup to about \(0.07^\circ\) input-angle uncertainty [2601.09022].

## 6. Tradeoffs, limitations, and interpretive boundaries

The technical benefits of ray-native representations are consistent across domains: they preserve local geometry, allow adaptive allocation of computation, align directional structure with learned or physical operators, and often expose interactions—occlusion, shadowing, reflection, attenuation, view dependence, or wavefront structure—that are awkward in fixed-pixel or global-parameter formulations [1010.5534][2402.14817][2501.19196][2601.09022].

The limitations are equally domain-specific and nontrivial. AIR requires experimentation in the choice of \(\xi_{\rm crit}\), hierarchical bookkeeping, and, for spectra, potentially high memory because one must store the spectrum for each pixel until completion; it is most beneficial when prominent features occupy a small fraction of the image [1010.5534]. Radiance textures require precomputation, incur substantial memory cost as bucket resolution grows, approximate angular detail by finite \(n \times n\) buckets, assume a surface-bound hemispherical representation, and need clamping to avoid bucket-boundary bleeding [2301.01719]. Texel splatting trades stability against visibility because a fixed probe origin cannot see everything; eye probes reduce disocclusion but reintroduce shimmer [2603.14587]. RaySplats requires a renderer that supports ray tracing and Gaussian primitive traversal, uses confidence ellipsoids and a maximum-response opacity approximation, and relies on hit caps and termination thresholds \(\varepsilon_1,\varepsilon_2\) for tractable traversal [2501.19196].

In camera modeling, the ray formulation is not a free abstraction. Ray Diffusion reports that denser ray grids improve performance but increase compute [2402.14817]. The joint video-trajectory model is trained on static and smoothly moving scenes, has 4x temporal compression that limits temporal precision of pose recovery, and currently omits text conditioning [2604.09429]. In event-based X-ray imaging, the ratio-to-position map assumes uniform hit distributions during calibration, stable cloud shape, separable \(x\) and \(y\) components, and manageable noise; center events are precisely the least certain, which is why coarser central grouping is necessary [1501.06825]. Fourier pixels rely on linear diffraction theory, the shallow-profile approximation \(e^{i\phi}\approx 1+i\phi\), and careful fabrication and alignment; plasmonic loss is stronger in the blue, and small structures broaden in Fourier space due to finite-size diffraction [2601.09022].

Taken together, these works support a precise but broad definition. “Rays as Pixels” is not a single algorithm. It is a representational doctrine in which the atomic discrete variable is tied to a ray path, a directional sample, or a local wavefront transform. Sometimes the result is a more efficient image-plane discretization, sometimes a more faithful rendering primitive, sometimes a distributed camera representation, and sometimes a physical pixel that senses or emits more than intensity. The unifying claim is not that pixels disappear, but that the meaning of a pixel is reassigned from fixed lattice cell to ray-conditioned sample [1402.2426][1010.5534][2604.09429].

Source: https://www.emergentmind.com/topics/rays-as-pixels