Rays as Pixels: Rethinking Imaging Primitives

• Rays as Pixels is a representational doctrine that redefines a pixel as a ray-conditioned sample rather than a fixed screen cell.
• It unifies imaging, graphics, and optical device modeling by treating detector samples, texture elements, and camera patches as directional ray measurements.
• This approach enables adaptive rendering and efficient computation while capturing nuanced interactions like occlusion, shadowing, and wavefront sampling.

Searching arXiv for the cited papers to ground the article and confirm bibliographic details. arXiv search query: "Rays as Pixels camera trajectories video diffusion (Jang et al., 10 Apr 2026)" “Rays as Pixels” denotes a family of formulations in which the primary discrete element is not a fixed screen pixel or a compact camera-parameter vector, but a ray, a ray bundle, or a direction-indexed sample. In this view, a detector sample may be treated as a line integral along a ray, a ray-traced image element, a per-patch Plücker ray, a cubemap texel indexed by world-space direction, a surface texel storing directional radiance, or a dense ray image (“raxel”). Across imaging, graphics, camera estimation, and reciprocal optics, the common move is to make ray geometry or directional response the native carrier of signal and then reconstruct images, poses, or wavefronts from that representation (Dillon et al., 2014, Parkin, 2010, Zhang et al., 2024, Jang et al., 10 Apr 2026).

1. Conceptual scope and recurring representations

The literature uses the phrase in several technically distinct but structurally related ways. In tomography and multiview imaging, each detector sample is a projection on a ray. In adaptive ray tracing, each pixel is a ray-sampled region that can be recursively subdivided. In camera modeling, each image patch predicts a ray, so a camera becomes a dense field of rays rather than a single pose vector. In rasterization-oriented methods, ray responses are stored in textures or cubemaps and later read back like pixels. In optical hardware, a photon event or a pixel-sized diffractive element becomes the effective unit of spatial or wavefront sampling (Dillon et al., 2014, Parkin, 2010, Fober, 2023, Nowak et al., 2015, Glauser et al., 13 Jan 2026).

Domain	Ray-like unit	Function
Tomography and multiview imaging	detector sample on a ray	line integral or attenuated projection
Adaptive image ray-tracing	adaptive ray/pixel cell	local refinement around prominent features
Radiance textures	$n \times n$ bucket per texel	directional radiance lookup
Camera pose and video diffusion	per-patch ray or raxel	distributed camera representation
Texel splatting and Gaussian ray tracing	cubemap texel or camera ray	stable world sample or Gaussian traversal
X-ray and Fourier-optical devices	photon event or Fourier pixel	sub-pixel localization or reciprocal wavefront control

A persistent misconception is that “rays as pixels” refers only to replacing screen pixels with traced rays at render time. The published usage is broader. It includes inverse problems in which a pixel is reinterpreted as a ray measurement, dense geometric representations in which each image patch stores a ray, and optical devices in which a local element senses or emits an entire directional or vectorial field rather than a scalar intensity (Dillon et al., 2014, Zhang et al., 2024, Glauser et al., 13 Jan 2026).

2. Ray measurements as the basis of imaging and inverse problems

“Imaging with Rays: Microscopy, Medical Imaging, and Computer Vision” formulates a wide range of imaging systems as ray-based projection operators (Dillon et al., 2014). For a simple projection along the $z$-axis, it writes

$p(x,y) = \int f(x,z,y)\,dz,$

with Fourier-slice form

$\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).$

In 2D, the same idea appears as a Radon-style projection

$g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.$

Within this framework, CT is modeled by x-rays as rays through the object, transmission tomography becomes log-linear after taking logarithms of

$p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},$

PET/SPECT are written as attenuated emission projections, computational confocal microscopy is treated as a sheared ray-space sampling process, and a pinhole camera is represented as a collection of cone-beam-like projections. After discretization, these systems reduce to a matrix equation $A x = b$, where each row of $A$ corresponds to a ray measurement (Dillon et al., 2014).

The same paper extends the ray framework to self-occluding opaque objects by jointly estimating brightness and attenuation. Its continuous forward model factorizes the object-dependent PSF as

$$H(\mathbf r,\mathbf r') = P(\mathbf r,\mathbf r')\,Q(\mathbf r,\mathbf r'),$$

and its discrete nonlinear model is

$$\mathbf b = \mathbf C \,\exp\!\bigl(\mathbf E \log \mathbf x\bigr),$$

with $z$0 concatenating attenuation and brightness. Reconstruction is posed as a regularized nonlinear optimization problem with an explicit empty-space prior and a positivity floor $z$1 (Dillon et al., 2014).

This establishes a strict interpretation of the phrase: a pixel is not fundamentally a square on a detector plane, but a sample associated with a ray path and its accumulated interaction with the scene. A plausible implication is that once ray measurements are treated as the primitive, distinctions among medical imaging, microscopy, and multiview computer vision become largely distinctions in forward operators and regularization rather than in measurement ontology.

3. Adaptive and ray-centric rendering

In “Adaptive image ray-tracing for astrophysical simulations,” the image plane is no longer a uniformly fixed pixel grid; it is an adaptive hierarchy of ray-sampled cells (Parkin, 2010). Conventional ray tracing “first discretize[s] the plane of the sky into a uniform array of pixels and then follow[s] the path of a ray for each respective pixel.” AIR instead begins from a coarse image, traces rays for current pixels, computes a truncation-error estimate, refines flagged pixels, and iterates until the desired image resolution is reached. The refinement criterion is a modified second-derivative interpolation error estimate from Löhner (1987), with $z$2, and refinement occurs when

$z$3

For the test problem, $z$4 works well, and for more complex images a starting point of $z$5 is suggested (Parkin, 2010).

Its test case used a $z$6 fixed image versus AIR with a $z$7 base image, 4 nested image refinement levels, and the same effective final resolution $z$8. The fixed image required 9604 s and 262144 pixels. AIR at $z$9 used 1339 s and 22748 pixels, with $p(x,y) = \int f(x,z,y)\,dz,$0 fractional error, summarized as about 1/7 the time, about 1/12 the number of pixels, and only 0.01% error in the integrated flux. The conclusion gives a typical speed-up factor of $p(x,y) = \int f(x,z,y)\,dz,$1–$p(x,y) = \int f(x,z,y)\,dz,$2 and a pixel reduction factor of $p(x,y) = \int f(x,z,y)\,dz,$3–$p(x,y) = \int f(x,z,y)\,dz,$4, with savings roughly the inverse of the image filling factor (Parkin, 2010).

A different ray-to-pixel conversion appears in “Radiance Textures for Rasterizing Ray-Traced Data” (Fober, 2023). There, “Each pixel of the model’s texture contains discrete radiance hemispherical map of size $p(x,y) = \int f(x,z,y)\,dz,$5, called ‘bucket’.” The object texture has size $p(x,y) = \int f(x,z,y)\,dz,$6, each texel is replaced by an $p(x,y) = \int f(x,z,y)\,dz,$7 radiance bucket, and the stored representation behaves like a larger texture of size $p(x,y) = \int f(x,z,y)\,dz,$8. The radiance is parameterized as $p(x,y) = \int f(x,z,y)\,dz,$9 or $\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).$0, incidence direction is mapped into a 2D bucket coordinate, and the fragment shader reconstructs color by directional lookup. The method explicitly trades storage for runtime cost: the computational footprint is that of simple diffuse-only graphics, while fidelity depends on bucket resolution and memory (Fober, 2023).

“Texel Splatting: Perspective-Stable 3D Pixel Art” turns each cubemap texel into a stable scene sample indexed by direction from a fixed probe origin (Ebert, 15 Mar 2026). Its premise is that orthographic camera snapping works because projection is linear and depth-independent, whereas perspective projection causes depth-dependent screen drift that no single snap can correct. The method therefore renders geometry into a cubemap from a fixed world-space origin and splats each texel back to the screen as a world-space quad. World position is reconstructed as

$\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).$1

Cubemap indexing gives rotation invariance, and grid-snapping the origin gives translation invariance. The primary limitation is equally explicit: a fixed origin cannot see all geometry, so disocclusion at probe boundaries remains the open tradeoff (Ebert, 15 Mar 2026).

“RaySplats: Ray Tracing based Gaussian Splatting” replaces projection-based 3D Gaussian Splatting with ray tracing through Gaussian confidence ellipsoids (Byrski et al., 31 Jan 2025). A ray $\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).$2 is intersected with

$\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).$3

and ray-wise color is composited as

$\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).$4

Because Gaussian primitives become ray-intersectable objects, shadows, reflections, transparency, refraction, and mesh integration become natural parts of the same traversal process (Byrski et al., 31 Jan 2025).

4. Cameras as dense ray fields

“Cameras as Rays: Pose Estimation via Ray Diffusion” argues against direct regression of a low-dimensional global camera parametrization $\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).$5 and instead represents a camera as a set of per-patch rays $\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).$6, with each ray $\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).$7 tied to a known pixel location $\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).$8 (Zhang et al., 2024). Rays are represented in Plücker coordinates

$\tilde{p}(k_x,k_y)=\tilde{f}(k_x,k_y,0).$9

with $g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.$0. Given a camera,

$g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.$1

The reverse conversion recovers the camera center by least squares and then uses a homography, DLT, RQ decomposition, and $g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.$2 to recover the conventional camera parameters (Zhang et al., 2024).

Its regression model uses a pretrained frozen DINOv2 backbone, concatenates patch features with normalized pixel coordinates, and processes all $g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.$3 tokens with a set-level transformer. A diffusion version denoises ray bundles directly in ray space with the same $g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.$4 reconstruction loss. On CO3Dv2, the ray representation outperforms direct $g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.$5 regression and pose-space diffusion. At 2 images on seen categories, the reported relative-rotation-within-$g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.$6 values are 49.1 for R+T Regression, 75.7 for PoseDiffusion, 81.8 for RelPose++, 88.8 for Ray Regression, and 91.8 for Ray Diffusion; on unseen categories they are 42.7, 63.2, 69.8, 79.0, and 83.5. An ablation further reports that $g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.$7 rays is best among the tested settings, while more rays increase compute (Zhang et al., 2024).

“Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories” generalizes the same principle from pose estimation to joint generative modeling (Jang et al., 10 Apr 2026). It learns the joint density

$g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.$8

over video latents $g(r,\theta)=\int f(x,r+x\cos\theta)\,dx.$9 and ray or trajectory latents $p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},$0, rather than only $p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},$1 or $p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},$2. Cameras are encoded as dense ray pixels, or raxels. After canonicalization by

$p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},$3

each pixel $p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},$4 yields a direction and origin

$p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},$5

and the raxel is defined as $p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},$6. The point of this construction is architectural: raxels are dense, 3-channel, VAE-compatible, and spatially aligned with video tensors (Jang et al., 10 Apr 2026).

The model builds on a pretrained Wan 2.1 14B video diffusion transformer, uses Flow Matching with

$p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},$7

and $p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},$8, and couples video and ray branches through Decoupled Self-Cross Attention. A single trained model supports three tasks: predicting camera trajectories from video, jointly generating video and trajectory from sparse images, and generating video from sparse images along a target trajectory. It is evaluated by a closed-loop self-consistency test, and it reports that trajectory prediction requires far fewer denoising steps than video generation; best pose results occur at 2 diffusion steps, with mean Relative Rotation Accuracy @30 around 95.91 on RealEstate10K, 88.37 on DL3DV-140, and 93.51 on Tanks and Temples (Jang et al., 10 Apr 2026).

5. Event-based and reciprocal optical realizations

In “Sub-pixel resolution with color X-ray camera SLcam(R),” the effective imaging unit is a single X-ray photon event rather than a fixed CCD pixel (Nowak et al., 2015). The system combines a pnCCD detector with 48 $p(x,y)=\exp\left\{-\int \eta(x,y,z)\,dz\right\},$9m pixel size and polycapillary optics. Because the detector is single-photon and energy-resolving, each event carries spatial and energy information. The absorbed photon creates a charge cloud,

$A x = b$0

whose distribution over neighboring pixels encodes sub-pixel position (Nowak et al., 2015).

The algorithm uses intensity ratios

$A x = b$1

defines $A x = b$2 and $A x = b$3 over a $A x = b$4 box, and then maps ratio histograms to positions by

$A x = b$5

A central contribution is that the method uses all photon events, including pixel-center events, rather than only corner events. With 8:1 optics, one CCD pixel corresponds to a $A x = b$6 area on the sample; $A x = b$7 sub-pixel division resolves one additional stripe in the Au bar pattern, tilted structures can resolve $A x = b$8 or even $A x = b$9 bars, and overly fine grids such as $A$0 create artifacts, so the best useful division for a $A$1 pnCCD pixel is approximately $A$2 (Nowak et al., 2015).

“Fourier pixels for reciprocal light control” extends the concept from event localization to reciprocal control of optical wavefronts (Glauser et al., 13 Jan 2026). A Fourier pixel is a miniaturized diffractive element based on Fourier optics that can both generate and sense optical fields with control over amplitude, phase, and polarization. The implementation uses coherent surface plasmon polariton waves on silver, launched by source gratings and diffracted by shallow wavy microstructures. In the scalar case, the reference wave is

$A$3

the surface response is $A$4, and shallow-profile linearization yields

$A$5

For vectorial operation, the pixel uses a two-component field representation and a diagonal transparency matrix, allowing polarization-dependent outputs and full Stokes sensing through

$A$6

Reported demonstrations include vortex beams with $A$7, vector beams of order $A$8 and $A$9, a diffraction-limited focus with measured spot size around $$H(\mathbf r,\mathbf r') = P(\mathbf r,\mathbf r')\,Q(\mathbf r,\mathbf r'),$$0, focusing efficiencies exceeding 40% power in to power out for visible wavelengths in the green/red range, and phase readout fluctuations limited by the setup to about $$H(\mathbf r,\mathbf r') = P(\mathbf r,\mathbf r')\,Q(\mathbf r,\mathbf r'),$$1 input-angle uncertainty (Glauser et al., 13 Jan 2026).

6. Tradeoffs, limitations, and interpretive boundaries

The technical benefits of ray-native representations are consistent across domains: they preserve local geometry, allow adaptive allocation of computation, align directional structure with learned or physical operators, and often expose interactions—occlusion, shadowing, reflection, attenuation, view dependence, or wavefront structure—that are awkward in fixed-pixel or global-parameter formulations (Parkin, 2010, Zhang et al., 2024, Byrski et al., 31 Jan 2025, Glauser et al., 13 Jan 2026).

The limitations are equally domain-specific and nontrivial. AIR requires experimentation in the choice of $$H(\mathbf r,\mathbf r') = P(\mathbf r,\mathbf r')\,Q(\mathbf r,\mathbf r'),$$2, hierarchical bookkeeping, and, for spectra, potentially high memory because one must store the spectrum for each pixel until completion; it is most beneficial when prominent features occupy a small fraction of the image (Parkin, 2010). Radiance textures require precomputation, incur substantial memory cost as bucket resolution grows, approximate angular detail by finite $$H(\mathbf r,\mathbf r') = P(\mathbf r,\mathbf r')\,Q(\mathbf r,\mathbf r'),$$3 buckets, assume a surface-bound hemispherical representation, and need clamping to avoid bucket-boundary bleeding (Fober, 2023). Texel splatting trades stability against visibility because a fixed probe origin cannot see everything; eye probes reduce disocclusion but reintroduce shimmer (Ebert, 15 Mar 2026). RaySplats requires a renderer that supports ray tracing and Gaussian primitive traversal, uses confidence ellipsoids and a maximum-response opacity approximation, and relies on hit caps and termination thresholds $$H(\mathbf r,\mathbf r') = P(\mathbf r,\mathbf r')\,Q(\mathbf r,\mathbf r'),$$4 for tractable traversal (Byrski et al., 31 Jan 2025).

In camera modeling, the ray formulation is not a free abstraction. Ray Diffusion reports that denser ray grids improve performance but increase compute (Zhang et al., 2024). The joint video-trajectory model is trained on static and smoothly moving scenes, has 4x temporal compression that limits temporal precision of pose recovery, and currently omits text conditioning (Jang et al., 10 Apr 2026). In event-based X-ray imaging, the ratio-to-position map assumes uniform hit distributions during calibration, stable cloud shape, separable $$H(\mathbf r,\mathbf r') = P(\mathbf r,\mathbf r')\,Q(\mathbf r,\mathbf r'),$$5 and $$H(\mathbf r,\mathbf r') = P(\mathbf r,\mathbf r')\,Q(\mathbf r,\mathbf r'),$$6 components, and manageable noise; center events are precisely the least certain, which is why coarser central grouping is necessary (Nowak et al., 2015). Fourier pixels rely on linear diffraction theory, the shallow-profile approximation $$H(\mathbf r,\mathbf r') = P(\mathbf r,\mathbf r')\,Q(\mathbf r,\mathbf r'),$$7, and careful fabrication and alignment; plasmonic loss is stronger in the blue, and small structures broaden in Fourier space due to finite-size diffraction (Glauser et al., 13 Jan 2026).

Taken together, these works support a precise but broad definition. “Rays as Pixels” is not a single algorithm. It is a representational doctrine in which the atomic discrete variable is tied to a ray path, a directional sample, or a local wavefront transform. Sometimes the result is a more efficient image-plane discretization, sometimes a more faithful rendering primitive, sometimes a distributed camera representation, and sometimes a physical pixel that senses or emits more than intensity. The unifying claim is not that pixels disappear, but that the meaning of a pixel is reassigned from fixed lattice cell to ray-conditioned sample (Dillon et al., 2014, Parkin, 2010, Jang et al., 10 Apr 2026).