Pixel-as-Ray Strategy

• Pixel-as-Ray is a strategy that treats each pixel as a ray or ray bundle rather than a discrete sample, unifying imaging across tomography, optics, and 3D detection.
• It enables methods such as exact ray footprint calculation and pixel-driven sinogram evaluation, leading to precise reconstruction and rendering techniques.
• The approach balances geometric fidelity with computational challenges, addressing issues in inversion, algorithm convergence, and calibration across various imaging modalities.

Pixel-as-Ray Strategy denotes a class of constructions in which a pixel, pixel-aligned measurement, fragment, or query is treated not merely as a discrete sample on a grid but as a ray, a bundle of rays, or a localized ray footprint in a larger imaging geometry. In the cited literature, this viewpoint appears in nonlinear tomography, where a single detector pixel integrates attenuation over all directions through $x$; in multi-camera 3D detection, where each image pixel induces a 3D camera ray; in rendering, where each fragment or display sample casts a ray in space or space-time; and in optics, where a sensor image is interpreted as a superposition of aperture-conditioned views (Lai et al., 2021, Chu et al., 2024).

1. Geometric meaning and unifying abstractions

A canonical camera-space formulation maps a pixel $(u,v)$ to a ray by first writing the homogeneous image coordinate $\mathbf p = [u,v,1]^\top$, then computing

$$\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad \mathbf d = R^{-1}\mathbf d_c, \qquad \mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,$$

with intrinsics $K$, extrinsics $(R,t)$, and camera center $\mathbf C=\mathbf t$ (Chu et al., 2024). In GPU ray tracing, the same geometry is written as

$$d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad ray(t)=O+t d,$$

so that each pixel sample becomes an independently traced ray, optionally jittered for anti-aliasing (Qin et al., 2015).

The strategy is broader than pinhole emission. In pixel-driven CT backprojection, each spatial pixel center $x_{ij}$ is projected to detector coordinate $x_{ij}\cdot \theta_{\phi_q}$ and accumulated through a triangular detector kernel,

$(u,v)$0

so that a pixel behaves like a ray footprint in sinogram space (Huber, 5 Mar 2025). Conversely, in single-pixel X-ray imaging, a detector pixel may aggregate all rays through $(u,v)$1 into one scalar. The common abstraction is therefore geometric rather than algorithmic: a pixel indexes a ray family, and the computational task is to propagate, aggregate, invert, or sample that family in a way consistent with the acquisition model.

This plurality is central. Pixel-as-Ray is not synonymous with one forward operator, one discretization, or one rendering pipeline. In some settings it means sparse per-pixel ray casting; in others it means angular aggregation; in others it means that pixel centers are mapped onto detector or scene rays. The literature uses the same geometric identification to support inverse problems, detection, rendering, and optical simulation.

2. Nonlinear single-pixel inverse problems

The most explicit operator-level realization is the nonlinear single pixel X-ray transform

$(u,v)$2

defined on $(u,v)$3, $(u,v)$4, with compactly supported $(u,v)$5 in bounded domains for mapping and stability analysis (Lai et al., 2021). Here the pixel at $(u,v)$6 does not measure one ray; it measures the angular integral of exponentially attenuated line integrals over all directions through $(u,v)$7. The nonlinearity is entirely due to the attenuation factor $(u,v)$8, which makes $(u,v)$9 a nonlinear, monotone decreasing operator in $\mathbf p = [u,v,1]^\top$0. The baseline is nonzero,

$\mathbf p = [u,v,1]^\top$1

so the empty object produces a constant offset rather than vanishing data.

The relation to classical tomography is given by linearization. The adjoint and normal operator of the X-ray transform satisfy

$\mathbf p = [u,v,1]^\top$2

and

$\mathbf p = [u,v,1]^\top$3

A Taylor expansion yields

$\mathbf p = [u,v,1]^\top$4

and differentiation at $\mathbf p = [u,v,1]^\top$5 gives

$\mathbf p = [u,v,1]^\top$6

Since $\mathbf p = [u,v,1]^\top$7 for Schwartz data and compactly supported distributions, the exact inversion formula for the linearized single-pixel transform is

$\mathbf p = [u,v,1]^\top$8

The operator admits both linearized and nonlinear stability results. For the linearized map,

$\mathbf p = [u,v,1]^\top$9

For $$\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad \mathbf d = R^{-1}\mathbf d_c, \qquad \mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,$$0 with $$\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad \mathbf d = R^{-1}\mathbf d_c, \qquad \mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,$$1, the nonlinear map satisfies a Lipschitz estimate in $$\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad \mathbf d = R^{-1}\mathbf d_c, \qquad \mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,$$2 with constants growing like $$\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad \mathbf d = R^{-1}\mathbf d_c, \qquad \mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,$$3, and a conditional lower bound follows under an inverse Poincaré-type condition for sufficiently small $$\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad \mathbf d = R^{-1}\mathbf d_c, \qquad \mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,$$4 (Lai et al., 2021). This small-data regime is also visible numerically: on a $$\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad \mathbf d = R^{-1}\mathbf d_c, \qquad \mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,$$5 Shepp–Logan phantom, reconstructions of $$\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad \mathbf d = R^{-1}\mathbf d_c, \qquad \mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,$$6 and $$\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad \mathbf d = R^{-1}\mathbf d_c, \qquad \mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,$$7 are good, $$\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad \mathbf d = R^{-1}\mathbf d_c, \qquad \mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,$$8 degrades, and $$\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad \mathbf d = R^{-1}\mathbf d_c, \qquad \mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,$$9 fails; under Gaussian noise, the $K$0 reconstruction error scales almost linearly with noise level. The paper also extends the construction to a compact, non-trapping Riemannian manifold $K$1 with smooth strictly convex boundary by defining

$K$2

and proves that injectivity of $K$3 on $K$4 implies injectivity of $K$5.

3. CT discretization, exact ray footprints, and convergence

A second major use of Pixel-as-Ray appears in discrete tomography, where the continuous Radon transform

$K$6

is approximated by ray-driven and pixel-driven finite-rank operators on spatial and detector grids (Huber, 5 Mar 2025). The ray-driven forward projector uses exact intersection-length weights of lines with pixels, while the pixel-driven backprojector evaluates the sinogram along $K$7 by two-bin linear interpolation. The paper recasts both as convolutional methods and shows that, under balanced spatial and detector resolutions with refining angular resolution, the common pairing “ray-driven forward + pixel-driven backprojection” converges in the strong operator topology. By contrast, ray-driven backprojection requires $K$8; balanced resolutions do not suffice.

This provides a rigorous interpretation of the phrase that a pixel “acts like a ray” in detector space. The backprojected contribution of $K$9 is localized near detector coordinate $(R,t)$0, and the triangular kernel becomes an approximate identity as $(R,t)$1. The forward operator realizes the dual picture: a ray acts on piecewise-constant pixels through exact strip footprints.

The exact footprint computation itself is developed in a separate algorithmic line that models the image by unit pixel or voxel basis functions and computes the X-ray transform by analytic ray–cell intersections (Chen et al., 2020). For a ray $(R,t)$2 and an axis-aligned box, axiswise entry and exit times define

$(R,t)$3

with intersection if and only if $(R,t)$4. For unit-speed rays the contribution length is

$(R,t)$5

The method derives valid index ranges analytically, visits only units with non-vanishing intersectability, and achieves $(R,t)$6 work per ray and $(R,t)$7 over $(R,t)$8 rays. It is formulated for 2D/3D parallel beam, 2D fan beam, and 3D circular and helical cone beams through explicit geometry transforms. The same work also discusses intrinsic ambiguities for grazing rays or rays lying on grid boundaries and resolves them by deterministic tie-breaking using the global 1D index of the affected unit.

Taken together, these results show that Pixel-as-Ray in tomography has two distinct but compatible meanings: exact geometric interaction of rays with pixels or voxels, and pixel-centered evaluation of sinograms along ray coordinates. The first emphasizes footprint exactness; the second emphasizes convergence of unmatched discretizations.

4. Ray-centric perception and 3D scene understanding

In multi-camera 3D object detection, Pixel-as-Ray appears as a query design principle rather than a reconstruction operator. RayFormer starts from the observation that query-based detectors such as DETR3D and PETR initialize object queries in 3D Cartesian space and sample image features at projected 2D locations, so nearby 3D queries often project to very similar pixels along the same camera ray, yielding ambiguous query features (Chu et al., 2024). The remedy is to align both query initialization and cross-view feature extraction with camera-ray geometry.

RayFormer first transforms perspective-view image features into bird’s eye view through Lift-Splat-Shoot with BEVDepth-style supervised depth, then partitions the BEV field into polar sectors using azimuth

$(R,t)$9

The full field-of-view radian range is divided into $\mathbf C=\mathbf t$0 angular sectors and the radial interval $\mathbf C=\mathbf t$1 into $\mathbf C=\mathbf t$2 sparse depth segments. In the nuScenes configuration, the perception radius is $\mathbf C=\mathbf t$3, the number of base rays is $\mathbf C=\mathbf t$4, and the number of base queries per ray is $\mathbf C=\mathbf t$5, giving $\mathbf C=\mathbf t$6 base queries. Additional foreground queries are selected by projecting ray midpoints into 2D detections; the reported setup uses approximately $\mathbf C=\mathbf t$7 foreground rays with $\mathbf C=\mathbf t$8 queries each, for approximately $\mathbf C=\mathbf t$9 queries in total.

Feature extraction is likewise ray-structured. For a query at depth $$d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad ray(t)=O+t d,$$0 and azimuth $$d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad ray(t)=O+t d,$$1, the method defines a ray segment between adjacent depth queries, samples $$d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad ray(t)=O+t d,$$2 equally spaced ray points, learns offsets, and expands each point by $$d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad ray(t)=O+t d,$$3 offsets. The default settings are $$d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad ray(t)=O+t d,$$4 for BEV sampling, $$d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad ray(t)=O+t d,$$5 for image sampling, and $$d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad ray(t)=O+t d,$$6. With $$d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad ray(t)=O+t d,$$7 frames, historical sample points are temporally aligned by constant-velocity warping:

$$d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad ray(t)=O+t d,$$8

Image and BEV features are then aggregated by deformable attention and Adaptive Mixing. Training uses Hungarian matching with a composite cost

$$d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad ray(t)=O+t d,$$9

where the radian term

$x_{ij}$0

encourages assignments along the same ray.

The reported results validate the design at the detection level. On the nuScenes test set with V2-99 at $x_{ij}$1, RayFormer reaches $x_{ij}$2 mAP and $x_{ij}$3 NDS. On validation, ResNet50 at $x_{ij}$4 and $x_{ij}$5 gives $x_{ij}$6 mAP and $x_{ij}$7 NDS, while ResNet101 at $x_{ij}$8 and $x_{ij}$9 gives $x_{ij}\cdot \theta_{\phi_q}$0 mAP and $x_{ij}\cdot \theta_{\phi_q}$1 NDS. The single-frame ResNet50 setting improves to $x_{ij}\cdot \theta_{\phi_q}$2 mAP and $x_{ij}\cdot \theta_{\phi_q}$3 NDS. Ablations attribute gains to image ray sampling, BEV ray sampling, and 2D-guided foreground queries; multi-frame input reduces mAVE from $x_{ij}\cdot \theta_{\phi_q}$4 to $x_{ij}\cdot \theta_{\phi_q}$5 when $x_{ij}\cdot \theta_{\phi_q}$6 historical frames are used. In this domain, Pixel-as-Ray does not collapse directionality; it explicitly preserves it to reduce feature overlap.

5. Rendering, particle tracing, and space-time sampling

In graphics, Pixel-as-Ray is often literal. A massively parallel OpenCL implementation maps each work item to one ray, traces sphere intersections independently, and progressively accumulates radiance through repeated kernel launches (Qin et al., 2015). The rendering equation is used in the standard differential form,

$x_{ij}\cdot \theta_{\phi_q}$7

while recursion is replaced by iterative depth-limited loops because OpenCL kernels do not support recursion. On an ATI Radeon HD 6750M, the first local-illumination render at $x_{ij}\cdot \theta_{\phi_q}$8 takes $x_{ij}\cdot \theta_{\phi_q}$9 seconds, the image improves after $(u,v)$00 seconds of accumulation, and global illumination at recursion depth $(u,v)$01 is generated in about $(u,v)$02 seconds.

A more specialized realization is 3D Gaussian Ray Tracing, which casts one primary ray per pixel, traverses a BVH built over per-particle proxy meshes, and composites semi-transparent Gaussian particle contributions in front-to-back depth order (Moenne-Loccoz et al., 2024). Each anisotropic Gaussian has mean $(u,v)$03, covariance $(u,v)$04, opacity scale $(u,v)$05, and view-dependent radiance $(u,v)$06. Hardware returns mesh-hit distance $(u,v)$07, but the Gaussian is evaluated at the maximum-response point

$(u,v)$08

with opacity

$(u,v)$09

The implementation uses a $(u,v)$10-buffer with $(u,v)$11, early termination thresholds $(u,v)$12 during training and $(u,v)$13 at inference, and proxy meshes derived from stretched regular icosahedra. On MipNeRF360, the ray tracer matches or slightly exceeds rasterization quality, reporting PSNR $(u,v)$14 versus $(u,v)$15 for 3DGS, but it is slower for pinhole primary rays; FPS is $(u,v)$16–$(u,v)$17 versus $(u,v)$18. With a generalized kernel of degree $(u,v)$19, tracing becomes approximately $(u,v)$20 faster at small quality cost.

Perceptual rasterization extends the same idea into space-time. Instead of one global frame time, each fragment defines a ray whose display time depends on screen location:

$(u,v)$21

Foveation is introduced by a nonlinear spatial mapping

$(u,v)$22

and the fragment program performs per-fragment ray-triangle intersection against geometry interpolated to the fragment’s display time (Ritschel et al., 2018). The resulting pipeline unifies rolling display timing and variable pixel density. In the Helicopter scene, rolling hull, adaptive, and Zenon bounds report $(u,v)$23, $(u,v)$24, and $(u,v)$25 respectively, while perceptual rasterization reaches SSIM $(u,v)$26 and warping gives approximately $(u,v)$27.

An even more radical extension abandons frames altogether. Adaptive frameless rendering treats each display location as a receiver of asynchronously scheduled samples $(u,v)$28, maintains a KD-tree tiling of the image plane, and streams individually scheduled rays through the scene according to spatial and temporal variation (Watson et al., 28 Jun 2025). A deep buffer stores color, time, depth, and auxiliary sample data; the reconstructor performs adaptive space-time filtering at display refresh. At equal sampling rates of $(u,v)$29, $(u,v)$30, and $(u,v)$31 samples per second, adaptive frameless rendering reports $(u,v)$32–$(u,v)$33 lower RMS error than framed and nonadaptive frameless baselines, and parity with framed rendering at roughly $(u,v)$34 fewer samples. Here Pixel-as-Ray no longer means one ray per frame pixel; it means one independently scheduled ray per space-time sample.

6. Lens, sensor, and radiance-level realizations

At the optical level, Pixel-as-Ray is tied to aperture structure and wavefront splitting. Lens imaging can be interpreted as a superposition of elemental images, each generated by a different point on the aperture and each corresponding to a camera-obscura-like view (Grusche, 2015). With thin-lens geometry,

$(u,v)$35

and for an aperture point $(u,v)$36 the screen intersection behind the lens is

$(u,v)$37

If $(u,v)$38, the screen coordinate is independent of aperture point and elemental images coincide sharply; if $(u,v)$39, the dependence on $(u,v)$40 produces blur through misalignment of views. This formulation makes explicit that a sensor pixel gathers a bundle of aperture-conditioned rays rather than a single geometric line.

Dual-pixel simulation makes that bundle computationally explicit. Sdirt treats each depth-map pixel as a 3D object point $(u,v)$41, samples the entrance pupil densely, traces rays through a multi-element lens by Snell’s law, and determines whether each ray falls into the left or right sub-pixel under a microlens (He et al., 14 Mar 2025). The left/right PSFs are

$(u,v)$42

The implementation uses the Canon RF50mm F/1.8 lens, a single wavelength at $(u,v)$43, and $(u,v)$44 rays per object point. Microlens parameters obtained by grid search are

$(u,v)$45

To accelerate rendering, an MLP with $(u,v)$46 hidden layers of $(u,v)$47 neurons predicts the concatenated left/right PSFs with kernel size $(u,v)$48 at F/4. A full $(u,v)$49 DP PSF map then takes approximately $(u,v)$50 instead of approximately $(u,v)$51 by brute-force ray tracing. Against real Canon RF50mm data, the simulator reports NCC $(u,v)$52 and NSD $(u,v)$53; on real DP119 depth estimation, planar scenes reach Acc-1 $(u,v)$54 and MAE $(u,v)$55.

A different sensor-side realization is Radiance Textures, where each texel stores an $(u,v)$56 radiance bucket indexed by incidence direction, so the atlas size becomes $(u,v)$57 for a base texture of size $(u,v)$58 (Fober, 2023). Runtime shading computes view, reflection, refraction, or shadow directions, maps them into bucket coordinates through an azimuthal equisolid projection and disc-to-square transform, and samples precomputed directional radiance. The technique is explicitly described as having “the computational footprint of simple diffuse-only graphics” at the cost of storage memory footprint. In this setting, Pixel-as-Ray is implemented by offline ray tracing and online directional lookup: the rasterized pixel stands in for one or more rays whose angular dependence has been prebaked.

7. Recurrent limitations, misconceptions, and open problems

A recurring misconception is that Pixel-as-Ray always preserves directional information. In single-pixel X-ray tomography the opposite is true: $(u,v)$59 collapses all directions at $(u,v)$60 into one scalar, so inversion depends on monotonicity, linearization, and small-data stability rather than direct angular recovery (Lai et al., 2021). The same work states that removing or weakening the inverse Poincaré-type condition in the nonlinear lower bound is open, and that nonlinear stability for $(u,v)$61 on simple manifolds is largely undeveloped.

Another misconception is that “ray-driven” and “pixel-driven” must be used as matched adjoints. The strong-convergence analysis shows that the practically common unmatched pair—ray-driven forward and pixel-driven backprojection—is the one justified under balanced resolutions, whereas ray-driven backprojection requires detector refinement faster than spatial refinement (Huber, 5 Mar 2025). The convergence proved there is in the strong operator topology, not operator norm, so approximation quality remains input-dependent.

In scene understanding, ray-centric organization is not cost-free. RayFormer depends on accurate intrinsics and extrinsics, and severe occlusions, truncations, or inaccurate calibration can misguide ray selection and temporal wrapping; increasing $(u,v)$62, $(u,v)$63, $(u,v)$64, and $(u,v)$65 improves accuracy at greater memory and compute cost (Chu et al., 2024). In graphics, per-ray independence simplifies implementation but induces divergence, load imbalance, and progressive noise; BVH rebuilds add overhead, and in Gaussian ray tracing dense overlapping particle clouds increase hit counts and reduce the speed advantage over rasterization (Qin et al., 2015, Moenne-Loccoz et al., 2024).

Optical simulation introduces its own constraints. Sdirt currently requires a fixed-focus lens with available lens data and a DP sensor, approximates the microlens and sub-pixel structure, omits CFA pattern, demosaicing, read noise, and rolling shutter, and neglects or approximates diffraction and chromatic aberration (He et al., 14 Mar 2025). Perceptual rasterization likewise assumes linear interpolation of model-view matrices and vertex motion over the frame; its prototype is slower than highly optimized fixed-function rasterization even though it outperforms generic ray tracing for the reported rolling and foveated tasks (Ritschel et al., 2018). Adaptive frameless rendering reduces sample count, but it must control temporal noise, ghosting under reprojection, and depth-gating failures when older samples are reused (Watson et al., 28 Jun 2025).

These limitations do not define a single failure mode; they define the main design choice underlying the strategy. Treating pixels as rays gains geometric fidelity, optical correctness, or adaptive sampling flexibility, but it shifts difficulty toward inversion, calibration, ordering, memory, or reconstruction. Across tomography, detection, graphics, and optics, Pixel-as-Ray is therefore best understood as a geometric principle whose benefits depend on how each field manages that shift.