---
title: Pixel-as-Ray Strategy
url: https://www.emergentmind.com/topics/pixel-as-ray-strategy
type: topic
---

# Pixel-as-Ray Strategy

Pixel-as-Ray Strategy denotes a class of constructions in which a pixel, pixel-aligned measurement, fragment, or query is treated not merely as a discrete sample on a grid but as a ray, a bundle of rays, or a localized ray footprint in a larger imaging geometry. In the cited literature, this viewpoint appears in nonlinear tomography, where a single detector pixel integrates attenuation over all directions through $x$; in multi-camera 3D detection, where each image pixel induces a 3D camera ray; in rendering, where each fragment or display sample casts a ray in space or space-time; and in optics, where a sensor image is interpreted as a superposition of aperture-conditioned views [2112.13978][2407.14923].

## 1. Geometric meaning and unifying abstractions

A canonical camera-space formulation maps a pixel $(u,v)$ to a ray by first writing the homogeneous image coordinate $\mathbf p = [u,v,1]^\top$, then computing
$$
\mathbf d_c = \frac{K^{-1}\mathbf p}{\|K^{-1}\mathbf p\|}, \qquad
\mathbf d = R^{-1}\mathbf d_c, \qquad
\mathbf X(\lambda) = \mathbf C + \lambda \mathbf d, \quad \lambda>0,
$$
with intrinsics $K$, extrinsics $(R,t)$, and camera center $\mathbf C=\mathbf t$ [2407.14923]. In GPU ray tracing, the same geometry is written as
$$
d = \operatorname{normalize}(R K^{-1}[u+\xi_u, v+\xi_v, 1]^T), \qquad
ray(t)=O+t d,
$$
so that each pixel sample becomes an independently traced ray, optionally jittered for anti-aliasing [1504.03151].

The strategy is broader than pinhole emission. In pixel-driven CT backprojection, each spatial pixel center $x_{ij}$ is projected to detector coordinate $x_{ij}\cdot \theta_{\phi_q}$ and accumulated through a triangular detector kernel,
$$
[R^*_{\mathrm{pix}} g]_{i,j}
=
\Delta s \sum_q |\Phi_q| \sum_p
\omega_{\mathrm{pd}}(x_{ij}\cdot \theta_{\phi_q}-s_p)\, g_{q,p},
\qquad
\omega_{\mathrm{pd}}(t)=\frac{1}{\Delta s^2}\max\{\Delta s-|t|,0\},
$$
so that a pixel behaves like a ray footprint in sinogram space [2503.03069]. Conversely, in single-pixel X-ray imaging, a detector pixel may aggregate all rays through $x$ into one scalar. The common abstraction is therefore geometric rather than algorithmic: a pixel indexes a ray family, and the computational task is to propagate, aggregate, invert, or sample that family in a way consistent with the acquisition model.

This plurality is central. Pixel-as-Ray is not synonymous with one forward operator, one discretization, or one rendering pipeline. In some settings it means sparse per-pixel ray casting; in others it means angular aggregation; in others it means that pixel centers are mapped onto detector or scene rays. The literature uses the same geometric identification to support inverse problems, detection, rendering, and optical simulation.

## 2. Nonlinear single-pixel inverse problems

The most explicit operator-level realization is the nonlinear single pixel X-ray transform
$$
Xf(x,\theta) := \int_{\mathbb R} f(x+s\theta)\,ds,
\qquad
Kf(x) := \int_{\mathbb S^{n-1}} e^{-Xf(x,\theta)}\, d\theta,
$$
defined on $\mathbb R^n$, $n\ge 2$, with compactly supported $f$ in bounded domains for mapping and stability analysis [2112.13978]. Here the pixel at $x$ does not measure one ray; it measures the angular integral of exponentially attenuated line integrals over all directions through $x$. The nonlinearity is entirely due to the attenuation factor $e^{-Xf(x,\theta)}$, which makes $K$ a nonlinear, monotone decreasing operator in $f$. The baseline is nonzero,
$$
K[0](x)=|\mathbb S^{n-1}|,
$$
so the empty object produces a constant offset rather than vanishing data.

The relation to classical tomography is given by linearization. The adjoint and normal operator of the X-ray transform satisfy
$$
X' \psi (x) := \int_{\mathbb S^{n-1}} \psi\bigl(x-(x\cdot \theta)\theta,\theta\bigr)\, d\theta,
$$
and
$$
X'X f(x)
=
\int_{\mathbb S^{n-1}} \int_{\mathbb R} f(x+s\theta)\, ds\, d\theta
=
2\int_{\mathbb R^n} \frac{f(y)}{|x-y|^{n-1}}\, dy.
$$
A Taylor expansion yields
$$
K f(x)
=
|\mathbb S^{n-1}| - X'X f(x) + \int_{\mathbb S}^{n-1} R f(x,\theta)\, d\theta,
$$
and differentiation at $\varepsilon=0$ gives
$$
\partial_\varepsilon\big|_{\varepsilon=0} K[\varepsilon g](x) = -\,X'X g(x).
$$
Since $f = c_n |D| X'X f$ for Schwartz data and compactly supported distributions, the exact inversion formula for the linearized single-pixel transform is
$$
g = -\, c_n\, |D|\, \bigl(\partial_\varepsilon\big|_{\varepsilon=0} K[\varepsilon g]\bigr).
$$

The operator admits both linearized and nonlinear stability results. For the linearized map,
$$
C^{-1}\|g\|_{L^2(\mathbb R^n)}
\le
\left\|
\partial_\varepsilon\big|_{\varepsilon=0} K[\varepsilon g]
\right\|_{H^1(\Omega_1)}.
$$
For $f_1,f_2\in\mathcal M_1$ with $\|f_j\|_{C^1}\le M$, the nonlinear map satisfies a Lipschitz estimate in $H^1(\Omega_1)$ with constants growing like $e^{CM}$, and a conditional lower bound follows under an inverse Poincaré-type condition for sufficiently small $M$ [2112.13978]. This small-data regime is also visible numerically: on a $101\times 101$ Shepp–Logan phantom, reconstructions of $f$ and $10f$ are good, $20f$ degrades, and $40f$ fails; under Gaussian noise, the $L^2$ reconstruction error scales almost linearly with noise level. The paper also extends the construction to a compact, non-trapping Riemannian manifold $(M,g)$ with smooth strictly convex boundary by defining
$$
K_g f(x) := \int_{S_x M} e^{-X_g f(x,v)}\, dv,
$$
and proves that injectivity of $X_g$ on $C(M)$ implies injectivity of $K_g$.

## 3. CT discretization, exact ray footprints, and convergence

A second major use of Pixel-as-Ray appears in discrete tomography, where the continuous Radon transform
$$
(R f)(\phi,s) = \int_{\mathbb R} f(s\theta_\phi + t\theta_\phi^\perp)\, dt,
\qquad
(R^* g)(x) = \int_0^\pi g(\phi, x\cdot \theta_\phi)\, d\phi,
$$
is approximated by ray-driven and pixel-driven finite-rank operators on spatial and detector grids [2503.03069]. The ray-driven forward projector uses exact intersection-length weights of lines with pixels, while the pixel-driven backprojector evaluates the sinogram along $s=x\cdot \theta$ by two-bin linear interpolation. The paper recasts both as convolutional methods and shows that, under balanced spatial and detector resolutions with refining angular resolution, the common pairing “ray-driven forward + pixel-driven backprojection” converges in the strong operator topology. By contrast, ray-driven backprojection requires $\Delta s/\Delta x \to 0$; balanced resolutions do not suffice.

This provides a rigorous interpretation of the phrase that a pixel “acts like a ray” in detector space. The backprojected contribution of $x_{ij}$ is localized near detector coordinate $x_{ij}\cdot \theta_{\phi_q}$, and the triangular kernel becomes an approximate identity as $\Delta s\to 0$. The forward operator realizes the dual picture: a ray acts on piecewise-constant pixels through exact strip footprints.

The exact footprint computation itself is developed in a separate algorithmic line that models the image by unit pixel or voxel basis functions and computes the X-ray transform by analytic ray–cell intersections [2006.00686]. For a ray $r(t)=r_0+t u$ and an axis-aligned box, axiswise entry and exit times define
$$
t_{\min} = \max(t_x^{\min}, t_y^{\min}, t_z^{\min}),
\qquad
t_{\max} = \min(t_x^{\max}, t_y^{\max}, t_z^{\max}),
$$
with intersection if and only if $t_{\max}\ge t_{\min}$. For unit-speed rays the contribution length is
$$
\ell_i = \max(0, t_{\mathrm{out}} - t_{\mathrm{in}}).
$$
The method derives valid index ranges analytically, visits only units with non-vanishing intersectability, and achieves $O(N)$ work per ray and $O(NM)$ over $M$ rays. It is formulated for 2D/3D parallel beam, 2D fan beam, and 3D circular and helical cone beams through explicit geometry transforms. The same work also discusses intrinsic ambiguities for grazing rays or rays lying on grid boundaries and resolves them by deterministic tie-breaking using the global 1D index of the affected unit.

Taken together, these results show that Pixel-as-Ray in tomography has two distinct but compatible meanings: exact geometric interaction of rays with pixels or voxels, and pixel-centered evaluation of sinograms along ray coordinates. The first emphasizes footprint exactness; the second emphasizes convergence of unmatched discretizations.

## 4. Ray-centric perception and 3D scene understanding

In multi-camera 3D object detection, Pixel-as-Ray appears as a query design principle rather than a reconstruction operator. RayFormer starts from the observation that query-based detectors such as DETR3D and PETR initialize object queries in 3D Cartesian space and sample image features at projected 2D locations, so nearby 3D queries often project to very similar pixels along the same camera ray, yielding ambiguous query features [2407.14923]. The remedy is to align both query initialization and cross-view feature extraction with camera-ray geometry.

RayFormer first transforms perspective-view image features into bird’s eye view through Lift-Splat-Shoot with BEVDepth-style supervised depth, then partitions the BEV field into polar sectors using azimuth
$$
\phi = \operatorname{atan2}(y-y_c,\, x-x_c).
$$
The full field-of-view radian range is divided into $N_r$ angular sectors and the radial interval $[0,D]$ into $N_d$ sparse depth segments. In the nuScenes configuration, the perception radius is $D=65\,\mathrm m$, the number of base rays is $N_r=135$, and the number of base queries per ray is $N_d=6$, giving $810$ base queries. Additional foreground queries are selected by projecting ray midpoints into 2D detections; the reported setup uses approximately $30$ foreground rays with $3$ queries each, for approximately $900$ queries in total.

Feature extraction is likewise ray-structured. For a query at depth $\lambda_q$ and azimuth $\phi_q$, the method defines a ray segment between adjacent depth queries, samples $K$ equally spaced ray points, learns offsets, and expands each point by $P$ offsets. The default settings are $K=5$ for BEV sampling, $K=3$ for image sampling, and $P=4$. With $T=8$ frames, historical sample points are temporally aligned by constant-velocity warping:
$$
x_{t,i}=x_i + v_x (T_t-T_0), \qquad
y_{t,i}=y_i + v_y (T_t-T_0).
$$
Image and BEV features are then aggregated by deformable attention and Adaptive Mixing. Training uses Hungarian matching with a composite cost
$$
\Phi(i,j)=w_c \Phi_c(i,j)+w_b \Phi_b(i,j)+w_r \Phi_r(i,j),
$$
where the radian term
$$
\Phi_r(i,j)
=
\left|
\left(
|\theta_i-\theta'_j|+0.5
\right)\bmod 1 - 0.5
\right|
$$
encourages assignments along the same ray.

The reported results validate the design at the detection level. On the nuScenes test set with V2-99 at $1600\times 640$, RayFormer reaches $55.5\%$ mAP and $63.3\%$ NDS. On validation, ResNet50 at $704\times 256$ and $T=8$ gives $45.9\%$ mAP and $55.8\%$ NDS, while ResNet101 at $512\times 1408$ and $T=8$ gives $51.1\%$ mAP and $59.4\%$ NDS. The single-frame ResNet50 setting improves to $35.0\%$ mAP and $42.0\%$ NDS. Ablations attribute gains to image ray sampling, BEV ray sampling, and 2D-guided foreground queries; multi-frame input reduces mAVE from $0.800$ to $0.267$ when $H=7$ historical frames are used. In this domain, Pixel-as-Ray does not collapse directionality; it explicitly preserves it to reduce feature overlap.

## 5. Rendering, particle tracing, and space-time sampling

In graphics, Pixel-as-Ray is often literal. A massively parallel OpenCL implementation maps each work item to one ray, traces sphere intersections independently, and progressively accumulates radiance through repeated kernel launches [1504.03151]. The rendering equation is used in the standard differential form,
$$
L_o(x,\omega_o)
=
L_e(x,\omega_o)
+
\int_{H^2}
f_r(x,\omega_i,\omega_o)\,
L_i(x,\omega_i)\,
(n\cdot \omega_i)\, d\omega_i,
$$
while recursion is replaced by iterative depth-limited loops because OpenCL kernels do not support recursion. On an ATI Radeon HD 6750M, the first local-illumination render at $640\times 480$ takes $0.508$ seconds, the image improves after $6$ seconds of accumulation, and global illumination at recursion depth $6$ is generated in about $20$ seconds.

A more specialized realization is 3D Gaussian Ray Tracing, which casts one primary ray per pixel, traverses a BVH built over per-particle proxy meshes, and composites semi-transparent Gaussian particle contributions in front-to-back depth order [2407.07090]. Each anisotropic Gaussian has mean $\mu$, covariance $\Sigma$, opacity scale $\sigma$, and view-dependent radiance $c(d)$. Hardware returns mesh-hit distance $t_{\mathrm{hit}}$, but the Gaussian is evaluated at the maximum-response point
$$
t_{\max}
=
\frac{(\mu-o)^T \Sigma^{-1} d}{d^T \Sigma^{-1} d},
$$
with opacity
$$
\alpha_i = 1 - e^{-\sigma_i G_i(x_i)}.
$$
The implementation uses a $k$-buffer with $k=16$, early termination thresholds $T_{\min}=0.001$ during training and $T_{\min}=0.03$ at inference, and proxy meshes derived from stretched regular icosahedra. On MipNeRF360, the ray tracer matches or slightly exceeds rasterization quality, reporting PSNR $28.88$ versus $28.83$ for 3DGS, but it is slower for pinhole primary rays; FPS is $55$–$78$ versus $238$. With a generalized kernel of degree $n=2$, tracing becomes approximately $2\times$ faster at small quality cost.

Perceptual rasterization extends the same idea into space-time. Instead of one global frame time, each fragment defines a ray whose display time depends on screen location:
$$
t(u,v)= t_s + r((u,v))(t_e-t_s), \qquad r(x)=x\cdot d.
$$
Foveation is introduced by a nonlinear spatial mapping
$$
q(x)=x_f + \operatorname{norm}(x-x_f)\, p(\|x-x_f\|),
$$
and the fragment program performs per-fragment ray-triangle intersection against geometry interpolated to the fragment’s display time [1806.05385]. The resulting pipeline unifies rolling display timing and variable pixel density. In the Helicopter scene, rolling hull, adaptive, and Zenon bounds report $15.5\,\mathrm{ms}$, $5.8\,\mathrm{ms}$, and $4.1\,\mathrm{ms}$ respectively, while perceptual rasterization reaches SSIM $=1$ and warping gives approximately $0.70$.

An even more radical extension abandons frames altogether. Adaptive frameless rendering treats each display location as a receiver of asynchronously scheduled samples $(x,y,t)$, maintains a KD-tree tiling of the image plane, and streams individually scheduled rays through the scene according to spatial and temporal variation [2506.23001]. A deep buffer stores color, time, depth, and auxiliary sample data; the reconstructor performs adaptive space-time filtering at display refresh. At equal sampling rates of $100\text{k}$, $400\text{k}$, and $800\text{k}$ samples per second, adaptive frameless rendering reports $3$–$4\times$ lower RMS error than framed and nonadaptive frameless baselines, and parity with framed rendering at roughly $10\times$ fewer samples. Here Pixel-as-Ray no longer means one ray per frame pixel; it means one independently scheduled ray per space-time sample.

## 6. Lens, sensor, and radiance-level realizations

At the optical level, Pixel-as-Ray is tied to aperture structure and wavefront splitting. Lens imaging can be interpreted as a superposition of elemental images, each generated by a different point on the aperture and each corresponding to a camera-obscura-like view [1510.05992]. With thin-lens geometry,
$$
\frac{1}{f}=\frac{1}{d_o}+\frac{1}{d_i}, \qquad m=-\frac{d_i}{d_o},
$$
and for an aperture point $a$ the screen intersection behind the lens is
$$
y_S(a)
=
\left(1-\frac{d_s}{d_i}\right) y_A + \frac{d_s}{d_i} y_I.
$$
If $d_s=d_i$, the screen coordinate is independent of aperture point and elemental images coincide sharply; if $d_s\ne d_i$, the dependence on $y_A$ produces blur through misalignment of views. This formulation makes explicit that a sensor pixel gathers a bundle of aperture-conditioned rays rather than a single geometric line.

Dual-pixel simulation makes that bundle computationally explicit. Sdirt treats each depth-map pixel as a 3D object point $p$, samples the entrance pupil densely, traces rays through a multi-element lens by Snell’s law, and determines whether each ray falls into the left or right sub-pixel under a microlens [2503.11213]. The left/right PSFs are
$$
\mathrm{PSF}_L(i,j)=\sum_{k=1}^{n} A_k \,\delta_{L,k}(i,j),
\qquad
\mathrm{PSF}_R(i,j)=\sum_{k=1}^{n} A_k \,\delta_{R,k}(i,j).
$$
The implementation uses the Canon RF50mm F/1.8 lens, a single wavelength at $550\,\mathrm{nm}$, and $n=4096$ rays per object point. Microlens parameters obtained by grid search are
$$
h=0.78\,ps,\qquad f=1.44\,ps,\qquad w=0.60\,ps,\qquad r=0.50\,ps.
$$
To accelerate rendering, an MLP with $5$ hidden layers of $512$ neurons predicts the concatenated left/right PSFs with kernel size $ks=21$ at F/4. A full $512\times 768$ DP PSF map then takes approximately $0.395\,\mathrm s$ instead of approximately $801.5\,\mathrm s$ by brute-force ray tracing. Against real Canon RF50mm data, the simulator reports NCC $0.915$ and NSD $0.133$; on real DP119 depth estimation, planar scenes reach Acc-1 $0.9849$ and MAE $0.0845\,\mathrm m$.

A different sensor-side realization is Radiance Textures, where each texel stores an $n\times n$ radiance bucket indexed by incidence direction, so the atlas size becomes $W\cdot n \times H\cdot n$ for a base texture of size $W\times H$ [2301.01719]. Runtime shading computes view, reflection, refraction, or shadow directions, maps them into bucket coordinates through an azimuthal equisolid projection and disc-to-square transform, and samples precomputed directional radiance. The technique is explicitly described as having “the computational footprint of simple diffuse-only graphics” at the cost of storage memory footprint. In this setting, Pixel-as-Ray is implemented by offline ray tracing and online directional lookup: the rasterized pixel stands in for one or more rays whose angular dependence has been prebaked.

## 7. Recurrent limitations, misconceptions, and open problems

A recurring misconception is that Pixel-as-Ray always preserves directional information. In single-pixel X-ray tomography the opposite is true: $Kf(x)$ collapses all directions at $x$ into one scalar, so inversion depends on monotonicity, linearization, and small-data stability rather than direct angular recovery [2112.13978]. The same work states that removing or weakening the inverse Poincaré-type condition in the nonlinear lower bound is open, and that nonlinear stability for $K_g$ on simple manifolds is largely undeveloped.

Another misconception is that “ray-driven” and “pixel-driven” must be used as matched adjoints. The strong-convergence analysis shows that the practically common unmatched pair—ray-driven forward and pixel-driven backprojection—is the one justified under balanced resolutions, whereas ray-driven backprojection requires detector refinement faster than spatial refinement [2503.03069]. The convergence proved there is in the strong operator topology, not operator norm, so approximation quality remains input-dependent.

In scene understanding, ray-centric organization is not cost-free. RayFormer depends on accurate intrinsics and extrinsics, and severe occlusions, truncations, or inaccurate calibration can misguide ray selection and temporal wrapping; increasing $N_r$, $N_d$, $T$, and $P$ improves accuracy at greater memory and compute cost [2407.14923]. In graphics, per-ray independence simplifies implementation but induces divergence, load imbalance, and progressive noise; BVH rebuilds add overhead, and in Gaussian ray tracing dense overlapping particle clouds increase hit counts and reduce the speed advantage over rasterization [1504.03151][2407.07090].

Optical simulation introduces its own constraints. Sdirt currently requires a fixed-focus lens with available lens data and a DP sensor, approximates the microlens and sub-pixel structure, omits CFA pattern, demosaicing, read noise, and rolling shutter, and neglects or approximates diffraction and chromatic aberration [2503.11213]. Perceptual rasterization likewise assumes linear interpolation of model-view matrices and vertex motion over the frame; its prototype is slower than highly optimized fixed-function rasterization even though it outperforms generic ray tracing for the reported rolling and foveated tasks [1806.05385]. Adaptive frameless rendering reduces sample count, but it must control temporal noise, ghosting under reprojection, and depth-gating failures when older samples are reused [2506.23001].

These limitations do not define a single failure mode; they define the main design choice underlying the strategy. Treating pixels as rays gains geometric fidelity, optical correctness, or adaptive sampling flexibility, but it shifts difficulty toward inversion, calibration, ordering, memory, or reconstruction. Across tomography, detection, graphics, and optics, Pixel-as-Ray is therefore best understood as a geometric principle whose benefits depend on how each field manages that shift.

Source: https://www.emergentmind.com/topics/pixel-as-ray-strategy