---
title: Camera Passthrough Enhancement
url: https://www.emergentmind.com/topics/camera-passthrough-enhancement
type: topic
---

# Camera Passthrough Enhancement

Camera passthrough enhancement denotes the set of methods used to improve the quality, geometry, latency, and usability of camera-derived imagery in video see-through, augmented reality, virtual reality, and mixed reality systems. Recent work spans end-to-end RAW-to-RGB-and-feature encoding with integrated compression, learned real-time view synthesis from outward-facing stereo cameras, geometry-aware reprojection to the eye pose, low-light and HDR enhancement in RAW space, dual- and multi-camera fusion, and native on-device compositing pipelines [2411.10773] [2207.02186] [2502.11497] [2509.18929].

## 1. Scope and problem formulation

Camera passthrough is not merely a display problem. The literature treats it as a coupled sensing, imaging, geometry, and rendering problem in which the camera is displaced from the eye, the sensor output is degraded by optical and photometric artifacts, and the final imagery must satisfy real-time constraints. NeuralPassthrough explicitly formulates passthrough as a real-time view synthesis challenge because passthrough cameras cannot be physically co-located with the eyes, and identifies “flying pixels,” stretching, repeated textures, and temporal flicker as characteristic artifacts caused by inaccurate or sparse depth, especially for near-field and disoccluded objects [2207.02186]. Geometry Aware Passthrough frames the same issue as a fundamental camera-eye offset: direct passthrough renders the world from the camera’s origin, producing distorted scale, exaggerated motion parallax, and depth misperception, especially for near-field objects [2502.11497].

A second class of problems originates in the imaging stack itself. RealCamNet argues that fragmented ISP-plus-compression pipelines accumulate errors and duplicate computation across shading correction, demosaicing, denoising, tone mapping, and compression, while heterogeneous image artifacts remain spatially structured relative to the sensor [2411.10773]. RSE-RL makes the complementary point that many acquired camera images exhibit spatially heterogeneous artifacts, including moiré ringing, motion blur, color bleaching, and lens-based projection distortions, so a single global filter is often inadequate [2111.07499].

A common misconception in this area is that a raw or lightly rectified camera feed is an acceptable baseline. The comparative evidence does not support that view. Direct passthrough can retain lower warping error than depth-based reprojection, yet it produces substantially worse spatial reprojection error and significantly higher nausea, disorientation, and total cybersickness scores than geometry-aware alternatives [2502.11497]. Similarly, conventional cascaded ISPs can remain computationally redundant even when their individual modules are well understood [2411.10773]. This suggests that passthrough enhancement is best understood as a cross-layer optimization problem rather than a post-processing add-on.

## 2. Sensor-to-display imaging pipelines

A central axis of research concerns how much of the camera pipeline should remain modular and how much should be learned jointly. CameraNet proposes a two-stage decomposition in which Restore-Net operates in CIE XYZ and jointly handles denoising, demosaicking refinement, white balance, and artifact removal, while Enhance-Net operates in sRGB and performs color correction, tone mapping, contrast enhancement, and sharpening [1908.01481]. Its rationale is explicit: restoration operators do not strongly alter the data distribution, whereas enhancement operators do, so mixing these weakly correlated families in a single model degrades learning. The full data path is
$ I_{\mathrm{raw}}^{rgb} \rightarrow I_{\mathrm{raw}}^{xyz} \rightarrow I_{\mathrm{rest}}^{xyz} \rightarrow I_{\mathrm{rest}}^{srgb} \rightarrow I_{\mathrm{enh}}^{srgb} $,
with progressive training followed by joint fine-tuning [1908.01481].

RealCamNet takes the opposite position and collapses ISP and compression into a single end-to-end architecture. Its formal pipeline is
$ y = E(x; \phi) $,
$ \hat{y} = Q(y) $,
$ o = D(\hat{y}; \theta) $,
where $x$ is RAW, $o$ is reconstructed RGB, and $Q$ performs quantization and bitstream coding under an entropy model [2411.10773]. The encoder begins with UnpixelShuffle to stack Bayer RGGB into channels, applies Coordinate-Aware Distortion Restoration (CADR) before the rest of the ISP, extracts global and local color priors with CPE, and then uses Coordinate-Independent Mapping Compression (CIMC) to perform demosaicing, denoising, tone mapping, and feature redundancy compression [2411.10773]. The end-to-end objective is a standard Lagrangian rate–distortion loss,
$ \mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R $,
with $ R = - \mathbb{E}[\log_2 p(\hat{y})] $ under the learned entropy model [2411.10773].

Other pipelines operate even earlier in the capture loop. ACamera-Net predicts exposure and white balance directly from RAW through ACamera-Exposure and ACamera-Color, outputting an ISO distribution
$ y = \sum_{i=1}^{n} p_i b_i $
and RAW-domain color parameters that are converted to gains [2510.20550]. The reported model size is less than 2.3M parameters, and the paper reports inference latency under 20 ms on embedded platforms [2510.20550]. DarkDiff instead inserts a retasked latent diffusion model into the ISP after analytic preprocessing of Bayer RAW into brightened LRGB, using a residual VAE, region-based cross-attention, decoder-space reconstruction loss, and classifier-free guidance to perform a partial denoise plus LRGB-to-sRGB mapping [2505.23743].

These architectures encode different assumptions. The staged view isolates physically grounded restoration from display-oriented enhancement; the end-to-end view optimizes the whole RAW-to-display path under a unified objective; the parameter-prediction view leaves most of the ISP intact but changes its operating point; the generative view imports a large pretrained image prior into the low-light ISP regime. The coexistence of these paradigms indicates that “enhancement” can refer either to reconstructing better pixels or to choosing better camera parameters before pixels are reconstructed.

## 3. Geometry-aware reprojection and view synthesis

In head-mounted passthrough, geometric correctness is a first-order requirement rather than a cosmetic refinement. Geometry Aware Passthrough uses stereo RGB, device tracking, and per-pixel depth to reconstruct the scene and synthesize the view from the user’s current head or eye pose. Its core camera model is the pinhole projection
$ s p = K [R \mid t] P $,
with stereo depth recovered as
$ Z = \frac{fB}{d} $,
and the reprojected display image computed from the transformed 3D point cloud [2502.11497].

The paper’s benchmark separates spatial reprojection error from warping error. Averaged over 9 datasets of approximately 3000 frames each, direct passthrough yields a spatial reprojection error of Left $3.19 \pm 2.56$ px and Right $3.15 \pm 2.32$ px, whereas GAP – Depth (Smooth) yields Left $0.68 \pm 0.51$ px and Right $0.75 \pm 0.53$ px [2502.11497]. In the user study, the same geometric correction produced significantly lower condition-specific SSQ deltas than direct passthrough: Nausea $25.44 \pm 22.82$ for DP versus $14.71 \pm 20.05$ for GAP, Disorientation $37.70 \pm 47.96$ versus $18.56 \pm 34.72$, and Total Severity $32.57 \pm 36.79$ versus $19.17 \pm 29.97$ [2502.11497]. At the same time, GAP increases warping residuals on rigid planar targets: mean warping error is $0.96 \pm 0.08$ px for DP and $2.71 \pm 0.95$ px for GAP [2502.11497]. This trade-off is central to the field: correcting egocentric geometry can improve comfort even when it introduces more local warping near depth discontinuities.

NeuralPassthrough addresses the same problem with a learned real-time stereo synthesis pipeline. After rectification and RAFT-Stereo disparity estimation, it sharpens RGB-D edges, forward-splats both camera views into the eye view with occlusion-aware softmax weights, filters disocclusions, and applies a lightweight fusion U-Net [2207.02186]. The stereo relation is used in the conventional form
$ d = \frac{bf}{Z} $,
$ Z = \frac{bf}{d} $,
and the forward splat weights are depth-dependent [2207.02186]. On synthetic dynamic scenes, the reported quality is PSNR $30.74$ versus $27.38$, SSIM $0.9579$ versus $0.8818$, and ST-RRED $51.78$ versus $105.74$ when compared to MPI [2207.02186]. The total runtime at $1280 \times 720$ per eye on a desktop with dual Titan V GPUs is 32 ms per frame, with disparity estimation accounting for 24.0 ms and color reconstruction plus fusion accounting for 7.3 ms [2207.02186].

The geometric literature also shows that camera placement itself is part of enhancement. NeuralPassthrough models disocclusion width as
$ \beta = \max\left(0,\ \tan\left(\frac{\phi}{2}\right) - \delta\right)\cdot\left(\frac{d_f}{d_n} - 1\right) $,
and argues that using a stereo camera baseline wider than the user’s IPD can reduce disocclusions when rendering from the eyes [2207.02186]. This suggests that enhancement may begin with hardware geometry rather than software reconstruction alone.

## 4. Photometric restoration, low-light imaging, and dynamic range

A second major branch of passthrough enhancement concerns photometric fidelity under real sensor constraints: lens shading, dark shading, low light, backlighting, HDR scenes, and day–night operation.

RealCamNet treats coordinate-dependent optical distortions as sensor-fixed fields that should be corrected before demosaicing and tone mapping. For a cropped RAW $x_s \in \mathbb{R}^{h \times w \times 4}$ taken from a full RAW of size $H \times W$ at offset $\langle m,n \rangle$, it computes absolute normalized coordinates
$ c^{a}_{i,j} = ((i+m)/H, (j+n)/W) $
and applies learned multiplicative restoration
$ x_o = x_h \cdot x_e $,
where $x_h = \mathrm{Conv}(x_s)$ and $x_e = \mathrm{ReLU}(\mathrm{Conv}(c_a))$ [2411.10773]. This targets vignetting, lens shading, and dark shading that are spatially fixed relative to sensor coordinates. On the collected real-world dataset of 4,507 RAW–RGB pairs at $6000 \times 4000$, with 450 held out for evaluation, the full model reports BD-Rate $-39.08\%$, BD-PSNR $+2.96$ dB, BD-MSSSIM $+1.64$ dB, BD-LPIPS $-0.0162$, and BD-$\Delta E$ $-1.1709$ relative to the PyNet+VTM anchor [2411.10773].

AFNet addresses a different regime: dynamic illumination with glare, high-beam headlights, color bleeding, and varying noise. Its two-stage design performs illumination balancing followed by restoration, and its Fourier spectrum-based adversarial framework uses both a PatchGAN discriminator and a Fourier discriminator operating on normalized FFT magnitude and phase [2204.01795]. The generator objective combines $\mathbb{L}_{L1}$, $\mathbb{L}_{MS\text{-}SSIM}$, $\mathbb{L}_{SCAL}$, and spatial and Fourier adversarial losses with weights $\lambda_{L1}=1$, $\lambda_{MS\text{-}SSIM}=1$, $\lambda_{SCAL}=0.01$, $\lambda_{P\text{-}ADV}=0.5$, and $\lambda_{F\text{-}ADV}=0.5$ [2204.01795]. On LOL, AFNet reports PSNR $23.01$, SSIM $0.84$, NIQE $3.86$, LPIPS $0.27$, and 4.38 GMACs; on SID-Sony it reports PSNR $27.67$, SSIM $0.84$, NIQE $3.94$, LPIPS $0.37$, and 168.08 GMACs [2204.01795].

The HDR literature makes a different intervention by explicitly calibrating the camera response function and restricting fusion to the verified linear CRF zone. In the calibration-based minimalistic multi-exposure method, a 16-patch 90 dB target and histogram-based saturated-pixel detection are used to identify a linear zone from $V_{\min}=389$ to $V_{\max}=37486$, with an experimental linear dynamic range
$ \mathrm{LDRE} = 20\log_{10}(37486/389) \approx 39.66\ \mathrm{dB} $ [2101.05603]. Runtime HDR recovery then uses only two exposures, with
$ P_2 = 10^{\mathrm{LDRE}/20} \approx 100 $,
$ t_2 = P_2 t_1 $,
and per-exposure radiance estimates
$ L_i(x) = g^{-1}(z_i(x))/t_i $,
fused through linear-range masks [2101.05603]. The method reports robust recovery of a 78 dB target across a factor of 20 change in illumination, from 68 kLux down to 3.4 kLux [2101.05603].

Low-light enhancement also appears in spectral and generative forms. The VIS–NIR mixed-signal approach for 24-hour colorful imaging models the sensor response as a mixture of visible and NIR components and reports daytime PSNR $22.25$, SSIM $0.871$, Colorfulness $73.24$, and nighttime PSNR $17.17$, SSIM $0.835$, Colorfulness $89.74$, with runtime 4.03 ms on GPU [2005.04580]. DarkDiff, by contrast, reports the best LPIPS among compared methods on three low-light RAW benchmarks, with SID LPIPS $0.186$, ELD LPIPS $0.150$, and LRD LPIPS $0.103$, while remaining competitive in PSNR and SSIM [2505.23743]. These results indicate that low-light passthrough is increasingly evaluated not only by distortion metrics but also by perceptual detail recovery.

## 5. Multi-camera fusion, compositing, and viewpoint design

Passthrough enhancement increasingly exploits multiple cameras, multiple exposure regimes, or multiple viewpoints rather than relying on a single monocular feed.

In low light, “Dual-Camera Joint Deblurring-Denoising” uses a synchronized burst of short-exposure images from one camera and a simultaneous long exposure from another. Optical flow from the burst to the reference short exposure is computed with a pretrained PWC-Net, and those motion estimates are reused both for burst denoising and motion-aware deblurring of the long exposure [2309.08826]. The network is trained end-to-end with
$ \mathcal{L} = \mathcal{L}_1(J,G) + \lambda_{deblur}\mathcal{L}_1(\tilde{S},G) + \lambda_{denoise}\mathcal{L}_1(\hat{L},G) $,
where $\lambda_{deblur}=\lambda_{denoise}=0.25$ [2309.08826]. On synthetic dual-camera test data, the joint model reports PSNR $38.25$, SSIM $0.988$, and LPIPS $0.025$, outperforming LSD2, LSF, and D2HNet while using 17.4M parameters versus 82.3M for D2HNet [2309.08826].

In wide-plus-tele systems, “View Transition based Dual Camera Image Fusion” relaxes the assumption that output must remain in the wide view. It introduces a mixed view controlled by a transition ratio $r \in [0,1]$, using optical flow, large-kernel box filtering, ratio-constrained clipping, and pyramid blending to reduce occlusion while keeping the geometry change “not apparent to users” [2312.11184]. With view transition, reported occlusion area drops from $12.71\%$ to $4.27\%$ on OPPO72 and from $17.75\%$ to $4.28\%$ on CameraFusion [2312.11184]. The same paper reports no-reference quality improvements, including BRISQUE $17.87$, NIQE $4.20$, NRQM $6.08$, and PI $4.18$ on OPPO72 [2312.11184]. This is a direct example of a design controversy: preserving the nominal source view is not always optimal when the goal is to maximize the usable contribution of a higher-quality auxiliary camera.

Multi-camera fusion also appears in low-light 360-degree sensing. The LiDAR point cloud colourisation pipeline uses four cameras at $1280 \times 720$ and 30 FPS, per-channel linear colour correction to a colour checker reference, a brightness threshold $\tau = 0.12$ to trigger low-light enhancement, variance-of-Laplacian blur rejection at $150$, and temporal smoothing
$ I_t^{\mathrm{smooth}} = \eta I_t^{\mathrm{enh}} + (1-\eta) I_{t-1}^{\mathrm{smooth}} $
with $\eta \approx 0.6$ [2509.25859]. The reported average reprojection error for intrinsics calibration is approximately 0.35 px, extrinsic reprojection error is approximately 1.71 px, and the optimized software achieves real-time performance with nearly complete colourization of approximately $99.9\%$ [2509.25859]. Although the target application is colourised point clouds, the same modules—automatic calibration, colour correction, low-light enhancement, and overlap-aware fusion—transfer directly to passthrough mosaics.

Viewpoint design itself can be treated as an enhancement variable. FlexiCamAR augments head-fixed AR glasses with a finger-worn ring camera streaming $1920 \times 1080$ at 30 fps and end-to-end latency below 150 ms to a virtual screen overlay [2603.26012]. In a within-subjects study, FlexiCamAR reduced workload and improved usability relative to a head-camera baseline. In the QR task, mean completion time was 2.56 s for FlexiCamAR versus 2.95 s for the baseline, and SUS scores were 83.5 versus 60.4 in Follow-View and 78.5 versus 50.0 in Anchor-View [2603.26012]. This does not improve the captured image itself, but it improves the controllability of the capture viewpoint. A plausible implication is that passthrough enhancement sometimes requires changing where the camera is, not only how its output is processed.

## 6. Real-time deployment, evaluation, and limitations

The deployment literature shows that camera passthrough enhancement is constrained as much by thermal headroom and synchronization as by algorithmic quality. A quantitative feasibility study of native MR compositing on Meta Quest 3 models an on-headset pipeline built from the experimental Passthrough Camera API, on-device segmentation with Unity Sentis and FastSAM, temporal smoothing
$ \alpha_t = (1-\lambda)\alpha_{t-1} + \lambda \hat{\alpha}_t $,
and a multi-pass compositor [2509.18929]. At 720p30, the total modeled latency is approximately 31–32 ms within a 33.3 ms frame budget, corresponding to approximately 95% utilization and approximately 5% thermal headroom on Snapdragon XR2 Gen 2; the study further estimates only 5–10 minutes before thermal throttling when recording, while next-generation SoCs such as Snapdragon 8 Gen 3 leave approximately 34% headroom [2509.18929]. These values place an upper bound on how much enhancement can be inserted into a standalone pipeline without aggressive downsampling, duty cycling, ROI cropping, or quantization.

Several papers report concrete compute profiles that clarify the practical envelope. RealCamNet reports 49.01M parameters and 357G FLOPs in its full configuration, with average encode time 70.3 ms per image, decode time 59.2 ms per image, and 16.8 fps decoding throughput at $1024 \times 1024$ on an i9-13900K CPU plus RTX 4090 GPU [2411.10773]. NeuralPassthrough reports 32 ms per frame at $1280 \times 720$ per eye on dual Titan V GPUs, with 24.0 ms devoted to RAFT-Stereo and 7.3 ms to color reconstruction [2207.02186]. ACamera-Net reports under 20 ms inference latency on embedded platforms and real-time feasibility at 30 fps, with 60 fps passthrough described as practical with pipelined processing and NPU deployment [2510.20550]. Taken together, these results show that real-time feasibility depends strongly on whether the method is parameter prediction, feed-forward enhancement, joint compression, or iterative synthesis.

Generalization across sensors remains a persistent issue. Few-shot domain adaptation for low-light RAW enhancement shows that only a handful of labeled target samples can close much of the domain gap. Using a source camera dataset and a small target set, the method reports 30.30 dB PSNR and 0.913 SSIM on Nikon with four target shots, and 33.22 dB PSNR and 0.896 SSIM on Canon with six target shots, including a 0.9 dB PSNR improvement over full Canon training [2303.15528]. This suggests that deployment quality is not determined solely by model capacity; camera-domain adaptation and calibration are equally important.

The failure modes reported across the literature are consistent. RealCamNet notes under-correction under extreme vignetting or sensor saturation, degradation in ultra-low light, sensitivity to incorrect crop offsets or wrong camera resolution in absolute-coordinate restoration, and bitrate–distortion trade-offs controlled by $\lambda$ [2411.10773]. Geometry-aware methods note low-texture and reflective surfaces, depth discontinuities, disocclusions, and the latency–accuracy–stability trade-off in reprojection [2502.11497]. NeuralPassthrough identifies difficulties with view-dependent materials and complex geometries [2207.02186]. DarkDiff notes that diffusion ISPs remain slower than regression CNNs and that strong generative guidance can introduce hallucinated detail or poorer recovery for some text [2505.23743]. These limitations indicate that no single enhancement layer resolves all passthrough failures.

The field therefore converges on a multi-objective view of performance. Visual quality metrics such as PSNR, SSIM, LPIPS, NIQE, and BRISQUE remain important, but they do not capture on-display geometry, stereo comfort, thermal sustainability, or user-reported discomfort. Geometry Aware Passthrough explicitly shows that 3D depth MAE alone does not predict on-display geometric accuracy, and recommends prioritizing spatial reprojection error and warping residuals [2502.11497]. The on-device compositing literature, meanwhile, makes clear that utilization, bandwidth, encode power, and sustained runtime are first-class evaluation variables [2509.18929]. Camera passthrough enhancement has consequently evolved into a joint discipline of computational imaging, view synthesis, real-time systems, and human factors rather than a narrow image enhancement problem.

Source: https://www.emergentmind.com/topics/camera-passthrough-enhancement