Camera Passthrough Enhancement

• Camera passthrough enhancement is a set of methods that refines RAW-to-display pipelines to improve imagery quality and reduce visual artifacts in AR/VR systems.
• It integrates processes like denoising, tone mapping, and geometry-aware view synthesis to mitigate latency, warping errors, and cybersickness.
• Advanced techniques include multi-camera fusion and real-time deployment strategies to address low-light, HDR, and sensor artifact challenges.

Camera passthrough enhancement denotes the set of methods used to improve the quality, geometry, latency, and usability of camera-derived imagery in video see-through, augmented reality, virtual reality, and mixed reality systems. Recent work spans end-to-end RAW-to-RGB-and-feature encoding with integrated compression, learned real-time view synthesis from outward-facing stereo cameras, geometry-aware reprojection to the eye pose, low-light and HDR enhancement in RAW space, dual- and multi-camera fusion, and native on-device compositing pipelines (Xu et al., 2024, Xiao et al., 2022, Chemaly et al., 17 Feb 2025, Laghari et al., 23 Sep 2025).

1. Scope and problem formulation

Camera passthrough is not merely a display problem. The literature treats it as a coupled sensing, imaging, geometry, and rendering problem in which the camera is displaced from the eye, the sensor output is degraded by optical and photometric artifacts, and the final imagery must satisfy real-time constraints. NeuralPassthrough explicitly formulates passthrough as a real-time view synthesis challenge because passthrough cameras cannot be physically co-located with the eyes, and identifies “flying pixels,” stretching, repeated textures, and temporal flicker as characteristic artifacts caused by inaccurate or sparse depth, especially for near-field and disoccluded objects (Xiao et al., 2022). Geometry Aware Passthrough frames the same issue as a fundamental camera-eye offset: direct passthrough renders the world from the camera’s origin, producing distorted scale, exaggerated motion parallax, and depth misperception, especially for near-field objects (Chemaly et al., 17 Feb 2025).

A second class of problems originates in the imaging stack itself. RealCamNet argues that fragmented ISP-plus-compression pipelines accumulate errors and duplicate computation across shading correction, demosaicing, denoising, tone mapping, and compression, while heterogeneous image artifacts remain spatially structured relative to the sensor (Xu et al., 2024). RSE-RL makes the complementary point that many acquired camera images exhibit spatially heterogeneous artifacts, including moiré ringing, motion blur, color bleaching, and lens-based projection distortions, so a single global filter is often inadequate (Bajaj et al., 2021).

A common misconception in this area is that a raw or lightly rectified camera feed is an acceptable baseline. The comparative evidence does not support that view. Direct passthrough can retain lower warping error than depth-based reprojection, yet it produces substantially worse spatial reprojection error and significantly higher nausea, disorientation, and total cybersickness scores than geometry-aware alternatives (Chemaly et al., 17 Feb 2025). Similarly, conventional cascaded ISPs can remain computationally redundant even when their individual modules are well understood (Xu et al., 2024). This suggests that passthrough enhancement is best understood as a cross-layer optimization problem rather than a post-processing add-on.

2. Sensor-to-display imaging pipelines

A central axis of research concerns how much of the camera pipeline should remain modular and how much should be learned jointly. CameraNet proposes a two-stage decomposition in which Restore-Net operates in CIE XYZ and jointly handles denoising, demosaicking refinement, white balance, and artifact removal, while Enhance-Net operates in sRGB and performs color correction, tone mapping, contrast enhancement, and sharpening (Liang et al., 2019). Its rationale is explicit: restoration operators do not strongly alter the data distribution, whereas enhancement operators do, so mixing these weakly correlated families in a single model degrades learning. The full data path is $$I_{\mathrm{raw}}^{rgb} \rightarrow I_{\mathrm{raw}}^{xyz} \rightarrow I_{\mathrm{rest}}^{xyz} \rightarrow I_{\mathrm{rest}}^{srgb} \rightarrow I_{\mathrm{enh}}^{srgb}$$, with progressive training followed by joint fine-tuning (Liang et al., 2019).

RealCamNet takes the opposite position and collapses ISP and compression into a single end-to-end architecture. Its formal pipeline is $y = E(x; \phi)$, $\hat{y} = Q(y)$, $o = D(\hat{y}; \theta)$, where $x$ is RAW, $o$ is reconstructed RGB, and $Q$ performs quantization and bitstream coding under an entropy model (Xu et al., 2024). The encoder begins with UnpixelShuffle to stack Bayer RGGB into channels, applies Coordinate-Aware Distortion Restoration (CADR) before the rest of the ISP, extracts global and local color priors with CPE, and then uses Coordinate-Independent Mapping Compression (CIMC) to perform demosaicing, denoising, tone mapping, and feature redundancy compression (Xu et al., 2024). The end-to-end objective is a standard Lagrangian rate–distortion loss, $$\mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R$$, with $R = - \mathbb{E}[\log_2 p(\hat{y})]$ under the learned entropy model (Xu et al., 2024).

Other pipelines operate even earlier in the capture loop. ACamera-Net predicts exposure and white balance directly from RAW through ACamera-Exposure and ACamera-Color, outputting an ISO distribution

$y = \sum_{i=1}^{n} p_i b_i$

and RAW-domain color parameters that are converted to gains (Li et al., 23 Oct 2025). The reported model size is less than 2.3M parameters, and the paper reports inference latency under 20 ms on embedded platforms (Li et al., 23 Oct 2025). DarkDiff instead inserts a retasked latent diffusion model into the ISP after analytic preprocessing of Bayer RAW into brightened LRGB, using a residual VAE, region-based cross-attention, decoder-space reconstruction loss, and classifier-free guidance to perform a partial denoise plus LRGB-to-sRGB mapping (Zheng et al., 29 May 2025).

These architectures encode different assumptions. The staged view isolates physically grounded restoration from display-oriented enhancement; the end-to-end view optimizes the whole RAW-to-display path under a unified objective; the parameter-prediction view leaves most of the ISP intact but changes its operating point; the generative view imports a large pretrained image prior into the low-light ISP regime. The coexistence of these paradigms indicates that “enhancement” can refer either to reconstructing better pixels or to choosing better camera parameters before pixels are reconstructed.

3. Geometry-aware reprojection and view synthesis

In head-mounted passthrough, geometric correctness is a first-order requirement rather than a cosmetic refinement. Geometry Aware Passthrough uses stereo RGB, device tracking, and per-pixel depth to reconstruct the scene and synthesize the view from the user’s current head or eye pose. Its core camera model is the pinhole projection $y = E(x; \phi)$0, with stereo depth recovered as $y = E(x; \phi)$1, and the reprojected display image computed from the transformed 3D point cloud (Chemaly et al., 17 Feb 2025).

The paper’s benchmark separates spatial reprojection error from warping error. Averaged over 9 datasets of approximately 3000 frames each, direct passthrough yields a spatial reprojection error of Left $y = E(x; \phi)$2 px and Right $y = E(x; \phi)$3 px, whereas GAP – Depth (Smooth) yields Left $y = E(x; \phi)$4 px and Right $y = E(x; \phi)$5 px (Chemaly et al., 17 Feb 2025). In the user study, the same geometric correction produced significantly lower condition-specific SSQ deltas than direct passthrough: Nausea $y = E(x; \phi)$6 for DP versus $y = E(x; \phi)$7 for GAP, Disorientation $y = E(x; \phi)$8 versus $y = E(x; \phi)$9, and Total Severity $\hat{y} = Q(y)$0 versus $\hat{y} = Q(y)$1 (Chemaly et al., 17 Feb 2025). At the same time, GAP increases warping residuals on rigid planar targets: mean warping error is $\hat{y} = Q(y)$2 px for DP and $\hat{y} = Q(y)$3 px for GAP (Chemaly et al., 17 Feb 2025). This trade-off is central to the field: correcting egocentric geometry can improve comfort even when it introduces more local warping near depth discontinuities.

NeuralPassthrough addresses the same problem with a learned real-time stereo synthesis pipeline. After rectification and RAFT-Stereo disparity estimation, it sharpens RGB-D edges, forward-splats both camera views into the eye view with occlusion-aware softmax weights, filters disocclusions, and applies a lightweight fusion U-Net (Xiao et al., 2022). The stereo relation is used in the conventional form $\hat{y} = Q(y)$4, $\hat{y} = Q(y)$5, and the forward splat weights are depth-dependent (Xiao et al., 2022). On synthetic dynamic scenes, the reported quality is PSNR $\hat{y} = Q(y)$6 versus $\hat{y} = Q(y)$7, SSIM $\hat{y} = Q(y)$8 versus $\hat{y} = Q(y)$9, and ST-RRED $o = D(\hat{y}; \theta)$0 versus $o = D(\hat{y}; \theta)$1 when compared to MPI (Xiao et al., 2022). The total runtime at $o = D(\hat{y}; \theta)$2 per eye on a desktop with dual Titan V GPUs is 32 ms per frame, with disparity estimation accounting for 24.0 ms and color reconstruction plus fusion accounting for 7.3 ms (Xiao et al., 2022).

The geometric literature also shows that camera placement itself is part of enhancement. NeuralPassthrough models disocclusion width as $o = D(\hat{y}; \theta)$3, and argues that using a stereo camera baseline wider than the user’s IPD can reduce disocclusions when rendering from the eyes (Xiao et al., 2022). This suggests that enhancement may begin with hardware geometry rather than software reconstruction alone.

4. Photometric restoration, low-light imaging, and dynamic range

A second major branch of passthrough enhancement concerns photometric fidelity under real sensor constraints: lens shading, dark shading, low light, backlighting, HDR scenes, and day–night operation.

RealCamNet treats coordinate-dependent optical distortions as sensor-fixed fields that should be corrected before demosaicing and tone mapping. For a cropped RAW $o = D(\hat{y}; \theta)$4 taken from a full RAW of size $o = D(\hat{y}; \theta)$5 at offset $o = D(\hat{y}; \theta)$6, it computes absolute normalized coordinates

$o = D(\hat{y}; \theta)$7

and applies learned multiplicative restoration $o = D(\hat{y}; \theta)$8, where $o = D(\hat{y}; \theta)$9 and $x$0 (Xu et al., 2024). This targets vignetting, lens shading, and dark shading that are spatially fixed relative to sensor coordinates. On the collected real-world dataset of 4,507 RAW–RGB pairs at $x$1, with 450 held out for evaluation, the full model reports BD-Rate $x$2, BD-PSNR $x$3 dB, BD-MSSSIM $x$4 dB, BD-LPIPS $x$5, and BD-$x$6 $x$7 relative to the PyNet+VTM anchor (Xu et al., 2024).

AFNet addresses a different regime: dynamic illumination with glare, high-beam headlights, color bleeding, and varying noise. Its two-stage design performs illumination balancing followed by restoration, and its Fourier spectrum-based adversarial framework uses both a PatchGAN discriminator and a Fourier discriminator operating on normalized FFT magnitude and phase (Shyam et al., 2022). The generator objective combines $x$8, $x$9, $o$0, and spatial and Fourier adversarial losses with weights $o$1, $o$2, $o$3, $o$4, and $o$5 (Shyam et al., 2022). On LOL, AFNet reports PSNR $o$6, SSIM $o$7, NIQE $o$8, LPIPS $o$9, and 4.38 GMACs; on SID-Sony it reports PSNR $Q$0, SSIM $Q$1, NIQE $Q$2, LPIPS $Q$3, and 168.08 GMACs (Shyam et al., 2022).

The HDR literature makes a different intervention by explicitly calibrating the camera response function and restricting fusion to the verified linear CRF zone. In the calibration-based minimalistic multi-exposure method, a 16-patch 90 dB target and histogram-based saturated-pixel detection are used to identify a linear zone from $Q$4 to $Q$5, with an experimental linear dynamic range $Q$6 (Riza et al., 2021). Runtime HDR recovery then uses only two exposures, with $Q$7, $Q$8, and per-exposure radiance estimates $Q$9, fused through linear-range masks (Riza et al., 2021). The method reports robust recovery of a 78 dB target across a factor of 20 change in illumination, from 68 kLux down to 3.4 kLux (Riza et al., 2021).

Low-light enhancement also appears in spectral and generative forms. The VIS–NIR mixed-signal approach for 24-hour colorful imaging models the sensor response as a mixture of visible and NIR components and reports daytime PSNR $$\mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R$$0, SSIM $$\mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R$$1, Colorfulness $$\mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R$$2, and nighttime PSNR $$\mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R$$3, SSIM $$\mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R$$4, Colorfulness $$\mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R$$5, with runtime 4.03 ms on GPU (Lv et al., 2020). DarkDiff, by contrast, reports the best LPIPS among compared methods on three low-light RAW benchmarks, with SID LPIPS $$\mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R$$6, ELD LPIPS $$\mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R$$7, and LRD LPIPS $$\mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R$$8, while remaining competitive in PSNR and SSIM (Zheng et al., 29 May 2025). These results indicate that low-light passthrough is increasingly evaluated not only by distortion metrics but also by perceptual detail recovery.

5. Multi-camera fusion, compositing, and viewpoint design

Passthrough enhancement increasingly exploits multiple cameras, multiple exposure regimes, or multiple viewpoints rather than relying on a single monocular feed.

In low light, “Dual-Camera Joint Deblurring-Denoising” uses a synchronized burst of short-exposure images from one camera and a simultaneous long exposure from another. Optical flow from the burst to the reference short exposure is computed with a pretrained PWC-Net, and those motion estimates are reused both for burst denoising and motion-aware deblurring of the long exposure (Shekarforoush et al., 2023). The network is trained end-to-end with $$\mathcal{L}(\phi,\theta) = \mathbb{E}[D(rgt,o)] + \lambda R$$9, where $R = - \mathbb{E}[\log_2 p(\hat{y})]$0 (Shekarforoush et al., 2023). On synthetic dual-camera test data, the joint model reports PSNR $R = - \mathbb{E}[\log_2 p(\hat{y})]$1, SSIM $R = - \mathbb{E}[\log_2 p(\hat{y})]$2, and LPIPS $R = - \mathbb{E}[\log_2 p(\hat{y})]$3, outperforming LSD2, LSF, and D2HNet while using 17.4M parameters versus 82.3M for D2HNet (Shekarforoush et al., 2023).

In wide-plus-tele systems, “View Transition based Dual Camera Image Fusion” relaxes the assumption that output must remain in the wide view. It introduces a mixed view controlled by a transition ratio $R = - \mathbb{E}[\log_2 p(\hat{y})]$4, using optical flow, large-kernel box filtering, ratio-constrained clipping, and pyramid blending to reduce occlusion while keeping the geometry change “not apparent to users” (Cao et al., 2023). With view transition, reported occlusion area drops from $R = - \mathbb{E}[\log_2 p(\hat{y})]$5 to $R = - \mathbb{E}[\log_2 p(\hat{y})]$6 on OPPO72 and from $R = - \mathbb{E}[\log_2 p(\hat{y})]$7 to $R = - \mathbb{E}[\log_2 p(\hat{y})]$8 on CameraFusion (Cao et al., 2023). The same paper reports no-reference quality improvements, including BRISQUE $R = - \mathbb{E}[\log_2 p(\hat{y})]$9, NIQE $y = \sum_{i=1}^{n} p_i b_i$0, NRQM $y = \sum_{i=1}^{n} p_i b_i$1, and PI $y = \sum_{i=1}^{n} p_i b_i$2 on OPPO72 (Cao et al., 2023). This is a direct example of a design controversy: preserving the nominal source view is not always optimal when the goal is to maximize the usable contribution of a higher-quality auxiliary camera.

Multi-camera fusion also appears in low-light 360-degree sensing. The LiDAR point cloud colourisation pipeline uses four cameras at $y = \sum_{i=1}^{n} p_i b_i$3 and 30 FPS, per-channel linear colour correction to a colour checker reference, a brightness threshold $y = \sum_{i=1}^{n} p_i b_i$4 to trigger low-light enhancement, variance-of-Laplacian blur rejection at $y = \sum_{i=1}^{n} p_i b_i$5, and temporal smoothing

$y = \sum_{i=1}^{n} p_i b_i$6

with $y = \sum_{i=1}^{n} p_i b_i$7 (Ranasinghe et al., 30 Sep 2025). The reported average reprojection error for intrinsics calibration is approximately 0.35 px, extrinsic reprojection error is approximately 1.71 px, and the optimized software achieves real-time performance with nearly complete colourization of approximately $y = \sum_{i=1}^{n} p_i b_i$8 (Ranasinghe et al., 30 Sep 2025). Although the target application is colourised point clouds, the same modules—automatic calibration, colour correction, low-light enhancement, and overlap-aware fusion—transfer directly to passthrough mosaics.

Viewpoint design itself can be treated as an enhancement variable. FlexiCamAR augments head-fixed AR glasses with a finger-worn ring camera streaming $y = \sum_{i=1}^{n} p_i b_i$9 at 30 fps and end-to-end latency below 150 ms to a virtual screen overlay (Li et al., 27 Mar 2026). In a within-subjects study, FlexiCamAR reduced workload and improved usability relative to a head-camera baseline. In the QR task, mean completion time was 2.56 s for FlexiCamAR versus 2.95 s for the baseline, and SUS scores were 83.5 versus 60.4 in Follow-View and 78.5 versus 50.0 in Anchor-View (Li et al., 27 Mar 2026). This does not improve the captured image itself, but it improves the controllability of the capture viewpoint. A plausible implication is that passthrough enhancement sometimes requires changing where the camera is, not only how its output is processed.

6. Real-time deployment, evaluation, and limitations

The deployment literature shows that camera passthrough enhancement is constrained as much by thermal headroom and synchronization as by algorithmic quality. A quantitative feasibility study of native MR compositing on Meta Quest 3 models an on-headset pipeline built from the experimental Passthrough Camera API, on-device segmentation with Unity Sentis and FastSAM, temporal smoothing $y = E(x; \phi)$00, and a multi-pass compositor (Laghari et al., 23 Sep 2025). At 720p30, the total modeled latency is approximately 31–32 ms within a 33.3 ms frame budget, corresponding to approximately 95% utilization and approximately 5% thermal headroom on Snapdragon XR2 Gen 2; the study further estimates only 5–10 minutes before thermal throttling when recording, while next-generation SoCs such as Snapdragon 8 Gen 3 leave approximately 34% headroom (Laghari et al., 23 Sep 2025). These values place an upper bound on how much enhancement can be inserted into a standalone pipeline without aggressive downsampling, duty cycling, ROI cropping, or quantization.

Several papers report concrete compute profiles that clarify the practical envelope. RealCamNet reports 49.01M parameters and 357G FLOPs in its full configuration, with average encode time 70.3 ms per image, decode time 59.2 ms per image, and 16.8 fps decoding throughput at $y = E(x; \phi)$01 on an i9-13900K CPU plus RTX 4090 GPU (Xu et al., 2024). NeuralPassthrough reports 32 ms per frame at $y = E(x; \phi)$02 per eye on dual Titan V GPUs, with 24.0 ms devoted to RAFT-Stereo and 7.3 ms to color reconstruction (Xiao et al., 2022). ACamera-Net reports under 20 ms inference latency on embedded platforms and real-time feasibility at 30 fps, with 60 fps passthrough described as practical with pipelined processing and NPU deployment (Li et al., 23 Oct 2025). Taken together, these results show that real-time feasibility depends strongly on whether the method is parameter prediction, feed-forward enhancement, joint compression, or iterative synthesis.

Generalization across sensors remains a persistent issue. Few-shot domain adaptation for low-light RAW enhancement shows that only a handful of labeled target samples can close much of the domain gap. Using a source camera dataset and a small target set, the method reports 30.30 dB PSNR and 0.913 SSIM on Nikon with four target shots, and 33.22 dB PSNR and 0.896 SSIM on Canon with six target shots, including a 0.9 dB PSNR improvement over full Canon training (Prabhakar et al., 2023). This suggests that deployment quality is not determined solely by model capacity; camera-domain adaptation and calibration are equally important.

The failure modes reported across the literature are consistent. RealCamNet notes under-correction under extreme vignetting or sensor saturation, degradation in ultra-low light, sensitivity to incorrect crop offsets or wrong camera resolution in absolute-coordinate restoration, and bitrate–distortion trade-offs controlled by $y = E(x; \phi)$03 (Xu et al., 2024). Geometry-aware methods note low-texture and reflective surfaces, depth discontinuities, disocclusions, and the latency–accuracy–stability trade-off in reprojection (Chemaly et al., 17 Feb 2025). NeuralPassthrough identifies difficulties with view-dependent materials and complex geometries (Xiao et al., 2022). DarkDiff notes that diffusion ISPs remain slower than regression CNNs and that strong generative guidance can introduce hallucinated detail or poorer recovery for some text (Zheng et al., 29 May 2025). These limitations indicate that no single enhancement layer resolves all passthrough failures.

The field therefore converges on a multi-objective view of performance. Visual quality metrics such as PSNR, SSIM, LPIPS, NIQE, and BRISQUE remain important, but they do not capture on-display geometry, stereo comfort, thermal sustainability, or user-reported discomfort. Geometry Aware Passthrough explicitly shows that 3D depth MAE alone does not predict on-display geometric accuracy, and recommends prioritizing spatial reprojection error and warping residuals (Chemaly et al., 17 Feb 2025). The on-device compositing literature, meanwhile, makes clear that utilization, bandwidth, encode power, and sustained runtime are first-class evaluation variables (Laghari et al., 23 Sep 2025). Camera passthrough enhancement has consequently evolved into a joint discipline of computational imaging, view synthesis, real-time systems, and human factors rather than a narrow image enhancement problem.