Data-Driven Per-Pixel Remapping

• Data-driven per-pixel remapping is a technique that employs learnable functions to transform each pixel based on empirical data, correcting non-uniformities and non-linearities.
• It leverages methods such as dynamic convolutional kernels, basis expansions, and linear calibration to boost imaging accuracy and enhance photographic quality.
• This approach achieves efficiency and state-of-the-art performance across applications like X-ray sensor calibration, astronomical imaging, and adaptive CNN operations while maintaining low computational complexity.

Data-driven per-pixel remapping refers to a broad class of methodologies in computer vision, image processing, and sensor calibration where the transformation applied to each pixel (or, more generally, each spatial location or channel element) is parameterized and optimized using empirical data. These approaches are typically distinguished by their ability to model or calibrate pixelwise non-uniformities, non-linearities, or adaptivity—frequently surpassing traditional static or hand-designed mappings in accuracy, flexibility, or efficiency. The development and practical impact of data-driven remapping have been notable in imaging sensor calibration, digital photography, and adaptive neural network operations.

1. Mathematical and Algorithmic Foundations

The core principle of data-driven per-pixel remapping is the per-location learnable transformation, usually denoted as a function $f_\theta: \mathbb{R}^d \rightarrow \mathbb{R}^{d'}$ optimized to map the observed (raw or feature) value at position $p$ to a corrected or enhanced value, with $\theta$ representing learnable or fitted parameters. Typical functional forms include:

• Pixelwise lookup or curve functions: Assigning a remapping function per pixel or per channel, often implemented as 1D LUTs (curves) or higher-dimensional LUTs in color/image space (Cui et al., 11 Jan 2025).
• Dynamic convolutional kernels: Assigning a position-dependent filter kernel or feature transformation (Wang et al., 2021).
• Flexible, basis-expansion models: Modeling intra-pixel sensitivity or instrumental gain with per-pixel basis expansions and regularization (Hogg et al., 2013).
• Linear per-pixel calibration: For sensors, $E = a_i \mathrm{PHA} + b_i$, with $a_i$ (gain) learned per pixel from empirical spectra (Schneider et al., 2024).

Optimization is performed via supervised loss—either using ground-truth targets (e.g., enhanced images) or through maximizing likelihood with regularization to account for high parameter counts (Hogg et al., 2013, Cui et al., 11 Jan 2025).

2. Major Methodological Variants

Several distinct data-driven per-pixel remapping paradigms have emerged:

Framework	Remapping Domain	Function Class
Sensor calibration (Schneider et al., 2024)	Pulse height, raw charge, intensity	Pixelwise linear gain/offset
Photographic enhancement (Cui et al., 11 Jan 2025)	RGB pixel colors	Image-adaptive basis + 1D curves
Dynamic neural layers (Wang et al., 2021)	Feature maps (CNNs)	Local dynamic filter (atoms)
Explicit image calibration (Hogg et al., 2013)	Pixel intensities, sub-pixel location	Zonal/basis expansion, regularized

2.1 Per-pixel Linear Calibration

The extraction of per-pixel gain and offset, particularly in large CMOS or CCD sensors, is carried out using two-point (dual-line) fits to empirical histograms. This enables significant improvements in energy resolution, exemplified in X-ray sensor arrays where per-pixel gain correction reduces FWHM and approaches the Fano-limited resolution by $\sim$12 eV at 5.9 keV (Schneider et al., 2024).

2.2 Basis-Expanded Remapping and Flat-fielding

Flexible models express the per-pixel transformation as a sum across basis elements (e.g., zonal, polynomial splines), fitted using regularized maximum likelihood across many exposures. Crucially, for scientific imagers such as Kepler in drift-intensive operational modes, the diversity of point-source trajectories across pixels enables robust disambiguation of intra- and inter-pixel gain variation (Hogg et al., 2013).

2.3 Adaptive and Learned Convolutions

Architecture in deep learning adapts filters per spatial location using feature-derived parameters, typically via small local networks. This delivers strong intra-image adaptability and has been shown to outperform global-weight architectures in tasks with spatially heterogeneous patterns ( e.g., crowd counting, real-world denoising (Wang et al., 2021)).

2.4 Image-Adaptive Coordinate-based Remapping

A recent approach termed Image-Adaptive Coordinates (IAC) learns a linear, image-specific basis in color space, upon which independent 1D curves are applied. This combines the expressive power of 3D LUTs with the efficiency of curve-based remapping and has produced state-of-the-art results in exposure correction and color enhancement with minimal additional memory (Cui et al., 11 Jan 2025).

3. Training Objectives and Optimization

Training procedures are structured by the application:

• Sensor calibration: Loss functions minimize the difference between calibrated and known spectral line positions, with approaches including Gaussian peak fitting and Energy Calibration by Correlation (ECC). The latter is shown to be especially robust for low-statistics per-pixel spectra (Schneider et al., 2024).
• Image enhancement/adaptive mapping: Supervised losses combine pixelwise distances (e.g., smooth $L_1$ or $L_1$) and perceptual penalties (e.g., VGG-based) for enhanced output fidelity (Cui et al., 11 Jan 2025).
• Calibration/flat-field inference: Optimization involves block-coordinate descent alternating between solution of fluxes, PSF parameters, and flat-field coefficients, often exploiting sparse or regularized linear algebra (Hogg et al., 2013).

The high dimensionality of per-pixel parameterizations is controlled through heavy regularization, low parameterizations per function (e.g., $K=200$ for IAC curves), and statistical priors or Gaussian-process smoothness (Hogg et al., 2013, Cui et al., 11 Jan 2025).

4. Memory, Computational, and Statistical Efficiency

Remapping approaches vary widely in resource usage:

Method	Memory (per image/layer)	Per-pixel Compute
IAC (photographic) (Cui et al., 11 Jan 2025)	$\sim$40 K parameters + 600 floats	$p$0, $p$124 flops
3D LUT (Cui et al., 11 Jan 2025)	$p$2	$p$3, heavy memory
Dynamic conv. (ACDA) (Wang et al., 2021)	$p$4 intermediates	$p$5, linear with $p$6
Flat-fielding (Hogg et al., 2013)	$p$7 or polynomial terms per pixel	Dominated by block solve

Adaptive and curve-based methods dramatically reduce parameter and computational complexity relative to naive pixelwise or volumetric LUT schemes, often achieving comparable or better performance at two orders of magnitude smaller model footprint (Cui et al., 11 Jan 2025). Regularization strategies such as shared basis atoms, Tikhonov priors, and small atom-generation networks ensure tractability.

5. Applications and Experimental Results

Data-driven per-pixel remapping has shown state-of-the-art or close-to-physical-limit performance in:

• X-ray Sensor Calibration: Single-pixel event energy resolution reaching $p$8 eV, $p$95 eV from the Fano limit of silicon, with gain stability to 0.4% over $\theta$0 pixels (Schneider et al., 2024).
• Photographic Enhancement: On MIT-Adobe FiveK and similar benchmarks, IAC yields PSNR/SSIM of $\theta$1 with only $\theta$2 K parameters and $\theta$3s inference time, outperforming or matching previous work at 10–100$\theta$4 speed and $\theta$5 smaller model (Cui et al., 11 Jan 2025).
• Astronomical Imaging: Sub-pixel flat-field recovery residuals $\theta$6, photometric precision at $\theta$7 in crowded scenes, approaching photon-limited performance even under severe drift/jitter (Hogg et al., 2013).
• Dense Prediction/CNN Tasks: ACDA achieves lower error than static-weighted CNN baselines in crowd counting and real-image denoising (e.g., SIDD PSNR $\theta$8 vs $\theta$9, UCF-QNRF MAE $E = a_i \mathrm{PHA} + b_i$0 vs $E = a_i \mathrm{PHA} + b_i$1) (Wang et al., 2021).

These results collectively demonstrate that data-driven per-pixel remapping, appropriately regularized and parameterized, can operate at or near the empirical and statistical limits of calibration, enhancement, and adaptive representation across diverse imaging modalities.

6. Implementation Considerations and Practical Guidelines

Successful deployment of these methods depends on:

• Statistical sufficiency: Sufficient per-pixel event counts or diverse exposures are critical; ECC is favored for $E = a_i \mathrm{PHA} + b_i$2100 events/pixel, while curve or basis expansion models require coverage of local input space by data (Schneider et al., 2024, Hogg et al., 2013).
• Compact parameterization: Methods such as curve discretization (e.g., $E = a_i \mathrm{PHA} + b_i$3), low-rank atom bases, or heavy weight sharing are necessary to avoid computational and memory bottlenecks (Wang et al., 2021, Cui et al., 11 Jan 2025).
• Regularization: Gaussian-process, Tikhonov, or kernel-based priors counteract overfitting in under-sampled pixels or subcells (Hogg et al., 2013).
• Hardware and data constraints: Real-time requirements and massive sensor arrays mandate event-based preprocessing and efficient intermediate data storage/reduction pipelines (Schneider et al., 2024).

Adherence to these principles has enabled robust per-pixel remapping strategies for both high-throughput production and scientific-grade calibration pipelines.

7. Limitations and Prospects

While data-driven remapping approaches have dramatically improved per-pixel calibration and adaptive processing, limitations persist in ultra-low-statistics regimes, sensor modalities with strong occlusions, and tasks requiring highly nonlinear or spatially context-aware transformations. Advances in basis learning, Bayesian uncertainty quantification, and hierarchical regularization are promising directions for extending these techniques.

A plausible implication is that as sensor and imaging architectures become more complex and high-dimensional, data-driven per-pixel remapping—emphasizing parameter efficiency, statistical regularization, and hardware alignment—will remain central for bridging the gap between raw device response and physically/cognitively meaningful outputs (Hogg et al., 2013, Wang et al., 2021, Schneider et al., 2024, Cui et al., 11 Jan 2025).