ReconfigISP: Adaptive Signal Processing

• ReconfigISP is a modular framework for adaptive signal processing that dynamically reconfigures camera ISPs and reconfigurable intelligent surfaces to optimize performance in varied real-world conditions.
• In camera image processing, it employs a flexible pipeline with neural architecture search and proxy networks to significantly improve restoration and detection metrics while reducing latency by up to 20×.
• For electromagnetic surfaces, ReconfigISP leverages element permutation and efficient hardware design to achieve focused beamforming with negligible sidelobes and scalable power and control management.

ReconfigISP refers to a set of technological frameworks and algorithms for reconfigurable information signal processing, most notably in two domains: camera image signal processors (ISPs) and electromagnetic surfaces (reconfigurable intelligent surfaces, RISs). Across these settings, ReconfigISP denotes approaches enabling on-the-fly adaptation of processing pipelines or hardware-phase parameters to match diverse real-world requirements, tasks, and environments.

1. ReconfigISP in Camera Image Signal Processing

The ReconfigISP framework for imaging systems is a data- and task-adaptive pipeline for image restoration and computer vision, as formalized in "ReconfigISP: Reconfigurable Camera Image Processing Pipeline" (Yu et al., 2021). Rather than relying on a fixed sequence of hand-designed ISP modules, ReconfigISP employs a flexible supernet structure, gradient-based neural architecture search (NAS), and proxy networks for differentiable optimization and module parameterization.

Module Pool and Parametrization

The pipeline encompasses discrete or parameterized modules for denoising (Bilateral-Bayer, Median-Bayer, Non-Local Means, BM3D, path-based restoration), demosaicing (Laplacian, Nearest, Bilinear, DemosaicNet), color transformations (sRGB/white balance), global tone mapping (Reinhard, Filmic, Manual), and others. Each module is characterized by a small parameter set $p_j$; non-differentiable operations are replaced by CNN-based proxy networks $f_j(\cdot ; w_j)$, enabling backpropagation.

The overall pipeline is not fixed. Instead, for $K$ pipeline slots, each position $k$ can select from a set of valid modules $V_k$, relaxed via convex weights $\alpha_{k,j} \ge 0$ with $\sum_{j \in V_k} \alpha_{k,j} = 1$. Processing at step $k$ is

$$X_{k} = \sum_{j\in V_k} \alpha_{k,j} f_{j}(X_{k-1}, p_{k,j}; w_j)$$

At the end of the search, the most probable path is fixed by taking $a_k = \arg\max_j \alpha_{k,j}$.

Differentiable Proxy Training

Proxy networks are pre-trained to regress the outcome of their respective non-differentiable modules, minimizing

$$L_p = \mathbb{E}_{X, p_j} \| f_j(X, p_j; w_j) - \hat f_j(X, p_j) \|_1$$

and periodically fine-tuned during architecture search to mitigate distribution shift.

Neural Architecture Search and Optimization

The NAS is cast as a bi-level continuous optimization, adopting a DARTS-style alternating update:

• Model parameters $p$ are updated on training data.
• Architecture weights $\alpha$ are updated on validation data using a one-step lookahead.
• Efficiency is regulated through a latency penalty: $\mathrm{Lat}(p, \alpha)^\beta$ for tradeoff between accuracy and computational complexity. During search, candidates with $\alpha_{k,j} < \eta \cdot \max_j \alpha_{k,j}$ are pruned to maintain tractability.

Experimental Results

Empirical evaluation demonstrates strong image restoration and object detection on challenging real-world sensor data. For example, on the SID dataset, ReconfigISP achieves PSNR 25.65 and SSIM 0.7527 (raw RGB), surpassing fixed camera ISPs by over 10 dB in PSNR. For object detection (YOLOv3 frozen), the system yields mAP 0.601 on the OnePlus dataset, outperforming default Lightroom and traditional ISPs by a substantial margin. Runtime-optimized variants trade marginal quality for significant speedups ($\sim 20\times$ reduction) (Yu et al., 2021).

Comparison to deep ISPs (e.g., U-Net) reveals that ReconfigISP achieves superior parameter and data efficiency, retaining robustness even when only a small fraction of the dataset is available.

2. ReconfigISP in Reconfigurable Intelligent Surfaces (RISs)

ReconfigISP is also used to denote spatially-selective RIS architectures that achieve individually-configurable reflection properties. These electromagnetic implementations are primarily described in "Spatially Selective Reconfigurable Intelligent Surfaces Through Element Permutation" (Rusek et al., 2024).

Element-Permutation Model

The canonical RIS consists of an $M \times M$ grid ($N = M^2$ elements), where each sub-wavelength element imposes a programmable phase-shift on impinging EM waves, described as

$$A(k_{\text{in}}, k_{\text{out}}; C) = \left| a(k_{\text{out}})^\mathrm{T} C a(k_{\text{in}}) \right|^2$$

with $C$ a diagonal phase-shift matrix. Standard RISs are not spatially selective: any incident wave from direction $k'$ can be reflected into some direction $k''$ with full gain.

ReconfigISP introduces a fixed or reconfigurable permutation $\sigma$ of elements, so the signal is re-routed and re-radiated from a different element:

$$A(k_{\text{in}}, k_{\text{out}}; \sigma, C) = \left| a(k_{\text{out}})^\mathrm{T} P C a(k_{\text{in}}) \right|^2$$

where $P$ is the permutation matrix corresponding to $\sigma$.

When $\sigma$ is selected as a cyclic shift or similar pattern, spatial selectivity is achieved: for any $(k', k'') \ne (k_{\text{in}}, k_{\text{out}})$, the end-to-end gain $A$ is zero in the large-array limit, proven by disalignment of array response phases. This realizes a single sharply focused beam with negligible sidelobes outside the designed directions (Rusek et al., 2024).

Hardware Realizations

Three architectures are categorized:

1. Two-panel RIS with element re-wiring: Two back-to-back arrays whose elements are cross-connected, requiring only phase shifters and no circulators (insertion loss $\approx 0.5$–$1$ dB).
2. Single-panel RIS with circulator routing: Signals are routed from element to element via circulators; more complex, with higher insertion loss ($1$–$2$ dB).
3. Symmetric permutation: Restricted to reciprocal pairwise switches, avoids circulators but introduces a $4$ dB beamsplitting loss.

Hardware complexity and scalability are primarily governed by the number of required switches and phase shifters, with $O(N^2)$ scaling for fully reconfigurable crossbars, but $O(N)$ for fixed permutations.

Performance

Simulation shows that a permutation RIS achieves unit ($N^2$) gain exclusively in the configured direction pair, whereas the classical identity RIS lacks angular selectivity. For $M = 20$, $99\%$ of random permutations achieve maximum sidelobe gain $\tau < 0.1$, contrasting with $\tau = 1$ for standard RIS. This spatial filtering is robust and scalable (Rusek et al., 2024).

A summary of hardware and performance implications is provided in the table:

Architecture	Insertion Loss	Control Complexity	Scalability
Two-panel, rewiring	~0.5–1 dB	$O(N)$–$O(N^2)$	Excellent (as $N \to \infty$)
Circulator-based	+1–2 dB	$O(N^2)$	Moderate
Symmetric, no circ.	~4 dB (split)	$O(N)$	Excellent

3. Power and Control Considerations in ReconfigISP-Enabled RIS

The static and dynamic power budgets of reconfigurable electromagnetic surfaces require careful design, as detailed in "Static Power Consumption Modeling and Measurement of Reconfigurable Intelligent Surfaces" (Wang et al., 2023).

Power Model

The total static power is

$$P_{\text{static}} = P_{\text{FPGA}} + N_{\rm drv} \cdot P_{\rm drv}$$

where FPGA power ($1.5$–$5$ W) dominates; drive circuit power depends on the type (PIN diode, varactor, RF switch), grouping, and driver capabilities. For example, in a $256$-element PIN diode RIS, the drive circuits consume only $\sim$2 mW, while varactor-based RISs can incur $>1$ W due to high-power analog circuits.

Design Implications

• Digital RISs (PIN diode/RF switch) minimize static power and are well suited for panel scaling.
• Varactor-based (analog) designs support fine phase resolution but at higher power cost.
• Grouping of elements and driver consolidation reduces control overhead without heavily sacrificing flexibility.
• Practical guidelines recommend minimal FPGA overprovisioning and driver sharing.

4. Joint Control and Power Transmission

RIS reconfiguration energy can be delivered wirelessly via combined control and power signaling, as analyzed in "Simultaneous Control Information and Power Transmission for Reconfigurable Intelligent Surfaces" (Kisseleff et al., 2021).

SWIPT Receiver Architectures

Four receiver splits are considered: time-sharing (TS), power splitting (PS), dynamic power splitting (DPS), and antenna selection (AS), each optimizing the number of reflectors updated given constraints on SNR and harvested energy.

Key closed-form results include:

• Optimal update count $L^*$ as a function of SNR, received power, and per-update energy requirement.
• For high SNR, PS and AS enable full update rates, while TS is more robust to power variability.
• SNR/energy trade-offs are critical, with design guidelines emphasizing regime-specific scheduling.

5. ReconfigISP in Photonic Signal Processing

In RF photonics, ReconfigISP principles are exemplified by "Fully Automatic In-Situ Reconfiguration of RF Photonic Filters in a CMOS-Compatible Silicon Photonic Process" (Shawon et al., 2022). Here, automated algorithms tune integrated ring resonator filters to arbitrary center frequency, bandwidth, and filter type using minimal reference measurements.

Algorithmic Flow

• The user specifies target parameters (center frequency, bandwidth, rejection, filter type).
• One-time in-situ pre-characterization measures each heater’s responsivity.
• Closed-loop tuning iteratively aligns ring couplings and phase biases, leveraging feedback from integrated photodiodes using two reference wavelengths.
• The system supports rapid, fully automatic reconfiguration, with configuration times projected under $300$ s.

Integration and Scalability

• Fabricated in standard silicon photonics, with full electronics interface, the method scales to higher filter orders and multi-parameter reconfiguration.
• The workflow eliminates dependence on complex instrumentation, supporting software-defined radio and agile RF front-end scenarios.

6. Comparative Perspective and Outlook

ReconfigISP frameworks—whether for imaging, EM wave control, or photonics—center on modular, parameter-efficient, and data-driven reconfiguration for diverse domains. Canonical attributes include:

• Automated or learning-based mapping from task/environmental descriptors to pipeline or hardware parameters.
• Substantial improvement over fixed, monolithic system designs in adaptation and efficiency.
• Modest overhead versus rigid systems due to minimal parametric growth (hundreds for camera ISP; $O(N)$ for element-permuted RIS).

A plausible implication is that, as support electronics, control mechanisms, and reconfigurable hardware platforms mature, ReconfigISP principles will generalize to heterogeneous sensor, communication, and signal processing environments where adaptivity and efficiency are dominant constraints.

References: (Yu et al., 2021, Rusek et al., 2024, Wang et al., 2023, Shawon et al., 2022, Kisseleff et al., 2021).