AdaptiveISP: RL-based Task-Driven ISP

• AdaptiveISP is a real-time reinforcement learning-based image signal processor that optimizes module configuration for enhanced detection performance.
• It formulates ISP configuration as a Markov decision process, leveraging a lightweight RL agent and a frozen detector like YOLO-v3 as the reward metric.
• AdaptiveISP achieves a +1.3 mAP gain over handcrafted pipelines by dynamically switching modules to balance runtime efficiency and detection accuracy in varying environments.

AdaptiveISP is a real-time, reinforcement learning-driven image signal processor that dynamically selects and configures image-processing modules to optimize detection tasks, rather than perceptual image quality. Unlike traditional hand-engineered ISPs, which execute a fixed pipeline designed for human viewing, AdaptiveISP formulates the ISP configuration as a Markov decision process, employing a lightweight RL agent to determine the optimal sequence and parameters of processing steps on a per-image basis. This approach leverages a frozen, pre-trained object detector (e.g., YOLO-v3) as its reward metric, facilitating direct optimization for downstream machine vision performance in dynamic visual environments (Wang et al., 2024).

1. Background: ISP Pipeline Limitations and Task-Driven Motivation

Classic ISPs convert raw sensor data to sRGB imagery using cascaded modules such as denoising, demosaicing, color correction, gamma adjustment, sharpening, and other enhancements. These modules and their parameters are tuned for perceptual metrics, resulting in systematic over-processing of straightforward scenes and insufficient handling of challenging inputs (e.g., HDR or low light). Fixed pipelines cannot adapt online to scene variability; consequently, their efficacy for high-level computer vision tasks such as detection or segmentation is suboptimal. AdaptiveISP addresses these deficiencies by recasting ISP optimization as a task-centric RL-driven selection and configuration process (Wang et al., 2024).

2. ISP Module Set and Parametric Configurations

AdaptiveISP assumes a fixed raw-to-linear RGB preprocessor and operates exclusively in the differentiable sRGB domain. The modules under agent control include:

Module	Parameterization	Parameter Range/Constraints
Exposure Control	$I_{exposure}=I\cdot2^p$	$p\in[-3.5,3.5]$
White Balance	$[R';G';B']=\operatorname{diag}(p_R,p_G,p_B)[R;G;B]$	$p_i\in[e^{-0.5},e^{+0.5}]$
Color Correction	$[R';G';B']=P_{3\times3}[R;G;B]$	Rows of $P$ sum to $1$
Gamma Correction	$I_{gamma}=I^p$	$p\in[1/3,3]$
Denoising (NLM)	$I_{denoise}=\operatorname{NLM}(I,p)$	$p\in[0,1]$
Sharpen/Blur	$I_{sharp}=p\cdot I + (1-p)\cdot I_{blur}$	$p\in[0,2]$ (fixed $3\times3$ kernel blur)
Tone Mapping	Piecewise-linear (8 knots), sum of clipped slopes	$p_k \in [0.5,2.0]$
Contrast Adjustment	$I_{contrast}$ via cosine in luminance domain	$p\in[-1,1]$
Saturation	HSV conversion, S-channel adjustment	Form similar to contrast
Desaturation	$I_{desat}=(1-p)I+p(I_{lum},I_{lum},I_{lum})$	$p\in[0,1]$

This modular design enables per-image adaptation, reducing compute by activating only required processing stages (Wang et al., 2024).

3. RL Formulation: Model Structure and Reward Design

ISP configuration is posed as a finite-horizon Markov decision process $(S, A)$ with $T\leq10$ modules.

• State Space $S$: Each state $s_i$ comprises a downsampled image (e.g., $64\times64$), $N$ one-hot channels encoding applied modules, a channel denoting the current stage, and, for value computation, three global statistics: luminance, contrast, and saturation.
• Action Space $A$: Actions are $(a^M_i, a^\Theta_i)$, where $a^M_i$ selects a module and $a^\Theta_i$ provides its continuous parameters.
• Policy and Value Networks: Two CNNs (actor/critic, 4 Conv–BN–LReLU layers plus FC-128), with Softmax for module selection and Tanh-headed continuous parameter output.
• Reward Function: Immediate reward is the decrease in detection error for module $i$, evaluated by a frozen object detector:

$r_0(s_i, a_i) = D(s_i) - D(s_{i+1})$

where $D(\cdot)$ is the negative mAP of the detector. Regularizers include: - Module reuse penalty ($P_{reuse}$); - Entropy penalty ($P_e = \lambda_e\sum_m p(m)\log p(m)$; $\lambda_e$ decays during training); - Computational cost penalty ($P_c = \lambda_c \sum_m I_m M_c(m)$; $M_c(m)$ is module runtime);

The combined reward is:

$$r(s_i,a_i) = [D(s_i)-D(s_{i+1})] - [P_e + P_c + P_{reuse}]$$

and the global objective is $$R = \lambda_1\operatorname{mAP}(\theta) - \lambda_2 C(\theta)$$. Training uses on-policy actor-critic (A3C), with Adam optimizer ($3\times 10^{-5}$, batch size 8), and a discount factor $\gamma = 0.99$ (Wang et al., 2024).

4. Training Protocol and Adaptive Inference

Training proceeds for $100,000$ iterations on the LOD low-light dataset (24 hours, RTX 3090). Stages are capped at $T=5$, with terminated and truncated masks (based on bounds for the image mean and maximum stage limit). No $\epsilon$-greedy, exploration is regulated via entropy regularization. At inference, the agent selects modules greedily (highest $\pi^M(s_i)$, parameters $\pi^\Theta(s_i)$), halting when reaching $T_{max}$ or reward falls below threshold. Execution time is $1.2\,$ms per stage on GTX1660Ti ($3\,$stages $=3.6\,$ms). Trade-off between detection accuracy and runtime is managed via $\lambda_c$—pipelines for simpler scenes favor low-cost operations, more difficult inputs invoke multi-stage processing and heavier modules (Wang et al., 2024).

5. Quantitative Performance and Efficiency Trade-offs

Empirical evaluation on LOD and cross-dataset settings manifests consistent performance improvements:

Method	[email protected] (LOD)
Baseline Handcrafted ISP	70.1
Attention-aware Dynamic	70.9
ReconfigISP	69.4
RefactoringISP	68.3
AdaptiveISP	71.4

AdaptiveISP delivers a +1.3 absolute gain versus the best prior pipeline. Cross-dataset transfer (OnePlus, raw COCO) provides $1$–$2$ point increases in mAP. Segmentation results (raw COCO) show a $+0.6$ [email protected] gain over the next best non-adaptive method. Adjusting $\lambda_c$ for run-time efficiency can reduce mean per-sample time by $-22\%$ (from $14.7$ms to $11.5$ms) with minimal loss ($-0.4$ [email protected]). Module usage shifts accordingly: expensive modules (Sharpen/Blur, ToneMapping) become infrequent in favor of Exposure, CCM, and WhiteBalance. AdaptiveISP saturates in accuracy after 3–4 stages—fixed pipelines require all 10 stages for similar detection output but cannot adaptively terminate (Wang et al., 2024).

6. Qualitative Adaptation: Scene-Specific Pipeline Behaviors

Qualitative analysis reveals that in normal lighting, the agent prioritizes CCM to address minor color shifts. For low-light and high-ISO scenes, Desaturation is applied first to mitigate color noise; in high-dynamic-range scenarios, Tone Mapping is used early to manage highlights. Sharpen/Blur is employed selectively to enhance edge contrast for detection; denoising is rarely chosen, implying limited impact on object detection for typical scenes. These adaptive behaviors support the hypothesis that most images require minimal ISP intervention, while only a subset benefit from elaborate processing (Wang et al., 2024).

7. Significance, Implications, and Outlook

AdaptiveISP is distinguished as the first RL-based ISP framework that escapes fixed-stage, quality-centric optimization by integrating per-image scene analysis, module selection, and parameterization aimed at detection objectives. It enables real-time adaptation to environmental variability, learning when and how much to process, and jointly balancing latency and accuracy through explicit cost-aware RL rewards. A plausible implication is that such architectures may generalize to other downstream tasks (e.g., segmentation, tracking) and diverse hardware environments. Extensions may explore multi-task policy conditioning, hardware-specific latency modeling, or hybrid pipelines combining hand-crafted and learned stages.

AdaptiveISP’s approach represents a marked departure from conventional image quality metrics, instead leveraging direct supervision from downstream vision modules to reformulate low-level image formation for high-level semantic robustness (Wang et al., 2024).