POS-ISP: Pipeline Optimization at the Sequence Level for Task-aware ISP

Abstract: Recent work has explored optimizing image signal processing (ISP) pipelines for various tasks by composing predefined modules and adapting them to task-specific objectives. However, jointly optimizing module sequences and parameters remains challenging. Existing approaches rely on neural architecture search (NAS) or step-wise reinforcement learning (RL), but NAS suffers from a training-inference mismatch, while step-wise RL leads to unstable training and high computational overhead due to stage-wise decision-making. We propose POS-ISP, a sequence-level RL framework that formulates modular ISP optimization as a global sequence prediction problem. Our method predicts the entire module sequence and its parameters in a single forward pass and optimizes the pipeline using a terminal task reward, eliminating the need for intermediate supervision and redundant executions. Experiments across multiple downstream tasks show that POS-ISP improves task performance while reducing computational cost, highlighting sequence-level optimization as a stable and efficient paradigm for task-aware ISP. The project page is available at https://w1jyun.github.io/POS-ISP

• The paper introduces a sequence-level optimization framework that jointly learns a variable-length pipeline and image-adaptive parameters to enhance task-specific performance.
• It achieves significant improvements in object detection, instance segmentation, image enhancement, and depth estimation by addressing limitations of fixed pipeline orders and unstable RL methods.
• Empirical results demonstrate enhanced stability, improved runtime efficiency, and flexible accuracy-efficiency trade-offs, making it viable for real-world embedded and real-time vision applications.

POS-ISP: Sequence-Level Pipeline Optimization for Task-Aware ISP

Introduction

This work introduces POS-ISP, a novel framework for Image Signal Processing (ISP) pipeline optimization. In contrast to prior methods that either rely on fixed-length, stepwise, or reinforcement learning-driven approaches, POS-ISP formulates ISP design as a sequence-level optimization problem. The framework jointly learns both the sequence of modular ISP operations and their associated parameters, explicitly conditioning pipeline selection on the requirements of a downstream vision task. The paper provides comprehensive analysis and strong empirical results across multiple vision tasks—including object detection, instance segmentation, image enhancement, and depth estimation—demonstrating the method’s generalizability and stability.

Motivation and Limitations of Prior Work

Traditional ISP design, either in hardware-embedded pipelines or learned systems, typically follows a static ordering of operations such as denoising, white-balance, tone mapping, and color correction. This rigid structure is not optimally aligned with task-specific performance objectives. Recent methods attempt task-aware adaptation by either using deep reinforcement learning (DRL-ISP) or differentiable architectures with fixed module order (ReconfigISP, AdaptiveISP), but they are hindered by several factors:

• Instability from critic-driven RL, manifesting as oscillatory or divergent behaviors during training (as evidenced by pronounced spikes in critic loss trajectories).

  Figure 1: Training loss trajectories of critic networks in AdaptiveISP and DRL-ISP, illustrating significant instability due to bootstrapped critic errors in actor–critic training.

• Limited capacity to capture global pipeline interactions when optimizing stage-by-stage.
• Constrained flexibility due to fixed-length or uniform pipelines applied to all images, which precludes adaptation to task or environmental variations.

POS-ISP addresses these limitations by:

• Optimizing the entire sequence of ISP modules jointly via a stable sequence-level policy, avoiding actor–critic or bootstrapped value estimation.
• Enabling both dynamic module selection (including variable pipeline length via an <eos> token) and image-specific parameter adaptation.

Methodology

Pipeline and Sequence Learning

POS-ISP consists of two key components:

1. Sequence Predictor: A GRU-based autoregressive policy selects a variable-length sequence of ISP modules (e.g., exposure/gamma correction, denoising, blur, color correction, etc.), with an explicit termination token. Each training step samples module indices with a temperature-scheduled softmax, supporting dynamic exploration.
2. Parameter Predictor: A lightweight ConvMLP encoder predicts per-image parameter vectors for the selected modules, ensuring image-adaptivity. Conditioning is image-only, as sequence-conditioned predictors exhibit higher variance and reduced robustness.

The pipeline output is directly assessed against the downstream task objective (e.g., mAP for detection, RMSE for depth), and final-output rewards guide the joint optimization.

Module Pool

A rich pool of 10 ISP modules (exposure control, gamma, tone map, contrast, saturation/desaturation, white balance, denoise, sharpen/blur, color correction) supports diverse transformations, each parameterized and differentiable (see Implementation Details in the original paper for mathematical definitions).

Optimization Procedure

• Sequence-level optimization: The policy learns the most effective module orderings and length for each task, decoupling this optimization from unstable RL critics.
• Parameter adaptation: For each input image, parameters are predicted to suit scene content (e.g., lighting), as confirmed by clear domain separation in parameter histograms.

  Figure 2: Histogram comparisons of the predicted ISP parameters for images from different illumination domains, demonstrating the parameter predictor’s domain sensitivity.

• Hardware efficiency: The method can trade-off accuracy and latency by penalizing pipeline length during training, supporting real-world deployment scenarios.

Empirical Results

Object Detection and Instance Segmentation

POS-ISP consistently outperforms previous ISP optimization frameworks (DRL-ISP, ReconfigISP, AdaptiveISP) across LOD and LIS datasets. Margins are most pronounced in low-light (Dark) scenarios, with the following findings:

• On LOD-Dark detection, POS-ISP achieves up to 38.1 [email protected]:0.95 versus AdaptiveISP’s 36.8, with similar gains in segmentation.
• Variance across random seeds is substantially reduced, indicating reproducibility and stable convergence.

  Figure 3: Visual comparison of different ISP methods on object detection and instance segmentation tasks for LOD/LIS datasets.

• Visual differences show improved object/detail recoverability under challenging conditions.

Image Enhancement

On the Adobe FiveK benchmark (image retouching), POS-ISP achieves best-in-class full-reference metrics:

• PSNR: 23.11 ($\uparrow$)
• SSIM: 0.923 ($\uparrow$)
• LPIPS: 0.097 ($\downarrow$)

This surpasses AdaptiveISP and ablation studies confirm gains arise from joint sequence/parameter optimization, not merely longer pipelines.

Figure 4: Qualitative enhancement results and error maps for POS-ISP versus ground truth and competing methods, showing alignment with expert retouching.

Depth Estimation

Monocular depth prediction on KITTI shows POS-ISP exceeding baselines under varied loss objectives (RMSE, AbsDiff, AbsRel). For RMSE:

• AbsRel: 0.128 (POS-ISP) vs. 0.131 (AdaptiveISP)
• $\delta<1.25$: 0.847 (POS-ISP) vs. 0.840 (AdaptiveISP)

All error metrics are improved, and robustness is maintained across different backbone networks and losses.

Stability and Efficiency

• Optimization stability: Actor–critic methods (DRL-ISP, AdaptiveISP) exhibit unstable critic losses, while POS-ISP converges reliably.
• Runtime: On-device evaluation on a Galaxy S10 CPU (no quantization) shows a 4$\times$ speedup relative to AdaptiveISP when runtime penalty is applied. The framework enables explicit accuracy/efficiency tradeoff without significant quality loss.

Analysis and Ablation Studies

• Fixed vs. dynamic length: Even at fixed length, POS-ISP surpasses AdaptiveISP, highlighting that gains derive from the sequence-level optimization paradigm, not pipeline extensibility alone.
• Single sequence per task: Using a single optimized sequence (with image-adaptive parameters) provides sufficient flexibility and low worst-case performance drop, even across different lighting conditions.
• Task-awareness: Cross-task parameter swapping confirms that the sequence itself is crucial for downstream performance, not recoverable by parameter retraining alone, establishing the necessity of task-aware module ordering.

  Figure 5: Qualitative cross-domain parameter swapping on LOD, revealing that mismatched parameters lead to underexposure, supporting the adaptivity of POS-ISP’s parameter predictor.

Broader Implications and Theoretical Insights

POS-ISP unifies global sequence optimization with fine-grained parameter prediction, offering a flexible, stable, and task-adaptable ISP design paradigm. The sequence-level RL framework sidesteps the chronic instabilities of actor–critic RL observed in previous approaches, establishing a robust foundation for practical deployment. The empirical evidence that a single optimized sequence suffices for a broad domain, provided its parameters remain per-image adaptive, highlights a practical design avenue for ISP in embedded and edge systems.

The methodology’s capacity for runtime-aware optimization, backbone-agnosticity, and reliable performance across loss functions establishes its relevance both as a research tool and for integration into real-time vision systems.

Conclusion

POS-ISP presents a principled and practical approach to ISP pipeline learning, centered on sequence-level policy optimization and per-image parameter adaptation. Empirical results demonstrate clear performance improvements on diverse vision tasks, enhanced stability over RL-based alternatives, and flexible efficiency/accuracy trade-offs for deployment. The findings further elucidate the importance of task-focused module ordering and adaptive parameterization in ISP. Future research may explore integration with automated NAS for ISP, broader module sets, or cross-modal task optimization, deepening the synergy between computational photography and downstream perception.