Stochastic Primal-Dual Hybrid Gradient Algorithm with Arbitrary Sampling and Imaging Applications (1706.04957v2)

Abstract: We propose a stochastic extension of the primal-dual hybrid gradient algorithm studied by Chambolle and Pock in 2011 to solve saddle point problems that are separable in the dual variable. The analysis is carried out for general convex-concave saddle point problems and problems that are either partially smooth / strongly convex or fully smooth / strongly convex. We perform the analysis for arbitrary samplings of dual variables, and obtain known deterministic results as a special case. Several variants of our stochastic method significantly outperform the deterministic variant on a variety of imaging tasks.

Overview of the Stochastic Primal-Dual Hybrid Gradient Algorithm

The paper "Stochastic Primal-Dual Hybrid Gradient Algorithm with Arbitrary Sampling and Imaging Applications" proposes a stochastic extension to the primal-dual hybrid gradient (PDHG) algorithm. The original PDHG algorithm, introduced by Chambolle and Pock in 2011, is widely used to address convex-concave saddle point problems and has particular applications in imaging tasks. This stochastic adaptation aims to tackle similar problems more efficiently by employing a probabilistic approach for sampling, thereby reducing computational costs.

Key Contributions

The paper presents several significant contributions to the field of stochastic optimization:

• Generalization of Deterministic Case: The stochastic version is a direct extension of the deterministic PDHG algorithm. It recovers known deterministic results when all computations are performed in each iteration, making the stochastic version highly versatile.
• Improved Convergence Rates: The stochastic algorithm presents better convergence rates, especially when leveraging strong convexity assumptions, compared to previous models. This improvement is attributed to more flexible sampling strategies that are not strictly uniform.
• Arbitrary Sampling: The authors introduce a general framework for arbitrary samplings, expanding beyond fixed sampling schemes. This flexibility is advantageous in various scenarios, allowing for efficient sampling strategies tailored to specific optimization problems.
• Acceleration Techniques: The proposed algorithm features accelerations that significantly enhance convergence rates, particularly in cases of strong convexity. This upgrade translates convergence from $\mathcal{O}(1/K)$ to $\mathcal{O}(1/K^2)$ under certain conditions.

Technical Analysis

The paper performs an extensive analysis of the stochastic PDHG algorithm, covering various convex-concave scenarios including:

• General convex saddle point problems with no strong convexity assumptions.
• Semi-strongly convex problems where either the primal or dual component exhibits strong convexity
• Fully strongly convex problems where both primal and dual functions are strongly convex.

The results confirm that the algorithm achieves notable performance enhancements compared to its deterministic counterparts, especially in imaging applications.

Numerical Results and Imaging Applications

The paper details several imaging tasks to demonstrate the performance of the stochastic PDHG algorithm. The numerical experiments highlight that the stochastic method significantly outperforms traditional deterministic approaches in terms of speed and computational efficiency. Examples include:

• PET Reconstruction with Total Variation Regularization: The stochastic method rapidly converges to accurate solutions even with noise-prone datasets.
• TV Denoising and Deblurring: The algorithm maintains high performance with noise reduction tasks, showcasing its adaptability to varied imaging challenges.

Implications and Future Work

This work has profound implications for optimization practices in computational imaging and similar fields. Its success in reducing computational overheads while maintaining accuracy suggests applications could expand into machine learning and broader scientific computations. Future research may explore adaptive sampling, integrate iteration-dependent probabilities, and enhance parallel sampling strategies. These developments could further leverage the potential of stochastic optimization in practical scenarios.

The stochastic extension of PDHG represents a significant advance by fostering improved convergence rates and computational efficiency, illustrating the promising direction of stochastic methods in convex optimization and imaging applications.