GPU Accelerated Primal Heuristics

• GPU accelerated primal heuristics are algorithmic strategies that leverage massive GPU parallelism to rapidly generate feasible or near‐optimal solutions for complex optimization tasks.
• They decompose critical operations—such as matrix updates, candidate evaluations, and projection routines—into parallel tasks to substantially reduce computation time.
• These methods achieve dramatic speedups and scalability across combinatorial, continuous, and mixed‐integer problems, impacting fields from logistics to robotics.

GPU accelerated primal heuristics are algorithmic strategies that leverage the massive parallelism of graphics processing units (GPUs) to speed up the computation of feasible or near-optimal solutions to combinatorial and continuous optimization problems. These heuristics address the bottlenecks of classic CPU-bound approaches by parallelizing key components, such as move evaluation, matrix operations, and propagation routines. The application scope spans quadratic assignment, mixed-integer programming, linear and conic programming, and various metaheuristics. The following sections provide a comprehensive survey of GPU-accelerated primal heuristics, synthesizing foundational concepts, algorithmic innovations, parallelization methodologies, performance metrics, and representative implementation challenges.

1. Mathematical and Algorithmic Foundations

GPU-accelerated primal heuristics are rooted in established optimization frameworks; the distinguishing feature lies in adapting these methods for parallel execution. For combinatorial problems such as the Quadratic Assignment Problem (QAP), the Simulated Annealing (SA) heuristic is defined by the cost function

$$C = \sum_{i=1}^N \sum_{j=1}^N A_{i,j} B_{p(i), p(j)},$$

with swap decisions governed by the acceptance criterion,

$$\text{Accept if }\delta < 0 \text{ or } e^{-{\delta}/{T}} > r,$$

where $T$ is the temperature and $r$ is a uniform random variable (Paul, 2012).

In convex programming domains, GPU-first-order methods such as Primal-Dual Hybrid Gradient (PDHG) and affine scaling transform LP or QP systems into forms amenable to matrix-vector operations, for example:

$$x^{t+1} \leftarrow \mathrm{proj}_X(x^t - \tau(c - K^{\top}y^t)), \qquad y^{t+1} \leftarrow \mathrm{proj}_Y(y^t + \sigma(q - K(2x^{t+1} - x^t)))$$

(Lu et al., 2023, Lu et al., 2 Jun 2025, Lin et al., 1 May 2025). The restart and adaptive acceleration mechanisms (Halpern, reflection, PID-controlled primal weight) further enhance convergence properties (Lu et al., 18 Jul 2025).

In discrete optimization, methods such as Lagrange decomposition with Binary Decision Diagrams (BDDs) encode feasible sets in a compact layered graph structure, supporting massively parallel shortest-path calculations for min-marginal evaluation and dual updates (Abbas et al., 2021).

2. GPU Parallelization Strategies

Successful GPU acceleration requires explicit decomposition of algorithmic tasks into massively parallel kernels. Key strategies across papers include:

• Matrix-wise parallel updates: For approaches such as the 4-matrix SA for QAP, threads update blocks of swap costs $\Delta_{i,j}$, with each thread processing a batch of $16$ elements to amortize overhead and exploit coalesced memory access (Paul, 2012).
• Simultaneous candidate evaluation: In metaheuristics (multi-start UBQP, 2opt/tabu QAP), thousands of candidate solutions or swap operations are evaluated in single kernel launches, capitalizing on dense linear algebra primitives (Lewis, 2017, Novoa et al., 2023).
• Dynamic parallelism: Tabu search instances, each owning their internal memory and state, launch child threads to analyze neighborhood moves concurrently (Novoa et al., 2023).
• Pipelined propagation and probing caches: In MILP heuristics, bound propagation and rounding moves are processed in bulk, prioritized by dynamic metrics such as slack consumption and constraint violation degree. The probing cache precomputes implications, notably for binary variables, accelerating early infeasibility detection (Çördük et al., 23 Oct 2025).
• Associative scans and recursive decomposition: In legged robot MPC, Riccati recursions are reformulated using associative scans, enabling logarithmic complexity in prediction horizon length and direct state-space parallelization (Amatucci et al., 9 Jun 2025).
• Adaptive step sizes and restarts: GPU-friendly restart criteria based on KKT or fixed-point error norms facilitate robust convergence and allow regular reset of kernel anchors, harmonizing parallel progress across primal and dual updates (Lu et al., 2023, Lin et al., 1 May 2025, Lu et al., 18 Jul 2025).

3. Heuristic Enhancements and Algorithmic Innovations

Recent advances encapsulate several interlocking enhancements tailored for GPU architectures:

• Restarted Halpern PDHG methods with reflection interpolate between current iterates and an anchor point, enabling aggressive yet stable step sizes that empirically halve or quadruple convergence runtime (Lu et al., 18 Jul 2025).
• PID-controlled primal weight adjustment maintains logarithmic balance between primal and dual progress, dynamically tuning the ratio via proportional-integral-derivative control and ensuring concurrent improvement in both spaces (Lu et al., 18 Jul 2025).
• Problem-adaptive parallelism: Projection schemes on cones (second-order, exponential, or trivial) use thread-, block-, or grid-wise strategies depending on dimensionality and constraint structure (Lin et al., 1 May 2025). For discrete problems, deferred min-marginal averaging in dual updates eliminates tight synchronization (Abbas et al., 2021).
• Heuristic function generation via reinforcement learning: Off-policy deep RL frameworks generate compiler heuristics directly embedded in the IR pass, enabling self-tuning and resilience under continuous compiler evolution (Colbert et al., 2021).

4. Performance Metrics and Scalability

Benchmarks across problem classes consistently evidence dramatic speedups and robust scalability:

• Order-of-magnitude improvements: SA for QAP achieves $50$–$100\times$ acceleration over efficient CPU implementations and up to $10,000\times$ faster than classical SA (Paul, 2012); 2opt/tabu solvers yield $33\times$–$63\times$ speedup and solution errors within $1\%$/$0.15\%$ of best known values (Novoa et al., 2023).
• LP/QP benchmarks: cuPDLP.jl and its derivatives solve nearly all MIPLIB LP relaxations, delivering $3.7\times$–$20\times$ acceleration vs. simplex and barrier (Gurobi) and $2\times$–$4\times$ over previous PDLP solvers (Lu et al., 2023, Lu et al., 18 Jul 2025, Lu et al., 2 Jun 2025).
• Large-scale scalability: PDHCG for QP preserves linear convergence, achieving $5\times$ faster convergence than rAPDHG and $100\times$ efficiency over classic methods for $10^7$-dimensional, $10^{10}$ nonzero problems (Huang et al., 25 May 2024).
• Heuristics for MIP: GPU-extended Feasibility Pump produces $221$ feasible solutions and an optimality gap of $22\%$ on MIPLIB2017, outperforming fix-and-propagate portfolios and CPU-based ELS approaches (Çördük et al., 23 Oct 2025).
• Legged robot MPC: Whole Body Dynamics MPC achieves $60\%$–$700\%$ runtime improvement over acados and crocoddyl, scaling control to $16$ robots with under $25$ ms latency (Amatucci et al., 9 Jun 2025).

Empirical results highlight robust stability, with techniques such as time-decayed Q-tables yielding zero significant regressions under year-long compiler code evolution (Colbert et al., 2021).

5. Implementation Challenges and Remedies

Critical obstacles and corresponding remedies include:

• Irregular memory access: Irregular index updates (e.g., via permutation vectors) lead to non-coalesced memory access. Maintaining auxiliary matrices (e.g., row/column swapped $B'$ for QAP) regularizes memory stride and enables contiguous block operations (Paul, 2012).
• Thread synchronization: Native synchronization is block-bound on most GPUs. Custom barriers and limiting kernel concurrency to the number of SMPs circumvent deadlocks and enable safe multi-block operations (Paul, 2012).
• Kernel launch overhead: Batched processing and combining multi-element updates within kernel launches minimize costly overhead, especially in move evaluation or bound propagation (Paul, 2012, Çördük et al., 23 Oct 2025).
• Projection operations for diverse cones: For conic solvers (PDCS/cuPDCS), bijection-based projection routines and adaptive parallelism guarantee efficiency across cone types and sizes (Lin et al., 1 May 2025).
• Memory transfers: Full problem storage and compute reside in GPU memory, minimizing slow host-device data movement; constant memory caches shared kernel arguments for efficient repeated use (Divakar, 2015, Lu et al., 2023).

6. Applications and Impact

GPU-accelerated primal heuristics facilitate efficient solution of:

• Combinatorial optimization: QAP (facility layout, logistics), UBQP (scheduling, assignment), structured binary integer programs (MAP inference, cell tracking) (Paul, 2012, Lewis, 2017, Abbas et al., 2021, Novoa et al., 2023).
• Continuous and conic programming: Large-scale LP/QP/SOCP/Exponential Cone applications (energy systems, finance, portfolio optimization, Fisher markets, Lasso regression) (Lu et al., 2023, Huang et al., 25 May 2024, Lin et al., 1 May 2025, Lu et al., 2 Jun 2025).
• Mixed-integer programming: Primal heuristic portfolios for MILP, particularly in energy dispatch, routing, and presolve pipelines, demonstrate orders-of-magnitude advances in finding feasible solutions for previously intractable large-scale models (Kempke et al., 12 Mar 2025, Çördük et al., 23 Oct 2025).
• Model predictive control and robotics: Real-time, high-dimensional MPC problems for legged robots and multi-agent systems benefit from parallel associative scan-based KKT solvers (Amatucci et al., 9 Jun 2025).
• Compiler autotuning: Reinforcement learning-based heuristic generation achieves consistent frame rate improvements (average 1.6%, up to 15.8%) across production GPU compilers, exhibiting generalization and stability under evolving compiler ecosystems (Colbert et al., 2021).

7. Prospects and Ongoing Research

Ongoing work investigates:

• Deeper hierarchical and nested parallelism, frequency-based memory, and expanded probing cache logic for further scalability and solution diversity (Novoa et al., 2023, Çördük et al., 23 Oct 2025).
• Hybrid CPU-GPU and distributed implementations to support massive datasets and further amortize kernel launch cost across clusters (Novoa et al., 2023).
• Metaheuristic synergy: Combining aggressive primal heuristics (FP, FJ, ELS) with robust GPU PDLP approximations in unified pipelines continues to improve both feasible solution count and objective gap (Çördük et al., 23 Oct 2025).
• Algorithmic generalization: Extending reinforcement learning-guided heuristics to continuous decision spaces and other compiler passes, and adaptive parameter selection for dynamic resource control (Colbert et al., 2021, Lu et al., 18 Jul 2025).

A plausible implication is that continued refinement of memory layout, restart strategies, and parallel projection will yield further reductions in runtime and unlock broader application to problems previously beyond reach for heuristic optimization.

---

In summary, GPU accelerated primal heuristics have transformed the landscape of large-scale optimization, demonstrating dramatic empirical performance gains across diverse problem classes. By fusing algorithmic innovation with detailed hardware-aware engineering, these methods provide powerful tools for practitioners seeking rapid, scalable, and high-quality approximate solutions.