Augmented Lagrangian Solver for Embedded Optimization

• Augmented Lagrangian Solver is a numerical method that augments the standard Lagrangian with a quadratic penalty to smooth the dual function and accelerate convergence.
• The solver adapts primal and dual computations to fixed-point arithmetic, addressing quantization errors and using multiplier projection to prevent data overflow.
• A practical early stopping criterion based on local quadratic growth guarantees efficient convergence, making it ideal for resource-constrained embedded systems.

An augmented Lagrangian solver is a numerical optimization algorithm that solves constrained optimization problems by augmenting the classical Lagrangian with additional penalty terms, thereby smoothing the dual function and promoting faster or more reliable convergence. In the fixed-point arithmetic setting addressed by (Zhang et al., 2018), the solver is adapted for implementation on low-cost, low-power embedded devices, where computations are performed with finite-precision fixed-point numbers. This introduces specific challenges, especially data overflow and limited numerical precision, that require algorithmic modifications to guarantee robustness and convergence.

1. Formulation: Convex/Nonsmooth Objective with Linear Constraints

The principal problem class considered is

$$\min_{x \in \mathbb{R}^n} f(x) \quad \text{subject to} \quad A x = b, \quad x \in \mathcal{X}$$

where $f$ is convex (possibly nonsmooth), $A \in \mathbb{R}^{p \times n}$, $b \in \mathbb{R}^p$, and $\mathcal{X}$ is a convex feasible set (typically a box). The standard Lagrangian is

$$L(x; \lambda) = f(x) + \langle A x - b, \lambda \rangle,$$

and the augmented Lagrangian is defined by

$$L_\rho(x; \lambda) = f(x) + \langle A x - b, \lambda \rangle + \frac{\rho}{2} \|A x - b\|^2.$$

The dual function is

$$\Phi_\rho(\lambda) = \min_{x \in \mathcal{X}} L_\rho(x; \lambda).$$

The quadratic penalty term $\frac{\rho}{2}\|A x - b\|^2$ “augments” the standard Lagrangian, which both regularizes the dual function (making it smooth, with a Lipschitz gradient of constant $1/\rho$) and accelerates dual convergence, even in the absence of strict convexity.

2. Fixed-Point Arithmetic: Error Modeling and Constraints

Fixed-point arithmetic is dictated by constraints on data width (word length) and dynamic range; its advantage on embedded platforms comes at the price of quantization and a risk of overflow. In the solver considered, every operation in both the primal (minimization subproblem) and dual (multiplier update) is performed in fixed-point, leading to quantization errors. Two error terms are explicitly modeled:

• Primal error: $$L_\rho(\tilde{x}_k; \lambda_k) - L_\rho(x_k^*; \lambda_k) \leq \epsilon_\mathrm{in}^k$$, where $\tilde{x}_k$ is an inexact solution.
• Dual error: Associated with inaccuracies in the fixed-point update of the dual variable.

The solver prescribes using uniform bounds on these errors $B_\mathrm{in}, B_\mathrm{out}$ to ensure quantifiable convergence. Designing the number representation (bit-width) is performed to control these errors within budget and guarantee no data loss through overflow.

3. Multiplier Projection and Overflow Prevention

A key innovation is the use of a projection operation in the multiplier update to prevent dynamic-range overflow: $$\lambda_{k+1} = \Pi_D \left[ \lambda_k + \frac{1}{L}(A \tilde{x}_k - b + \epsilon_\mathrm{out}^k) \right]$$ where $\Pi_D$ denotes projection onto a compact bounding set $D$ (typically a box constructed to contain $\{0, 2\lambda^*, \lambda^* + 1\}$, with $\lambda^*$ the optimal multiplier). This ensures that the fixed-point representation of $\lambda$ never exceeds designated bounds, a crucial feature for robust embedded implementations, and is analytically justified by constructing $D$ so as to contain the entire trajectory of feasible dual iterates.

4. Inexactness, Convergence Rate, and Accuracy Bounds

The fixed-point ALM is inherently inexact, with error terms in both primal and dual updates. The dual update is viewed as an inexact projected gradient ascent on the dual function $\Phi_\rho$, leading to explicit suboptimality guarantees: $$\Phi_\rho(\lambda^*) - \Phi_\rho(\bar{\lambda}_K) \leq \frac{L}{2K}\|\lambda_0 - \lambda^*\|^2 + \delta,$$ where $L = 2/\rho$ and $\delta$ collects accumulated primal/dual errors.

For the primal iterate average $\bar{x}_K$, optimality and feasibility gaps are bounded as: $$-\left( \frac{\varphi^1(2\lambda^*)}{K} + E \right) \leq f(\bar{x}_K) - f(x^*) \leq \frac{\varphi^1(0)}{K} + E,$$

$$\|A \bar{x}_K - b\| \leq \frac{1}{K}\varphi^1\left(\lambda^* + \frac{r(\bar{x}_K)}{\|r(\bar{x}_K)\|}\right) + E,$$

with Lyapunov merit function $\varphi^1$, $r(\cdot)$ is the primal residual, and $E$ a bound in terms of the error budgets $B_\mathrm{in}, B_\mathrm{out}, B_\lambda$. This ensures convergence to an $O(B_\mathrm{in} + B_\mathrm{out} + B_\lambda)$-accurate solution—the radius of the primal/dual gaps is controlled by the precision settings.

5. Iterative Solvers and Embedded-Friendly Early Stopping

In practice, the subproblem $\min_{x \in \mathcal{X}} L_\rho(x; \lambda)$ is often solved using iterative methods (e.g., projected gradient). Waiting for worst-case convergence is not efficient on embedded platforms. The solver leverages a local quadratic growth condition: $$f(x) \geq f^* + \frac{\sigma}{2} \operatorname{dist}^2(x, X^*)$$ to propose a practical early stopping criterion based on the norm of an optimality residual: $$\|\partial L_\rho(x_t; \lambda_k) + s_t^*\| \leq \sqrt{(\sigma/2) B_\mathrm{in}} \implies L_\rho(x_t; \lambda_k) - L_\rho(x_k^*; \lambda_k) \leq B_\mathrm{in},$$ where $s_t^*$ is the projection of the subgradient onto the normal cone of the constraints at $x_t$. This allows on-device checking, saving computational effort and energy, while rigorously guaranteeing the required accuracy for the ALM update.

6. Numerical Validation and Practical Embedded Deployment

The method is validated on a network utility maximization problem typical of communication networks. Each node solves a local instance of the ALM in fixed-point, with typical subproblem dimensions around $n = 9$. Simulations using Matlab Fixed-Point Designer confirm that, for a range of bit-widths (e.g., 10–21 bits depending on desired accuracy $\epsilon$), the actual feasibility and suboptimality gaps observed are significantly below the worst-case theoretical bounds. Specifically, longer word lengths yield better accuracy at the expense of greater resource use.

Parameters such as the projection set $D$ for $\lambda$ are chosen via sampling or a priori estimation. Simulation tables demonstrate the practicality and conservative nature of the method: performance consistently exceeds worst-case predictions, validating the approach for embedded networked systems with strict computational and reliability constraints.

7. Significance for Embedded and Resource-Constrained Optimization

The proposed ALM is the first fixed-point augmented Lagrangian method able to handle both convex (including nonsmooth) objectives, equality and box constraints, and to:

• Explicitly prevent data overflow through dual projection,
• Quantify and control the effect of finite arithmetic errors on convergence through analytic bounds,
• Allow adaptive early termination of inner solvers without degrading solution quality,
• Guarantee convergence (to the error radius dictated by word length) rigorously, with monitored primal and dual gap certificates,
• Be scalable via problem structure and iterative subproblem solution, making it viable for high-volume or energy-limited deployments.

The formality of the error analysis, the use of projections to bound all iterates in the presence of hardware limitations, and the principled early stopping in iterative solvers collectively constitute a robust methodology for deploying convex constrained optimization in environments where floating-point arithmetic is infeasible or undesirable. This paradigm is highly significant for on-device control, IoT, and resource-constrained large-scale embedded optimization.