Alternating Bi-Objective Gradient Optimization

• Alternating bi-objective gradient optimization is a technique that alternates gradient updates for two conflicting objectives to efficiently approach Pareto-optimal solutions.
• It employs quadratic or linear programming subproblems to compute robust descent directions while minimizing gradient evaluations per iteration.
• By alternating updates across objectives, blocks, or stochastic minibatches, the method recovers nonconvex regions of the Pareto front that scalarized methods often miss.

Alternating Bi-Objective Gradient Optimization refers to a suite of algorithms and analysis frameworks that optimize two conflicting objectives by alternating gradient (or subgradient) updates, rather than forming a static or simultaneous aggregation (e.g., scalarization) of objectives. This alternation can occur at multiple levels: objective functions, parameter blocks, stochastic minibatches, or coordinate selection. The methodology is designed to efficiently approximate Pareto-optimal solutions, balance computational complexity, and, in many non-convex or structured settings, to recover nonconvex segments of the Pareto front that are inaccessible to scalarized methods.

1. Formal Problem Statement and Pareto Criticality

Alternating bi-objective gradient optimization focuses on unconstrained multicriteria minimization: $$\min_{x\in\mathbb{R}^n} \; F(x) = \big( f_1(x),\; f_2(x) \big)$$ where $f_1, f_2:\mathbb{R}^n\to\mathbb{R}$ are continuously differentiable objectives. A point $x^*$ is Pareto critical if no direction $v\in\mathbb{R}^n$ exists with $\nabla f_1(x^*)^\top v<0$ and $\nabla f_2(x^*)^\top v<0$; that is, there is no vector simultaneously descending both objectives. The optimization goal is to generate a sequence $\{x^k\}$ that approaches the set of such points, thereby approximating the Pareto set and front (Oliveira et al., 2021, Santa, 2024).

2. Central Descent and Shared Descent Directions

Classical scalarization approaches construct minimizers of $S(x,\lambda)=\lambda f_1(x) + (1-\lambda)f_2(x)$ for varying $\lambda$, indirectly sweeping the convex hull of the front. Alternating approaches instead seek directions $v$ that are descent (or at least non-ascent) for both $f_1, f_2:\mathbb{R}^n\to\mathbb{R}$0 and $f_1, f_2:\mathbb{R}^n\to\mathbb{R}$1 at the current iterate.

Central Descent Direction:

Defined as the solution to the following quadratic program: $f_1, f_2:\mathbb{R}^n\to\mathbb{R}$2 This direction is robust, scale-invariant, and guarantees maximum angular separation from non-descent cones. The associated incremental algorithm alternates which objective’s gradient is freshly computed at each step, while solving for $f_1, f_2:\mathbb{R}^n\to\mathbb{R}$3 using the stored and updated gradients, ensuring that only a constant number of gradients are evaluated per iteration and total gradient-query complexity is independent of the number of objectives (Oliveira et al., 2021).

Alternatively, recent approaches define a shared descent or non-ascent direction using compact LP subproblems that select $f_1, f_2:\mathbb{R}^n\to\mathbb{R}$4 to minimize a linear function of the gradients subject to normalized gradient constraints: $f_1, f_2:\mathbb{R}^n\to\mathbb{R}$5 This LP either yields strict descent for both or a shared non-ascent direction (possibly nonzero) when such a descent direction does not exist; it is used with a backtracking line search that allows strict decrease or non-dominated non-ascent moves (Santa, 2024).

3. Alternating Incremental and Stochastic Optimization Schemes

Alternating methods are implemented across several algorithmic architectures:

• Incremental Alternation: Only one new gradient $f_1, f_2:\mathbb{R}^n\to\mathbb{R}$6 is computed per iteration (alternated between objectives), while the other gradient is reused. After solving for a shared descent direction (via QP or LP), an Armijo-type or diminishing step-size update is performed in that direction. This ensures $f_1, f_2:\mathbb{R}^n\to\mathbb{R}$7 per-iteration cost and $f_1, f_2:\mathbb{R}^n\to\mathbb{R}$8 convergence in proximity-to-criticality, with total required gradient queries $f_1, f_2:\mathbb{R}^n\to\mathbb{R}$9 (Oliveira et al., 2021).
• Stochastic Alternating Algorithms: Fix two positive integers $x^*$0, representing steps on $x^*$1 and $x^*$2 per outer iteration. At each outer iteration, alternate $x^*$3 stochastic gradient steps for $x^*$4 and $x^*$5 for $x^*$6. This process converges to a minimizer of the corresponding weighted-sum scalarization with effective preference $x^*$7. The O(1/T) or O(1/\sqrt{T}) convergence is established under (strongly) convexity and Lipschitz continuity, with Pareto front tracability by sweeping the $x^*$8 ratio (Liu et al., 2022).
• Block Coordinate and Function Alternation: For high-dimensional or block-structured problems, alternation occurs both over objective functions and variable blocks. Each block is updated a prescribed number of steps with respect to each objective, where the sequence of objectives per block can be randomized. This scheme matches per-iteration complexity with conventional full-gradient methods when the number of per-block alternations equals the number of objectives, but is more efficient in block-sparse or large-scale systems (Tran et al., 12 May 2026).
• Explainable Alternating Optimization (X-ANFIS): For neuro-fuzzy inference systems, objectives such as predictive accuracy and linguistic distinguishability are alternately optimized at each epoch. Performance parameters (Gaussian/Cauchy membership centers and widths) are updated with respect to predictive loss, then (some or all) are updated with respect to explainability loss in a distinct, decoupled step, preventing destructive interference and enabling traversal of nonconvex front regions that scalarization cannot access (Khaled et al., 22 Feb 2026).

4. Convergence Guarantees and Complexity

Extensive convergence theory supports alternating bi-objective optimization schemes. A summary of established complexity results includes:

Setting	Convergence Rate	Complexity per iteration	References
Smooth, nonconvex, incremental alternation	$x^*$9	1 gradient + 1 QP	(Oliveira et al., 2021)
Stochastic, strongly convex	$v\in\mathbb{R}^n$0	$v\in\mathbb{R}^n$1 subgradients	(Liu et al., 2022)
Stochastic, convex	$v\in\mathbb{R}^n$2	$v\in\mathbb{R}^n$3 subgradients	(Liu et al., 2022)
Stochastic PL (Block-SMOO)	$v\in\mathbb{R}^n$4	$v\in\mathbb{R}^n$5 partial grads	(Tran et al., 12 May 2026)
X-ANFIS, nonconvex objectives	Empirically robust	2 alternating steps/epoch	(Khaled et al., 22 Feb 2026)

Rigorous non-ascent step acceptance (e.g., during flat Pareto-critical regions) allows the algorithm to traverse and then escape such manifolds, empirically increasing global Pareto coverage (Santa, 2024). In the context of explainable fuzzy systems, convergence to target explainability is achieved without significant loss of predictive accuracy (Khaled et al., 22 Feb 2026).

5. Pareto Front Exploration and Coverage

Alternating bi-objective schemes enable targeted exploration of the Pareto front, especially where convex aggregation methods are insufficient:

• By varying update frequencies ($v\in\mathbb{R}^n$6 in stochastic setups, or block-wise allocations in block alternation), different trade-off points along the front are produced. In convex settings, the entire front can be approximated via grid search over update ratios (Liu et al., 2022, Tran et al., 12 May 2026).
• In nonconvex or flat-front landscapes, relaxed acceptance strategies (allowing non-dominated non-ascent moves) significantly increase the empirical fraction of runs returning globally non-dominated points, with reported Pareto coverage improvements (e.g., 92.8% on Viennet with LP_new + new backtracking) (Santa, 2024).
• Alternating optimization approaches such as X-ANFIS yield solutions in nonconvex bulges of the true front that scalarized or evolutionary methods cannot reach, with direct targeting of secondary objectives (e.g., explainability) and minimal impact on primary objective performance (Khaled et al., 22 Feb 2026).

6. Practical Implementation Considerations

Core implementation features include:

• Alternate gradient evaluation for each objective; after refreshing the selected gradient, solve the corresponding QP or LP for the shared descent direction (Oliveira et al., 2021, Santa, 2024).
• Armijo or inexact backtracking line searches to assure sufficient decrease or non-dominated acceptance; step sizes may be dynamically tuned or pre-specified (Oliveira et al., 2021, Santa, 2024).
• Storage and pruning of non-dominated iterates, especially relevant when the underlying Pareto front is large or contains extended critical sets (Santa, 2024).
• For block-wise alternation, random permutation of block order and alternate cycling or stochastic allocation of steps are used to avoid cyclic patterns and to efficiently share computational cost among objectives (Tran et al., 12 May 2026).
• Empirical guidelines support using fixed step sizes in many applications, and balancing trade-off exploration against computational budget by tuning update ratios and batch sizes (Liu et al., 2022, Tran et al., 12 May 2026).

7. Broader Impact and Theoretical Insights

Alternating bi-objective gradient optimization has established several theoretical and practical benefits over purely simultaneous or scalarized methods:

• Reduces gradient-query complexity to a minimum per iteration that is independent of the number of objectives or variable blocks (Oliveira et al., 2021).
• Enables recovery of solutions in nonconvex front regions missed by weighting or evolutionary multi-objective optimization (Khaled et al., 22 Feb 2026).
• Yields superior empirical Pareto coverage, especially in settings with flat or high-dimensional critical manifolds (Santa, 2024).
• Theoretical iteration complexity bounds improve upon simultaneous updates, e.g., in minimax setups, alternating gradient methods provably break the worst-case coupling barriers of the simultaneous case (improved scaling in the cross-coupling condition number) (Lee et al., 2024).
• In stochastic and non-convex settings, bias correction can be incorporated to maintain asymptotic unbiasedness and guarantee convergence to Pareto-stationary solutions, even under fixed mini-batch sizes (Fernando et al., 2022).

The alternating paradigm thus offers a unifying and computationally principled approach for bi-objective (and, by clear extension, multi-objective) optimization under differentiability, smoothness, and convexity/non-convexity, with rigorous guarantees, efficient frontier exploration, and robust empirical performance in diverse domains—from algorithm theory to explainable machine learning (Oliveira et al., 2021, Santa, 2024, Khaled et al., 22 Feb 2026, Liu et al., 2022, Tran et al., 12 May 2026, Lee et al., 2024, Fernando et al., 2022).