Filtering-Based Robust Multi-Task Gradient Descent

• The paper introduces a filtering-based multi-task gradient descent that excludes harmful gradient components to robustly mitigate adversarial contamination and inter-task conflicts.
• It employs both statistical filtering and geometric techniques—such as covariance-based selection, orthogonal decomposition, and clipping—to ensure accurate global and task-specific updates.
• Empirical results and theoretical guarantees demonstrate improved convergence rates and performance over standard methods by balancing robustness and personalization in multi-task learning.

Filtering-based robust multi-task gradient descent denotes a class of multi-task optimization procedures that improve shared updates by suppressing task-gradient contributions judged harmful before they are aggregated into the backbone step. In the strict sense, the term refers to contamination-aware multi-task empirical risk minimization in which an $\epsilon$ fraction of tasks may be arbitrarily corrupted and robustness is obtained by filtering task-level gradients while still estimating both a global minimizer and clean task-specific minimizers (Tian et al., 2 Jul 2026). In a broader interpretive sense, recent multi-task learning work also instantiates filtering through orthogonal decomposition, subspace projection, taskwise clipping and norm equalization, cone-constrained direction selection, or conflict-aware scheduling; these methods do not all model adversarial contamination, but they do suppress gradient components, task updates, or admissible directions associated with negative transfer (Dong et al., 2023, Zhu et al., 5 Mar 2025).

1. Definition and conceptual scope

Filtering-based robust multi-task gradient descent is best understood as a family of update rules for shared-parameter multi-task learning in which the raw task-gradient set is not accepted wholesale. Instead, the optimizer first applies a criterion that separates desirable from undesirable gradient information and only then forms the shared update. In the explicit contamination setting, the filtered objects are task-level empirical gradients, and the goal is robustness to arbitrarily corrupted tasks under heterogeneity (Tian et al., 2 Jul 2026). In conflict-mitigation settings, the filtered objects are usually gradient directions, subspace coordinates, high-norm task contributions, or task participation sets, and the goal is to reduce negative transfer rather than to withstand adversarial corruption (Zhang et al., 2023, Patapati et al., 21 Sep 2025).

Two problem classes recur. The first is contamination-robust multi-task ERM, where some tasks are malicious or grossly corrupted and ordinary pooling fails statistically. The second is conflict-aware multi-task optimization, where all tasks are legitimate but their gradients oppose, dominate, or destabilize one another. The first class asks for robust estimation; the second asks for robust optimization geometry. The papers in this area show that these are related but not identical. A plausible implication is that “filtering” should be read at two levels: as formal robust-statistical filtering of corrupted task gradients, and as geometric or scheduling-based suppression of harmful inter-task update components (Tian et al., 2 Jul 2026, Hassanpour et al., 31 Jan 2025).

2. Canonical contaminated heterogeneous formulation

The most explicit formulation studies $K$ tasks, each with $n$ i.i.d. observations, loss $\ell:\mathcal Z\times\mathbb R^d\to\mathbb R_+$, task risk

$L^{(k)}(\theta)=\mathbb E[\ell(z^{(k)},\theta)],$

and average population risk

$L(\theta)=\frac1K\sum_{k=1}^K L^{(k)}(\theta).$

The inferential targets are both the global minimizer

$\theta^*=\arg\min_{\theta\in\Theta} L(\theta)$

and the clean task-specific minimizers

$$\theta^{(k)*}=\arg\min_{\theta\in\Theta} L^{(k)}(\theta), \qquad k\in [K].$$

Contamination is task-wise: for a clean-task set $S\subseteq[K]$ with $|S|\ge K(1-\epsilon)$, tasks in $K$0 remain untouched, while tasks in $K$1 may have all samples replaced arbitrarily. The observed empirical risks are

$K$2

This formulation makes robustness genuinely task-level rather than sample-level (Tian et al., 2 Jul 2026).

The same framework is heterogeneous rather than homogeneous. Clean tasks are not assumed to share a common parameter; instead, heterogeneity is quantified through gradient dispersion: $K$3 and

$K$4

The quantity $K$5 governs average cross-task deviation, while $K$6 governs local task deviation. This is the structural reason the method targets both robustness and personalization rather than only pooled estimation (Tian et al., 2 Jul 2026).

A central negative result accompanies this setup. In the Gaussian mean model, several common paradigms—adaptive and robust regularization around a shared center, global matrix regularization, decomposition-based regularization, and score-based outlier-task detection—suffer from a worst-case contamination error of order $K$7, whereas the minimax lower bound is $K$8. The paper identifies this as a dimension-dependent barrier for these approaches (Tian et al., 2 Jul 2026).

3. Explicit filtering by robust task-gradient estimation

The canonical filtering-based robust multi-task gradient descent procedure operates directly on the set of task empirical gradients. At iteration $K$9, task $n$0 forms

$n$1

and the global robust gradient is defined by

$n$2

where JRGE is a joint robust gradient estimator implemented by iterative filtering (Tian et al., 2 Jul 2026).

The resulting global and local updates are

$n$3

and, for each task,

$n$4

with personalized local gradient

$n$5

The vector soft-thresholding operator is

$n$6

This decomposition is the personalization mechanism: task $n$7 inherits the robust global descent direction, then adds a task-specific correction only when the residual is large enough to survive thresholding (Tian et al., 2 Jul 2026).

The filtering step itself is covariance-based. Given a current active set $n$8, the algorithm computes

$n$9

where $\ell:\mathcal Z\times\mathbb R^d\to\mathbb R_+$0 denotes the task-level vectors being filtered, here instantiated by task gradients. It then forms the top eigenvector $\ell:\mathcal Z\times\mathbb R^d\to\mathbb R_+$1 of

$\ell:\mathcal Z\times\mathbb R^d\to\mathbb R_+$2

If

$\ell:\mathcal Z\times\mathbb R^d\to\mathbb R_+$3

the set is declared too anisotropic relative to the target covariance. The method identifies the top $\ell:\mathcal Z\times\mathbb R^d\to\mathbb R_+$4 tasks with largest projected squared residuals

$\ell:\mathcal Z\times\mathbb R^d\to\mathbb R_+$5

removes one of them at random with probability proportional to that same quantity, and repeats until the spectral discrepancy falls below threshold. The output mean is then used as the robust aggregate (Tian et al., 2 Jul 2026).

The covariance estimate entering this filter is itself task-structured. For task $\ell:\mathcal Z\times\mathbb R^d\to\mathbb R_+$6,

$\ell:\mathcal Z\times\mathbb R^d\to\mathbb R_+$7

and a safe subset of tasks is selected via pairwise covariance proximity before averaging them into $\ell:\mathcal Z\times\mathbb R^d\to\mathbb R_+$8. The method therefore filters in two senses: it filters suspicious task gradients, and it stabilizes that decision with a covariance estimate built from a task subset judged mutually compatible (Tian et al., 2 Jul 2026).

4. Broader filtering mechanisms in multi-task optimization

Beyond explicit contamination-robust ERM, several multi-task optimizers realize filtering in an interpretive but technically meaningful sense. They differ mainly in what is filtered: coordinates in a learned gradient basis, oversized backbone contributions, Pareto weights, admissible directions, task participation sets, or span components.

Method	Mechanism	Filtered or suppressed object
GDOD	Orthogonal decomposition with sign-consensus masking	Task-conflict basis coordinates
TBGC	Taskwise clipping and norm equalization	Large-norm backbone task contributions
EMGD	Elastic Pareto-constrained reweighting	Task influence via adaptive coefficients
ConicGrad	Cone-constrained direction selection	Update directions outside a cone around $\ell:\mathcal Z\times\mathbb R^d\to\mathbb R_+$9
Graph-coloring scheduler	Interference-aware task partitioning	Tasks not in the active compatible group
GradOPS	Projection onto orthogonal complements of other-task spans	Conflict-inducing span components

GDOD constructs a mini-batch subspace $L^{(k)}(\theta)=\mathbb E[\ell(z^{(k)},\theta)],$0 from per-example, per-task gradients, computes an orthogonal basis $L^{(k)}(\theta)=\mathbb E[\ell(z^{(k)},\theta)],$1 from the SVD of the gradient matrix, projects each task gradient into basis coordinates $L^{(k)}(\theta)=\mathbb E[\ell(z^{(k)},\theta)],$2, and then applies the sign-consensus mask

$L^{(k)}(\theta)=\mathbb E[\ell(z^{(k)},\theta)],$3

The update keeps only the reconstructed shared components,

$L^{(k)}(\theta)=\mathbb E[\ell(z^{(k)},\theta)],$4

The paper calls these “task-shared” and “task-conflict” components; the filtering interpretation is that conflict coordinates in a learned orthogonal basis are zeroed out before aggregation (Dong et al., 2023).

TBGC addresses a different failure mode: task-dominant backbone gradient norms. For task $L^{(k)}(\theta)=\mathbb E[\ell(z^{(k)},\theta)],$5, with backbone gradient $L^{(k)}(\theta)=\mathbb E[\ell(z^{(k)},\theta)],$6 and norm $L^{(k)}(\theta)=\mathbb E[\ell(z^{(k)},\theta)],$7, it first applies taskwise clipping,

$L^{(k)}(\theta)=\mathbb E[\ell(z^{(k)},\theta)],$8

then backbone-oriented rescaling,

$L^{(k)}(\theta)=\mathbb E[\ell(z^{(k)},\theta)],$9

so that $L(\theta)=\frac1K\sum_{k=1}^K L^{(k)}(\theta).$0, and finally aggregates

$L(\theta)=\frac1K\sum_{k=1}^K L^{(k)}(\theta).$1

This does not explicitly resolve angular conflict; it filters the magnitude advantage of dominant tasks on shared parameters by clipping and equalization before aggregation (Zhang et al., 2023).

EMGD works through constrained soft suppression rather than hard deletion. With negative task gradients $L(\theta)=\frac1K\sum_{k=1}^K L^{(k)}(\theta).$2, it replaces standard Pareto steepest descent by elastic constraints

$L(\theta)=\frac1K\sum_{k=1}^K L^{(k)}(\theta).$3

whose dual is

$L(\theta)=\frac1K\sum_{k=1}^K L^{(k)}(\theta).$4

The elastic factors $L(\theta)=\frac1K\sum_{k=1}^K L^{(k)}(\theta).$5 are computed either from momentum-smoothed gradient magnitudes or from gradient cosine similarities. This is a continuous filtering mechanism: no task is discarded, but tasks are protected or relaxed asymmetrically through the feasible coefficient simplex (Lyu et al., 2024).

ConicGrad filters admissible update directions rather than tasks. It optimizes

$L(\theta)=\frac1K\sum_{k=1}^K L^{(k)}(\theta).$6

where

$L(\theta)=\frac1K\sum_{k=1}^K L^{(k)}(\theta).$7

The angular constraint defines a cone around the reference gradient $L(\theta)=\frac1K\sum_{k=1}^K L^{(k)}(\theta).$8, so directions outside that cone are excluded from the feasible set. The method is therefore a directional filter over joint update vectors rather than a task filter (Hassanpour et al., 31 Jan 2025).

The graph-coloring scheduler filters task participation temporally. It defines interference by negative cosine similarity,

$L(\theta)=\frac1K\sum_{k=1}^K L^{(k)}(\theta).$9

smooths gradients with

$\theta^*=\arg\min_{\theta\in\Theta} L(\theta)$0

constructs a conflict graph with edges

$\theta^*=\arg\min_{\theta\in\Theta} L(\theta)$1

colors that graph greedily, and activates only one color class $\theta^*=\arg\min_{\theta\in\Theta} L(\theta)$2 per step: $\theta^*=\arg\min_{\theta\in\Theta} L(\theta)$3 Nothing is projected or reweighted; incompatible tasks are filtered by scheduling and deferred to different updates (Patapati et al., 21 Sep 2025).

GradOPS removes the component of each task gradient lying in the span of the other task gradients. For task $\theta^*=\arg\min_{\theta\in\Theta} L(\theta)$4, if $\theta^*=\arg\min_{\theta\in\Theta} L(\theta)$5 conflicts with at least one other task, it constructs an orthogonal basis $\theta^*=\arg\min_{\theta\in\Theta} L(\theta)$6 for $\theta^*=\arg\min_{\theta\in\Theta} L(\theta)$7 and sets

$\theta^*=\arg\min_{\theta\in\Theta} L(\theta)$8

The aggregate is then

$\theta^*=\arg\min_{\theta\in\Theta} L(\theta)$9

By construction, each $$\theta^{(k)*}=\arg\min_{\theta\in\Theta} L^{(k)}(\theta), \qquad k\in [K].$$0 is non-conflicting with every original task gradient, so any nonnegative combination remains weakly non-conflicting. This is a global subspace filter rather than pairwise surgery (Zhu et al., 5 Mar 2025).

5. Guarantees, rates, and empirical behavior

In the explicit contamination setting, the central theoretical result is minimax. The lower bounds for heterogeneous contaminated ERM state that any estimator of the global parameter must incur

$$\theta^{(k)*}=\arg\min_{\theta\in\Theta} L^{(k)}(\theta), \qquad k\in [K].$$1

in the worst case, while personalized clean-task estimation must incur

$$\theta^{(k)*}=\arg\min_{\theta\in\Theta} L^{(k)}(\theta), \qquad k\in [K].$$2

The filtering-based robust multi-task gradient descent method matches these rates up to logarithmic factors over a broad regime and removes the extra $$\theta^{(k)*}=\arg\min_{\theta\in\Theta} L^{(k)}(\theta), \qquad k\in [K].$$3 contamination dependence that broad regularization-based methods and score-based outlier detection suffer (Tian et al., 2 Jul 2026).

The global and local finite-sample guarantees are coupled to robust gradient estimation error. Under local strong convexity, smoothness, and sub-Gaussian gradient assumptions, the descent theorem gives geometric contraction up to a noise floor: $$\theta^{(k)*}=\arg\min_{\theta\in\Theta} L^{(k)}(\theta), \qquad k\in [K].$$4 and similarly for each clean task. After substituting the filtering-based gradient error and covariance-estimation bounds, the final global rate becomes

$$\theta^{(k)*}=\arg\min_{\theta\in\Theta} L^{(k)}(\theta), \qquad k\in [K].$$5

with the corresponding local personalized analogue including $$\theta^{(k)*}=\arg\min_{\theta\in\Theta} L^{(k)}(\theta), \qquad k\in [K].$$6 (Tian et al., 2 Jul 2026).

Conflict-mitigation methods outside the contamination framework usually provide smoothness-based descent or Pareto-critical convergence results rather than minimax robustness statements. GDOD proves convergence to the optimal value in the convex case or to a stationary point in the non-convex case under Lipschitz gradients and $$\theta^{(k)*}=\arg\min_{\theta\in\Theta} L^{(k)}(\theta), \qquad k\in [K].$$7, with non-convex rate $$\theta^{(k)*}=\arg\min_{\theta\in\Theta} L^{(k)}(\theta), \qquad k\in [K].$$8. Empirically, on Shared-Bottom, GDOD reports BookCrossing task1/task2 AUC $$\theta^{(k)*}=\arg\min_{\theta\in\Theta} L^{(k)}(\theta), \qquad k\in [K].$$9, IJCAI-15 task1/task2 AUC $S\subseteq[K]$0, and Alipay Advertising task1/task2/task3 AUC $S\subseteq[K]$1, outperforming Adam, PCGrad, and CAGrad on those comparisons (Dong et al., 2023).

TBGC is empirical rather than theory-heavy, but its reported challenge results directly target gradient norm imbalance. On detection, segmentation, and fine-grained classification, vanilla clipping gives overall $S\subseteq[K]$2, while TBGC$S\subseteq[K]$3 gives $S\subseteq[K]$4 and full TBGC gives $S\subseteq[K]$5. The task metrics shift from $S\subseteq[K]$6 under vanilla clipping to $S\subseteq[K]$7 under TBGC, which the paper interprets as evidence that lower-gradient tasks benefit most while the dominant task is not harmed (Zhang et al., 2023).

EMGD proves that the accumulation point of the continuous optimization by Algorithm 1 is Pareto critical. In task-incremental PS-CIFAR-100, EMGD(GMC) obtains $S\subseteq[K]$8 average accuracy and EMGD(GMC)+ME obtains $S\subseteq[K]$9, compared with MGDA’s $|S|\ge K(1-\epsilon)$0, PCGrad’s $|S|\ge K(1-\epsilon)$1, DER++’s $|S|\ge K(1-\epsilon)$2, and MaxDO’s $|S|\ge K(1-\epsilon)$3. On task-incremental PS-ImageNet-TINY, EMGD(GS)+ME reaches $|S|\ge K(1-\epsilon)$4 versus MGDA $|S|\ge K(1-\epsilon)$5, PCGrad $|S|\ge K(1-\epsilon)$6, DER++ $|S|\ge K(1-\epsilon)$7, and MaxDO $|S|\ge K(1-\epsilon)$8 (Lyu et al., 2024).

ConicGrad proves an $|S|\ge K(1-\epsilon)$9-type stationarity bound for the reference objective $K$00 under $K$01-Lipschitz gradients and $K$02. Empirically it reports best $K$03 on CityScapes, best $K$04 and best $K$05 on CelebA, and $K$06 success on Meta-World MT10, approaching the STL upper bound $K$07 (Hassanpour et al., 31 Jan 2025).

The graph-coloring scheduler proves that for a $K$08-compatible set,

$K$09

and gives a nonconvex convergence bound of order $K$10. Across six datasets, the scheduler is often competitive as a standalone method and often stronger when combined with direct optimizers; examples reported include CIFAR-10 accuracy $K$11 for Scheduler AdaTask versus $K$12 for PCGrad and $K$13 for AdaTask, and STOCKS-HEALTH accuracy $K$14 for Scheduler PCGrad versus $K$15 for PCGrad (Patapati et al., 21 Sep 2025).

GradOPS proves convergence to a Pareto stationary point under differentiability, $K$16-Lipschitz gradients, and sufficiently small step size. It also argues that after strong deconfliction, simple trade-off rules become effective because any nonnegative combination of transformed gradients remains non-conflicting with every original task gradient. On UCI, GradOPS($K$17) outperforms PCGrad on all tasks, and on NYUv2 and the Taobao recommendation dataset GradOPS variants achieve the best $K$18, best mean rank, or best average performance among compared methods (Zhu et al., 5 Mar 2025).

6. Limitations and methodological boundaries

The literature draws a sharp boundary between explicit robust filtering and broader conflict mitigation. The contamination-aware method is statistically robust to an $K$19 fraction of arbitrarily corrupted tasks, but its guarantees rely on local strong convexity, local smoothness, sub-Gaussian gradients, and tuning involving $K$20, $K$21, and $K$22. Its covariance-estimation step also introduces extra terms and an eigenvector-based inner loop, so the method is not a lightweight drop-in replacement for standard joint training (Tian et al., 2 Jul 2026).

Methods that are “filtering-based” only by interpretation address narrower failure modes. GDOD filters sign-inconsistent basis coordinates but requires per-example gradients and an SVD each iteration; the paper explicitly notes that per-example gradients plus SVD each iteration are expensive, and that strict sign-consensus can become overly conservative as the number of tasks grows, motivating weighted-GDOD on the 6-task Census-Income dataset (Dong et al., 2023). TBGC filters task-dominant magnitudes, not directional conflict, and assumes that all tasks should have roughly equal influence on the backbone (Zhang et al., 2023). EMGD reweights all tasks inside a Pareto framework rather than rejecting outliers, so if a task is truly bad or adversarial it is not explicitly removed; the optional memory editing also requires second-order differentiation (Lyu et al., 2024).

Directional and scheduling filters bring their own trade-offs. ConicGrad depends on the cone parameter $K$23: if $K$24 is too large the method collapses toward plain gradient descent on $K$25, whereas if $K$26 is too small the anchoring effect weakens (Hassanpour et al., 31 Jan 2025). The graph-coloring scheduler has refresh overhead $K$27 and amortized per-step overhead $K$28, and because only one color class is active per step some tasks may wait multiple steps between updates (Patapati et al., 21 Sep 2025). GradOPS can be read as a global subspace filter, but its cost grows with task count and its residual can become small or zero when much of a task’s signal lies in the span of the others (Zhu et al., 5 Mar 2025).

Two adjacent lines of work clarify the boundaries of the topic. The flatness-based SAM extension to MTL is not an explicit filtering method; it handles harmful gradients implicitly by decomposing shared SAM gradients into loss and flatness components and aggregating those separately, which acts more like a soft geometric stabilizer against sharp, overfitting-prone directions than a robust filter (Phan et al., 2022). MARIGOLD is likewise not a filtering-based robustness method in the usual sense; it reformulates gradient balancing as a bi-level optimization problem and uses a zeroth-order hypergradient estimator to reduce balancing cost from $K$29 to $K$30, making it an efficiency-oriented relative rather than a direct instance of filtering-based robust multi-task gradient descent (Chen et al., 8 Mar 2026).

Taken together, these works indicate that filtering-based robust multi-task gradient descent is not a single algorithmic template but a spectrum. At one end lies formal robust statistics over task gradients under adversarial contamination. At the other lie optimization filters that suppress negative transfer by basis selection, clipping, constrained direction sets, or scheduler-level task exclusion. This suggests that future unification will likely require combining the robust-statistical guarantees of task-gradient filtering with the geometric flexibility of conflict-aware multi-task optimization (Tian et al., 2 Jul 2026).