MTCL: Multi-Task Critic Learning

• Multi-Task Critic Learning is a methodology that equips reinforcement learning agents with unified or task-specific critic networks to optimize performance across multiple tasks.
• Dynamic gradient weighting techniques mitigate gradient conflict, balancing feedback across tasks for fair updates and accelerated convergence.
• Scalable and distributed MTCL architectures enhance sample efficiency, enable robust real-world deployment, and support privacy-preserving decentralized learning.

Multi-Task Critic Learning (MTCL) refers to a set of methodologies that equip reinforcement learning agents with the capacity to solve multiple tasks through optimized and coordinated value-function (critic) learning. Rather than training and deploying separate critics for each task, or relying on simple aggregation of task losses, MTCL leverages either shared or task-specific critic architectures, dynamic weighting of learning signals, and architectural scaling, all with the goal of improving sample efficiency, stability, and generalization across diverse tasks.

1. Foundations and Motivation

Multi-task reinforcement learning challenges the agent to perform well on a set $\mathcal{T} = \{T_1, ..., T_N\}$, where each task $T_i$ is an MDP with its own dynamics and reward function. The key objective is to maximize average or minimum performance over tasks, or to guarantee individual task performance within pre-specified bounds. In classical approaches, the critic—the function approximator estimating action-values or state-values—is a central feedback mechanism for actor updates. MTCL generalizes this, asking how critic learning should be structured to enable efficient and robust multi-task transfer, avoid negative interference (“gradient conflict”), and scale with task diversity and model size (McLean et al., 7 Mar 2025, Xing et al., 17 Dec 2024, Wang et al., 25 May 2024, Sharma et al., 2017).

2. Architectures: Shared, Multi-Critic, and Scalable Designs

Three primary architectural paradigms are prevalent in MTCL:

a. Shared Critic: A single critic network is trained using data aggregated from all tasks, with the underlying assumption that common structure or representation suffices for all tasks. This is computationally efficient and leverages trans-task generalization but can result in suboptimal performance when task rewards or dynamics are highly distinct (Sharma et al., 2017).

b. Multi-Critic Designs: In this approach, a separate critic is maintained for each task while sharing a common actor (policy). Each critic provides task-specific feedback to the shared actor, allowing fine-grained adaptation to the particular objectives and rewards of each environment. For example, in quadrotor control, critics for stabilization, velocity tracking, and racing are trained in parallel, each optimizing their task-specific TD losses, while the actor’s update leverages their collective feedback (Xing et al., 17 Dec 2024).

c. Parameter Scaling and Model Capacity: Recent empirical evidence indicates that scaling the critic’s network width (and thus total parameter count) provides substantial benefits, often exceeding those obtained by sophisticated modular or expert-based architectures. Scaling up the critic—more so than the actor—proves critical when the agent must represent and learn value functions over diverse task distributions (McLean et al., 7 Mar 2025). This effect is amplified as task diversity increases, with richer task sets serving as a natural regularizer that helps utilize the entire capacity of the model.

Paradigm	Critic Structure	Actor Structure	Best For
Shared Critic	Single, all tasks	Single, all tasks	Homogeneous tasks, transfer focus
Multi-Critic	One per task	Shared	Heterogeneous task objectives
Scaled Critic	Large/wide (often per-task)	Shared or scalable	High task diversity, scalability

3. Dynamic Gradient Weighting and Conflict Mitigation

A central challenge in MTCL is gradient conflict, where some tasks’ value function gradients dominate the shared update, leading to performance degradation on less-represented tasks. To address this, recent methods introduce dynamic loss weighting techniques.

Conflict-Avoidant MTCL (MTAC): The update direction is chosen to minimize the potential for task gradient conflict. This is formalized as

$$\lambda^* \in \arg\min_{\lambda \in \Lambda} \|\lambda^\top \nabla J(\theta)\|^2,$$

where $\nabla J(\theta)$ contains all task gradients, and $\Lambda$ is the simplex (i.e., $\lambda_k \geq 0$, $\sum_k \lambda_k = 1$). The actor is then updated using the gradient weighted by $\lambda^*$. This approach ensures more balanced progress and Pareto optimality. The MTAC-CA sub-procedure brings the estimated update direction close to the conflict-free ideal, at the cost of higher sample complexity ($\mathcal{O}(\epsilon^{-5})$), while the fast-convergence version MTAC-FC offers greater efficiency with a trade-off in “distance to CA direction” (Wang et al., 25 May 2024).

Empirical and theoretical results demonstrate that conflict-avoidant optimization not only improves fairness across tasks but also accelerates convergence to Pareto-stationary solutions, subject to function approximation error and sample budget.

4. Distributed and Decentralized Multi-Task Critic Learning

Scaling MTCL to distributed settings is critical for multi-agent systems and real-world deployments. Methods such as Diff-DAC (Diffusion Distributed Actor-Critic) and Federated NAC (Natural Actor-Critic) (Macua et al., 2017, Macua et al., 2021, Yang et al., 2023) have demonstrated that:

• Diffusion strategies: Each agent (assigned a different task) alternates local updates to its critic/policy (using on-task samples), with parameter averaging (“diffusion”) among neighbors. Over iterative communication rounds, all agents converge toward a common solution optimal (on average) for the group’s tasks.
• Federated frameworks: Agents exchange only processed policy and value parameters, not private trajectories or reward functions, thereby supporting privacy-preserving decentralized MTCL. This approach enjoys convergence rates nearly independent of the state-action space size, with sample efficiency governed by network connectivity.

These protocols enable fully decentralized MTCL, robust to agent failures and communication dropouts, and naturally generalize to regimes with heterogeneous (but related) tasks.

5. Constrained, Pareto-Optimal, and Adaptive MTCL Objectives

Beyond maximizing average performance, MTCL frameworks are frequently tasked with respecting constraints—bounds on per-task rewards, or balancing trade-offs among tasks:

• Constrained Multi-Task RL directly encodes per-task performance bounds $\ell_i \leq V_i^\pi \leq u_i$, optimizing via primal-dual actor-critic updates. Centralized and decentralized solutions provably converge with suboptimality and constraint violation decaying at established rates (e.g., $O(1/\sqrt{K})$ with exact gradients; $O(1/K^{1/6})$ for sample-based learning) (Zeng et al., 3 May 2024).
• Pareto-Optimality and Adaptive Weighting: Dynamic adaptation of loss or gradient weights (using meta-critic or adaptive scalarization) ensures coverage of the Pareto front, and, in some cases (e.g., controllable Pareto hypernetworks (Lin et al., 2020)), enables real-time performance adjustment given user-specified preferences.
• Meta-Critic Learning: Augmenting the standard critic with a meta-critic learned online to accelerate/steer the policy updates yields faster and more stable learning for individual or multiple tasks. The meta-critic can be extended to account for task context, potentially providing value in multi-task learning scenarios by rapidly tailoring loss signals to diverse objectives (Zhou et al., 2020).

6. Practical Applications and Empirical Evidence

MTCL frameworks have been validated in diverse application domains:

• Robotics and Control: Multi-critic architectures have enabled quadrotors to safely and efficiently transfer across tasks such as racing, stabilization, and tracking, outperforming single-task policies and showcasing knowledge transfer via shared encoders and task-specific critics (Xing et al., 17 Dec 2024).
• Sim2Real and Distributional Robustness: Q-weighted adversarial learning integrated with multi-task SAC allows agents to generalize to novel object configurations, leveraging prior data and adversarial reward shaping for robust robotic manipulation (Nehme et al., 2023).
• Benchmarks and Scaling Laws: On Meta-World MT10/MT50 and similar benchmarks, simple fully-connected critics scaled to match the parameter budgets of modular or “expert” architectures achieve or surpass state-of-the-art performance, attributed primarily to critic capacity and task diversity regularization rather than architectural complexity (McLean et al., 7 Mar 2025).

Task diversity is also linked to model “plasticity”: as more tasks are added, large models avoid neuron “dormancy” and sustain their representation capacity, further improving MTCL outcomes.

7. Challenges and Future Directions

While MTCL offers powerful tools for robust and efficient multi-task RL, several open challenges remain:

• Gradient conflict and interference still pose practical difficulties when tasks are highly unrelated or adversarial. Advanced conflict-avoidant gradient methods offer progress but may entail higher computational costs.
• Scalability vs. sample efficiency: Balancing critic capacity and the number of parallel tasks is nontrivial; sparsely sampled or highly heterogeneous tasks may require hybrid architectures or curriculum design.
• Decentralization and privacy: Distributed MTCL frameworks are robust and scalable but may require specialized communication protocols, especially under asynchronous or unreliable network conditions.
• Real-world deployment: Robustness to partial observability, delayed rewards, and shifting environment dynamics remains a central concern, requiring ongoing adaptation of MTCL techniques and integration with meta-learning or continual learning strategies.

Multi-Task Critic Learning is thus a critical and active area of research, intersecting scalable architecture design, optimization theory, and real-world reinforcement learning deployment, with empirical focus increasingly shifting toward scalable, dynamic, and distributed approaches that maximize both task performance and adaptability across changing environments.