Multi-task Soft Actor-Critic

• Multi-task SAC is an entropy-regularized actor-critic framework that extends SAC to handle multiple related or sequential tasks through task conditioning and reward reparameterization.
• The approach uses a shared SAC backbone with innovations such as task embeddings, cooperative critic coupling, and replay buffer reuse to improve training efficiency and performance across varied domains.
• Empirical studies in robotics, maze navigation, and quadcopter control demonstrate faster convergence, higher success rates, and robust multitask performance compared to single-task SAC.

to=arxiv.search ็ตทรู 菲龙json {"query":"Multi-task Soft Actor-Critic multitask SAC arXiv", "max_results": 5, "sort_by": "relevance"}}

to=arxiv.search ությանը 彩神争霸输钱json {"query":"Cooperative Soft Actor Critic multi-stage tasks arXiv", "max_results": 5, "sort_by": "relevance"}}

Multi-task Soft Actor-Critic (SAC) denotes a class of entropy-regularized, off-policy actor-critic methods that extend Soft Actor-Critic beyond a single task. In the literature, this extension takes more than one form. One line of work trains a single policy over a family of MDPs, either through explicit task conditioning or through task-specific reward weights. Another line of work decomposes a long-horizon problem into a fixed sequence of subtasks and modifies each subpolicy so that it optimizes not only its own critic but also the next stage’s critic. Across these formulations, the common backbone is SAC’s stochastic policy optimization, critic learning from replay, and entropy maximization, while the multi-task component enters through task embeddings, reward-feature recomputation, or cooperative coupling across subtasks (Liu et al., 11 Mar 2025, Nehme et al., 2023, Erskine et al., 2020).

1. Problem formulations and scope

Multi-task SAC is not a single canonical algorithm. In the available formulations, the problem setup ranges from a task distribution over related manipulation problems, to a family of reward-weighted attitude-control tasks, to a manually decomposed sequence of subtasks that must be completed in the same fixed order.

In the Meta-World formulation, each task is treated as a separate MDP within a task distribution $p(\mathcal{T})$, and the multi-task extension is implemented through a single task-conditioned policy $\pi(a\mid s,z)$, where $z$ is a task embedding. In the quadcopter formulation, multitask RL is defined as a family of MDPs

$$\mathcal{M}_w \equiv<\mathcal{S},\mathcal{A},p,r_w,\gamma>,$$

with task-specific reward

$$r_w(s_t,a_t,s_{t+1})=\phi(s_t,a_t,s_{t+1})^\top \cdot w,$$

where $\phi(\cdot)$ is a shared feature vector and $w$ encodes the task. In the cooperative multi-stage formulation, the task is manually decomposed into $n$ subtasks, each with its own reward signal, completion signal, policy, and critic, and the subtasks are always completed in the same fixed order (Liu et al., 11 Mar 2025, Nehme et al., 2023, Erskine et al., 2020).

These formulations differ materially in what “multi-task” means. In one case, many tasks share one policy and one transition dataset. In another, several subtask-specific agents are trained so that each stage leaves the system in a state useful for the following stage. This suggests that multi-task SAC should be understood as a family of SAC-based design patterns rather than a single architecture.

Formulation	Task organization	Defining mechanism
Cooperative Soft Actor Critic (CSAC)	Manually decomposed sequence of subtasks in fixed order	Policy for subtask $i$ maximizes its own critic and the next subtask’s critic
Meta-World MT-SAC	7 selected tasks from MT-10 / Meta-World	Single task-conditioned policy $\pi(a\mid s,z)$
Quadcopter multitask SAC	57 tasks split across tracking, smooth tracking, and stabilization task sets	Shared feature basis with task-weighted reward $\pi(a\mid s,z)$0

2. Core optimization structure

The SAC backbone is retained in all of these methods. The base policy objective used in the Meta-World work is

$\pi(a\mid s,z)$1

with automatic entropy tuning

$\pi(a\mid s,z)$2

The multi-task objective is then written as the average expected return over tasks sampled from $\pi(a\mid s,z)$3,

$\pi(a\mid s,z)$4

with an analogous expectation for the Q-function (Nehme et al., 2023).

The quadcopter formulation keeps the entropy-regularized SAC structure but moves the multi-task mechanism to the reward side and the policy input side. The replay buffer stores only $\pi(a\mid s,z)$5, not raw rewards. For each minibatch, rewards for multiple tasks are recomputed as

$\pi(a\mid s,z)$6

and the same transitions are reused across tasks. The policy is conditioned on the current state $\pi(a\mid s,z)$7, the task vector $\pi(a\mid s,z)$8, and an environment latent $\pi(a\mid s,z)$9 from RMA. This makes the task abstraction reward-feature based rather than separate-code-path based (Liu et al., 11 Mar 2025).

The cooperative multi-stage variant changes the policy objective directly. Standard SAC critic and policy losses are retained in form, but the policy for stage $z$0 is trained with

$z$1

where

$z$2

and

$z$3

Here $z$4 is the cooperative ratio. The final policy in the sequence has no next policy to cooperate with, so it is trained only on its own critic. This is a direct mechanism for inducing cross-stage cooperation through the policy loss itself rather than through explicit communication between agents (Erskine et al., 2020).

3. Task representation, reward parameterization, and policy architectures

A central design choice in multi-task SAC is how tasks are represented. In the Meta-World work, the observation is concatenated with a task embedding, increasing the input dimension from $z$5 to $z$6. Three task embeddings were tested: one-hot encoding, sine encoding, and learned encoding. The reported result is that sine encoding performed best, while learned embeddings performed unexpectedly poorly (Nehme et al., 2023).

In the quadcopter work, tasks are represented by a normalized weight vector $z$7 applied to a shared feature basis. The state for attitude control is

$z$8

and the action is

$z$9

The shared feature vector is engineered from roll/pitch tracking error, yaw tracking error, roll/pitch angular velocity, yaw angular velocity, and action smoothness, and representative task weights are given for tracking, smooth tracking, and stabilization. The paper notes a total of 57 tasks, split across these task sets, with task vectors normalized and sampled uniformly (Liu et al., 11 Mar 2025).

That same work introduces a Graph Convolutional Network policy. The policy takes the state $$\mathcal{M}_w \equiv<\mathcal{S},\mathcal{A},p,r_w,\gamma>,$$0, the task vector $$\mathcal{M}_w \equiv<\mathcal{S},\mathcal{A},p,r_w,\gamma>,$$1, and the environment latent $$\mathcal{M}_w \equiv<\mathcal{S},\mathcal{A},p,r_w,\gamma>,$$2, embeds them as node features, and applies graph convolution through a learnable adjacency matrix: $$\mathcal{M}_w \equiv<\mathcal{S},\mathcal{A},p,r_w,\gamma>,$$3 The resulting action-node embeddings are decoded into Gaussian policy parameters,

$$\mathcal{M}_w \equiv<\mathcal{S},\mathcal{A},p,r_w,\gamma>,$$4

The stated motivation is that graph structure can encode which inputs should influence which control outputs, reducing gradient conflict and supporting a compact policy suitable for onboard deployment (Liu et al., 11 Mar 2025).

The cooperative multi-stage formulation uses a different representational strategy. Each subtask has its own policy $$\mathcal{M}_w \equiv<\mathcal{S},\mathcal{A},p,r_w,\gamma>,$$5, critic $$\mathcal{M}_w \equiv<\mathcal{S},\mathcal{A},p,r_w,\gamma>,$$6, and replay buffer $$\mathcal{M}_w \equiv<\mathcal{S},\mathcal{A},p,r_w,\gamma>,$$7. Rather than a shared universal policy, the coupling is produced by the cooperative critic combination in the policy loss. This marks a substantive distinction between task-conditioned MT-SAC and sequential cooperative SAC: the former emphasizes parameter sharing across tasks, whereas the latter emphasizes value coupling across neighboring stages (Erskine et al., 2020).

4. Training organization and data reuse

The training pipeline varies sharply across multi-task SAC variants, and this difference is often more important than the shared SAC backbone.

In the quadcopter formulation, training is performed in IsaacGym, enabling over 1,000 parallel simulations on a single GPU. All agents are implemented in PyTorch, hyperparameters are tuned with Bayesian optimization, and each experiment is repeated three times with different seeds. The replay buffer stores transitions without raw rewards, and the same transitions are reused across many tasks by recomputing $$\mathcal{M}_w \equiv<\mathcal{S},\mathcal{A},p,r_w,\gamma>,$$8 from the shared feature basis. This is the core sample-efficiency mechanism of that formulation (Liu et al., 11 Mar 2025).

In the Meta-World formulation, MT-SAC is trained on 7 selected tasks for 10,000 episodes per task. The training pipeline is: train MT-SAC on the selected tasks, collect the final replay buffer, select the best task embedding based on performance, and use the resulting policy and replay buffer as prior data for MT-QWALE. In that work, MT-SAC is therefore both a multi-task learner and a source of prior experience for a subsequent single-life RL pipeline (Nehme et al., 2023).

In CSAC, data gathering and training are organized by subtask. Algorithm 1 initializes $$\mathcal{M}_w \equiv<\mathcal{S},\mathcal{A},p,r_w,\gamma>,$$9 agents, each with a policy, critic, and replay buffer; the current subpolicy acts until its subtask is complete; and all transitions for that subtask are stored in that subtask’s replay buffer. Algorithm 2 samples minibatches from the relevant replay buffer, trains critics, normalizes critic values across the minibatch, and updates the policy with the convex combination of normalized current and next critic values. Although the pseudocode shows single critics for clarity, the implementation uses twin critics and target networks, just like standard SAC. The overall loop is epoch-based and separates an exploration environment, where the agent can start anywhere in the full maze, from an evaluation environment, where the agent starts only in the initial area (Erskine et al., 2020).

These training organizations instantiate different notions of reuse. Quadcopter MT-SAC reuses transitions across task-weighted rewards. Meta-World MT-SAC reuses a shared policy and then repurposes its replay buffer as prior data. CSAC reuses structure across consecutive subtasks by training critics from data from the current and previous subtasks.

5. Empirical domains and reported performance

The empirical literature on multi-task SAC spans maze navigation, robotic manipulation, and quadcopter attitude control, with different evaluation criteria but a shared concern with end-to-end success and training efficiency.

In the multi-room maze domain for CSAC, each room is a subtask, the goal of each room is to exit into the next room, and each room has two doors, but one may lead to a dead end in the next room. In the two-room maze, CSAC and uncooperative methods learned a successful policy in a similar time initially, while the single-agent SAC took more than 3 times as long to reach similar performance. In the three-room and four-room mazes, the single-agent method failed to learn any successful policy within 3 million training steps. The paper’s summary statement is that CSAC achieves at least 20% higher success rate than uncooperative policies and converges at least 4 times faster than the single-agent method (Erskine et al., 2020).

In the quadcopter domain, the two jointly trained tasks are smooth attitude tracking and aggressive stabilization from arbitrary states. The reported findings are that multitask SAC converges faster and more stably on the tracking evaluation task than the single-task baseline, that increasing task diversity improves performance, and that even in single-task learning the graph architecture outperforms a plain feedforward architecture. The network-size study reports that 12 neurons per layer gives weaker sample efficiency, 24 neurons per layer gives the best tradeoff, and 36 neurons per layer gives no clear improvement. The final deployed policy has two layers, 24 neurons per layer, and 697 parameters total, runs on a Pixhawk-6X flight controller with an STM32H753 at approximately 400 Hz, and stabilizes the quadcopter after rotor re-engagement in all 13 trials (Liu et al., 11 Mar 2025).

In the Meta-World study, MT-SAC is evaluated on Pick and Place, Window Open, Window Close, Drawer Open, Drawer Close, Push, and Button Press. The reported MT-SAC findings are that sine encoding gives the best performance, that MT-SAC performs better on simple movement-only tasks such as Window Close and Drawer Close, and that it struggles on more complex tasks that require multiple primitives, especially Pick and Place and Drawer Open. In the same work, MT-SAC is compared against the downstream MT-QWALE system in novelty environments with randomly varied object positions within a radius of 0.3, and MT-QWALE is reported to show a large improvement over MT-SAC. This positions MT-SAC as a strong multi-task prior learner, but not a complete solution to distribution shift in that setting (Nehme et al., 2023).

6. Limitations, sensitivities, and broader context

A recurring limitation is sensitivity to the specific mechanism used to share information across tasks. In CSAC, performance depends strongly on the cooperative ratio $$r_w(s_t,a_t,s_{t+1})=\phi(s_t,a_t,s_{t+1})^\top \cdot w,$$0. The paper notes that this ratio is highly sensitive and environment-specific; around $$r_w(s_t,a_t,s_{t+1})=\phi(s_t,a_t,s_{t+1})^\top \cdot w,$$1, performance is often poor; and the best ratio is domain-specific and even subtask-specific. In the 3-room domain, tuning the cooperative ratio separately for each cooperative policy gave the best reported result, with first policy $$r_w(s_t,a_t,s_{t+1})=\phi(s_t,a_t,s_{t+1})^\top \cdot w,$$2 and second policy $$r_w(s_t,a_t,s_{t+1})=\phi(s_t,a_t,s_{t+1})^\top \cdot w,$$3. In Meta-World, learned embeddings performed poorly, and the broader motivation section states that multi-task algorithms including multi-task SAC demonstrate optimal performance only when operating within an environment or observation space that exhibits a similar distribution (Erskine et al., 2020, Nehme et al., 2023).

Another important limitation is conceptual rather than numerical: not every successful use of SAC across multiple control problems constitutes classical multi-task RL. The legged-locomotion work on RSL-RL-SAC explicitly states that it is not about a classical multi-task RL benchmark with explicit task IDs or task-conditioned networks; instead, it shows that a single SAC framework can be made to work across multiple locomotion tasks and robot platforms using the same reward functions and one shared hyperparameter setting. This distinction matters because it separates task-conditioned joint optimization from task-agnostic algorithmic robustness across environments (Sabatini et al., 24 May 2026).

Within the broader MTRL literature, SAC and its variants are described as dominant because replay buffer reuse makes off-policy learning more sample efficient and target networks smooth TD targets over time. At the same time, comparative work on Meta-World+ argues that many SAC-family methods have weak tail-task performance, including the claim that every published non-ARS multi-task SAC method floors at 0% success rate on the ten hardest MT50 tasks. This suggests that SAC’s historical role as the default backbone in MTRL reflects both genuine off-policy advantages and the specifics of benchmark optimization dynamics (Li et al., 12 May 2026).

A plausible implication is that future multi-task SAC variants may combine task conditioning with more adaptive control of entropy and optimization geometry. Meta-SAC does not claim a multi-task SAC result, but its metagradient framework is described as naturally compatible with learning task-adaptive entropy temperatures, which would matter when different tasks require different exploration schedules (Wang et al., 2020).