Task Vector in Distributed Learning

• Task vector is a mathematical representation that aggregates shared and task-specific SVM parameters for decentralized multi-task learning.
• ADMM optimizes these vectors iteratively by ensuring consensus across nodes, preserving privacy and reducing communication overhead.
• Empirical results, such as on MNIST splits, demonstrate that task vectors effectively lower risk and adapt dynamically in distributed transfer learning.

A task vector is a mathematical or architectural entity that encapsulates task-specific information—such as decision boundaries, parameter adjustments, or feature representations—used within a multi-task, transfer learning, or distributed learning framework. In the context of consensus-based distributed transfer learning for multi-agent multi-task systems, the task vector of a node-task pair is the fundamental vehicle through which knowledge transfer, efficiency, and privacy are achieved.

1. Task Vector: Formal Representation and Decomposition

In the consensus-based distributed transfer SVM (DTSVM) framework, each classification task t at node v is parameterized by a "task vector" $\mathbf{r}_{vt}$, which consolidates both common (shared) and task-specific model variables:

• $$\mathbf{r}_{vt} = [\mathbf{w}_{0vt}; b_{0vt}; \mathbf{w}_{vt}; b_{vt}] \in \mathbb{R}^{2p + 2}$$
  • $[\mathbf{w}_{0vt}; b_{0vt}]$ encodes the common parameter component shared among tasks and across nodes,
  • $[\mathbf{w}_{vt}; b_{vt}]$ comprises the task-specific adjustments.

Consensus constraints enforce alignment of the common component across all tasks at a node and across adjacent nodes for each task, ensuring distributed consistency without central aggregation.

2. Optimization via Alternating Direction Method of Multipliers (ADMM)

To optimize the task vectors in a decentralized setting, the DTSVM employs ADMM to decompose the joint problem into local quadratic programs:

• Local objective for node v, task t:

$$\min_{\mathbf{r}_{vt}, \xi_{vt}} \frac{\epsilon_1}{2} \mathbf{r}_{vt}^T \mathbf{M}_1 \mathbf{r}_{vt} + \frac{\epsilon_2}{2} \mathbf{r}_{vt}^T \mathbf{M}_2 \mathbf{r}_{vt} + VTC \xi_{vt}$$

Subject to SVM and consensus constraints.

• Iterative updates per ADMM involve:
  • Solving for Lagrange multipliers $\lambda_{vt}$
  • Updating the task vectors:

  $$\mathbf{r}_{vt}^{(k+1)} = \mathbf{U}_{vt}^{-1} \left( [\mathbf{I,I}]^T \mathbf{X}_{vt}^T \mathbf{Y}_{vt} \lambda_{vt}^{(k+1)} - \mathbf{f}_{vt}^{(k)} \right)$$ - Updating consensus multipliers $\alpha_{vt}^{(k+1)}$, $\beta_{vt}^{(k+1)}$, which enforce agreement both within nodes (across tasks) and across network edges (across neighboring nodes).

This iterative process enables each node to update its task vectors based solely on local data and minimal information from neighbors—never exchanging raw samples.

3. Privacy, Communication Efficiency, and Scalability

The central role of the task vector in this system is to:

• Preserve privacy: Only task vectors—low-dimensional aggregates of decision variables—are transmitted, preventing exposure of sensitive local data to the network.
• Reduce communication: Compared to communicating full local datasets or raw gradients, sharing task vectors significantly lowers overhead, a crucial property for large-scale, bandwidth-limited networks.
• Enable scalability: The system accommodates hundreds or thousands of nodes and tasks, as each computation and communication step scales with vector dimensions rather than dataset sizes.

4. Numerical Performance and Real-Time Task Adaptivity

Extensive experiments on MNIST multi-class splits validate the efficacy of the task vector approach:

Scenario	Metric/Result
Target tasks with label imbalance	DTSVM achieves lower risk than centralized and distributed SVMs
Nodes without access to source data	Risk reduced via knowledge transfer in received task vectors
Real-time task insertion/removal	Tasks can be dynamically added/dropped without restarting algorithms
Data-scarce targets	Task vector transfer from rich source tasks closes the risk gap

The approach demonstrably improves under-represented or poorly-labeled tasks by communicating only the sufficient task vectors needed for model adaptation.

5. Consensus Constraints and Dynamic Network Operation

Two levels of consensus constraints are enforced:

• Intra-node (across tasks):

$$[\mathbf{I}, 0]\mathbf{r}_{vt} = [\mathbf{I}, 0]\mathbf{r}_{vs},\quad \forall\ t, s\ \text{at node } v$$

These force agreement on shared knowledge within the node.

• Inter-node (across neighbors for same task):

$$\mathbf{r}_{vt} = \mathbf{r}_{ut},\quad u \in \mathcal{B}_v$$

These propagate consistent task-specific decisions throughout the network.

Through repeated ADMM updates, the global consensus on common knowledge is reached while preserving task individuality—despite asynchronous node arrivals, departures, or data skew.

6. Applications and Research Implications

The task vector architecture is particularly suited for settings where data are distributed, privacy is required, and tasks are numerous and possibly dynamically varying:

• Federated learning with multi-task personalization: Each device or agent adapts global models to its local task(s) via task vector updates, maintaining data locality.
• Wireless sensor and smart grid networks: When sensors operate over distinct but related classification problems, task vectors enable knowledge sharing to improve reliability and coverage.
• Decentralized transfer learning in ad-hoc or peer-to-peer environments: Real-time task vector adaptation supports dynamic task and node churn.

Potential extensions include:

• Accommodating nonlinear models via kernelization or deep architectures.
• Enhanced privacy regimes (e.g., quantized, encrypted, or differentially private task vectors).
• Hierarchical consensus or multi-level task vector aggregation for more complex networks.

7. Summary Table: Task Vector Properties

Property	Description/Implication
Structure	Aggregates shared and specific SVM (w,b) parameters
Privacy	Transmits only decision variables, not raw features/labels
Scalability	Communication and computation scale with vector—not data—size
Flexibility	Supports dynamic task entry/exit, asynchronous operation
Performance	Reduces risk on data-poor or unbalanced tasks through transfer
Consensus Role	Enforces intra- and inter-node agreement for global consistency

Task vectors thus constitute the central representation for consensus-based transfer learning in distributed, privacy-constrained, and dynamic multi-task systems, providing a mathematically principled mechanism for scalable, adaptive, and efficient knowledge transfer.