Task-Specific Sigmoid BN for Multi-Task Learning

• The paper demonstrates TSσBN’s ability to modulate channel activations per task, achieving competitive or superior multi-task learning performance.
• It integrates sigmoid gating into standard batch normalization, enabling soft channel control in both convolutional networks and vision transformers.
• Empirical evaluations show TSσBN yields significant performance gains with negligible parameter overhead compared to more complex architectures.

Task-Specific Sigmoid Batch Normalization (TS$σ$BN) is a normalization-based mechanism for multi-task learning (MTL) that enables per-task allocation of network capacity by soft, channel-wise control of feature activations. TS$σ$BN replaces every shared normalization layer with small, sigmoid-gated per-task scales, thus allowing each task to softly claim or release network channels while keeping all other parameters fully shared. Despite its minimalist design, TS$σ$BN matches or outperforms more elaborate MTL architectures—in both convolutional and transformer-based models—across various vision benchmarks, with negligible parameter overhead and no need for task-specific experts or routing modules (Suteu et al., 23 Dec 2025).

1. Mathematical Formulation

TS$σ$BN builds on the standard batch normalization (BN) pipeline. Given activations $x \in \mathbb{R}^{B\times F\times H\times W}$, BN normalizes channel-wise:

$$\hat x_{b,k,i,j} = \frac{x_{b,k,i,j}-\mu_{B,k}}{\sqrt{\sigma^2_{B,k} + \varepsilon}}$$

where $\mu_{B,k}$ and $\sigma^2_{B,k}$ denote mean and variance for channel $k$ across the batch and spatial positions, and the affine transform is:

$$\mathrm{BN}(x;\gamma,\beta)_{b,k,i,j} = \gamma_k\,\hat x_{b,k,i,j} + \beta_k$$

In Sigmoid BatchNorm (σBN), the scale $\gamma_k$ is replaced by a sigmoid gate. Specifically:

$\sigmaBN(x;\alpha)_{b,k,i,j} = \sigma(\alpha_k)\,\hat x_{b,k,i,j}$

with $$\sigma(\alpha_k) = \frac{1}{1+e^{-\alpha_k}} \in (0,1)$$ and the bias term $\beta$ is omitted.

TS$σ$BN generalizes this approach to Multi-Task Learning by introducing a bank of per-task, per-channel parameters:

$g_{t,k} = \sigma(\alpha_{t,k})$

In the forward pass for task $t$, the normalized activation becomes:

$y_{t,k} = g_{t,k}\,\hat x_k$

An alternative, more general interpolation (noted as unnecessary in practice) is:

$$y_{t,k} = g_{t,k}\,\mathrm{BN}(x)_k + (1-g_{t,k})\,x_k$$

This enables each task to modulate how much it trusts the normalized versus raw activations.

2. Parameterization and Optimization

Initialization strategies depend on whether the network is randomly initialized or pretrained:

• From scratch: Set $\alpha_{t,k}=0$, yielding $g_{t,k} = 0.5$ for every task and channel.
• From a pretrained model: Copy BN scale $\gamma_k \in (0,1)$ and set $\alpha_{t,k} = \log\frac{\gamma_k}{1-\gamma_k}$ for each task; freeze the pretrained BN bias $\beta_k$.

Optimization disables weight decay on all TS$σ$BN parameters. To promote rapid capacity allocation, a large task-specific learning rate multiplier is applied (typically $\alpha_{\sigma BN} \approx 10^2$ for $\{\alpha_{t,k}\}$), while all other parameters use a base learning rate. The sigmoid nonlinearity stabilizes training even with such high multipliers.

3. Integration with Neural Architectures

TS$σ$BN is applicable to both convolutional neural networks (CNNs) and vision transformers:

• CNNs: Replace each shared BN layer (in encoder, decoder, or fusion modules) with a bank of $T$ σBN layers, selecting the appropriate parameters at each forward pass. All convolutional weights are fully shared.
• Vision Transformers:
  • LayerNorms are converted to Task-Specific σ-LayerNorms (TS$σ$LN) using per-task, per-channel sigmoid-gated scales.
  • TS$σ$BN or TS$σ$LN is inserted in patch embedding and patch merging blocks before/after linear layers.
  • In multi-scale fusion modules, shared BN layers are replaced with TS$σ$BN.

This integration is plug-and-play, requiring only minor changes to the codebase and no additional adapters, experts, or routing infrastructure.

4. Empirical Evaluation and Performance

TS$σ$BN demonstrates competitive or superior performance with negligible additional parameters across a range of MTL benchmarks and architectures. Salient empirical results include:

Setting	Tasks	Gains / Metrics	Parameter Overhead
NYUv2 (SegNet, from scratch)	3	$+6.93\%$ average MTL gain, segmentation mIoU $43.75$, depth $0.484$, surface normals $24.09^\circ$	$0.33\times$ single-task encoder
Cityscapes (DeepLabV3)	3	$+0.57\%$ gain, seg mIoU $56.45$, instance-seg $53.26$, disparity $0.814$	$0.60\times$ parameters
CelebA (CNN, from scratch)	40 (binary)	F1 $69.45$ ($+1.81\%$)	$3\%$ of single-task parameters
PascalContext (ViT-S+MoE)	5	Seg mIoU $77.12$, parts $64.73$, saliency $85.24$, normals $14.04^\circ$, boundary $70.00$ F	$29M$ params, $214$ MFLOPs
PascalContext (Swin-T PEFT)	4	$\Delta m\ +0.91\%$ (TS$σ$BN), $+1.63\%$ (TS$σ$BN+LoRA $r=16$)	$3.08$M (TS$σ$BN), $4.25$M (+LoRA)

In every setting—including both random and pretrained initialization and both CNNs and transformers—TS$σ$BN matches or outperforms more complex soft-sharing, mixture-of-experts, and PEFT-MTL methods, typically with less than $1\%$ increase in parameters and no additional FLOPs beyond gating multiplications (Suteu et al., 23 Dec 2025).

5. Analysis, Ablations, and Robustness

Ablation studies confirm several aspects of TS$σ$BN:

• Gating Nonlinearity: Direct affine scaling (TSBN, i.e., without sigmoid) yields moderate gains (e.g., $+4.4\%$ on NYUv2), but is unstable under high learning-rate multipliers. TS$σ$BN with sigmoid gating remains robust up to $\times 10^2$ multipliers, failing only at $\times 10^3$.
• Discriminative Learning Rates: Increasing $\alpha_{\sigma BN}$ to $10^2$ maximizes average multi-task gain before inducing hard-partitioning effects.
• Loss-Scale Perturbations: TS$σ$BN exhibits minimal variance in performance when individual task losses are artificially scaled, indicating stability against task-dominance issues without explicit loss weighting.

6. Interpretability and Capacity Allocation

TS$σ$BN provides a direct lens into network capacity allocation and filter specialization. The learned gate matrix $I\in\mathbb{R}^{T\times F}$, with $I_{t,i}=\sigma(\alpha_{t,i})$, quantifies per-task, per-channel utilization:

• Total task capacity: $C_t = \frac{1}{F}\sum_i I_{t,i}$
• Shared/independent capacity: Projecting $I_t$ into the shared subspace yields $C_t^{shared}$ and $C_t^{indep}$, with experiments indicating most capacity is shared.
• Task relationships: Pairwise cosine similarity of $I_t$ vectors reveals stable attribute clusters (e.g., hair, makeup, glasses on CelebA), matching semantic groupings.
• Filter specialization: A filter $i$ is specialized to task $t$ if $$\sigma(\alpha_{t,i}) / \sum_u \sigma(\alpha_{u,i}) > 0.5$$. Specialized filters concentrate in deeper layers, and pruning them most degrades the corresponding task's performance, validating their inferred importance.

A plausible implication is that this interpretability could support diagnosis and refinement of MTL architectures or serve as a quantitative indicator of interfering or synergistic task relationships.

7. Design Considerations, Limitations, and Scope

TS$σ$BN achieves simplicity by requiring no new convolutional layers, experts, routing, or complex solvers—only per-task scaling gates in normalization layers. The total parameter increase is under $1\%$ per task, and the only additional computation per sample is a channel-wise multiply-add.

Limitations are as follows:

• Gates are static for each task; input-dependent (dynamic) routing is not addressed.
• Hard gating (large $|\alpha|$) may over-partition the backbone as tasks diverge, potentially fragmenting capacity.
• Extensions to non-homogeneous domains (such as video vs. single-frame image tasks) may require separate normalization statistics.

TS$σ$BN demonstrates that per-task sigmoid-gated normalization suffices to resolve cross-task interference, dynamically allocate shared capacity, and provide interpretable representations—without the complexity of expert modules or dynamic router networks (Suteu et al., 23 Dec 2025).