---
title: Task-Specific Sigmoid BN for Multi-Task Learning
url: https://www.emergentmind.com/topics/task-specific-sigmoid-batch-normalization-ts-bn
type: topic
---

# Task-Specific Sigmoid BN for Multi-Task Learning

Task-Specific Sigmoid Batch Normalization (TS$σ$BN) is a normalization-based mechanism for multi-task learning (MTL) that enables per-task allocation of network capacity by soft, channel-wise control of feature activations. TS$σ$BN replaces every shared normalization layer with small, sigmoid-gated per-task scales, thus allowing each task to softly claim or release network channels while keeping all other parameters fully shared. Despite its minimalist design, TS$σ$BN matches or outperforms more elaborate MTL architectures—in both convolutional and transformer-based models—across various vision benchmarks, with negligible parameter overhead and no need for task-specific experts or routing modules [2512.20420].

## 1. Mathematical Formulation

TS$σ$BN builds on the standard batch normalization (BN) pipeline. Given activations $x \in \mathbb{R}^{B\times F\times H\times W}$, BN normalizes channel-wise:

$$
\hat x_{b,k,i,j} = \frac{x_{b,k,i,j}-\mu_{B,k}}{\sqrt{\sigma^2_{B,k} + \varepsilon}}
$$

where $\mu_{B,k}$ and $\sigma^2_{B,k}$ denote mean and variance for channel $k$ across the batch and spatial positions, and the affine transform is:

$$
\mathrm{BN}(x;\gamma,\beta)_{b,k,i,j} = \gamma_k\,\hat x_{b,k,i,j} + \beta_k
$$

In Sigmoid BatchNorm (σBN), the scale $\gamma_k$ is replaced by a sigmoid gate. Specifically:

$$
\sigmaBN(x;\alpha)_{b,k,i,j} = \sigma(\alpha_k)\,\hat x_{b,k,i,j}
$$

with $\sigma(\alpha_k) = \frac{1}{1+e^{-\alpha_k}} \in (0,1)$ and the bias term $\beta$ is omitted.

TS$σ$BN generalizes this approach to Multi-Task Learning by introducing a bank of per-task, per-channel parameters:

$$
g_{t,k} = \sigma(\alpha_{t,k})
$$

In the forward pass for task $t$, the normalized activation becomes:

$$
y_{t,k} = g_{t,k}\,\hat x_k
$$

An alternative, more general interpolation (noted as unnecessary in practice) is:

$$
y_{t,k} = g_{t,k}\,\mathrm{BN}(x)_k + (1-g_{t,k})\,x_k
$$

This enables each task to modulate how much it trusts the normalized versus raw activations.

## 2. Parameterization and Optimization

Initialization strategies depend on whether the network is randomly initialized or pretrained:

- **From scratch:** Set $\alpha_{t,k}=0$, yielding $g_{t,k} = 0.5$ for every task and channel.
- **From a pretrained model:** Copy BN scale $\gamma_k \in (0,1)$ and set $\alpha_{t,k} = \log\frac{\gamma_k}{1-\gamma_k}$ for each task; freeze the pretrained BN bias $\beta_k$.

Optimization disables weight decay on all TS$σ$BN parameters. To promote rapid capacity allocation, a large task-specific learning rate multiplier is applied (typically $\alpha_{\sigma BN} \approx 10^2$ for $\{\alpha_{t,k}\}$), while all other parameters use a base learning rate. The sigmoid nonlinearity stabilizes training even with such high multipliers.

## 3. Integration with Neural Architectures

TS$σ$BN is applicable to both convolutional neural networks (CNNs) and vision transformers:

- **CNNs:** Replace each shared BN layer (in encoder, decoder, or fusion modules) with a bank of $T$ σBN layers, selecting the appropriate parameters at each forward pass. All convolutional weights are fully shared.
- **Vision Transformers:**
  - LayerNorms are converted to Task-Specific σ-LayerNorms (TS$σ$LN) using per-task, per-channel sigmoid-gated scales.
  - TS$σ$BN or TS$σ$LN is inserted in patch embedding and patch merging blocks before/after linear layers.
  - In multi-scale fusion modules, shared BN layers are replaced with TS$σ$BN.
  
This integration is plug-and-play, requiring only minor changes to the codebase and no additional adapters, experts, or routing infrastructure.

## 4. Empirical Evaluation and Performance

TS$σ$BN demonstrates competitive or superior performance with negligible additional parameters across a range of MTL benchmarks and architectures. Salient empirical results include:

| Setting                      | Tasks        | Gains / Metrics            | Parameter Overhead              |
|------------------------------|-------------|----------------------------|---------------------------------|
| NYUv2 (SegNet, from scratch) | 3           | $+6.93\%$ average MTL gain, segmentation mIoU $43.75$, depth $0.484$, surface normals $24.09^\circ$ | $0.33\times$ single-task encoder |
| Cityscapes (DeepLabV3)       | 3           | $+0.57\%$ gain, seg mIoU $56.45$, instance-seg $53.26$, disparity $0.814$ | $0.60\times$ parameters        |
| CelebA (CNN, from scratch)   | 40 (binary) | F1 $69.45$ ($+1.81\%$)     | $3\%$ of single-task parameters |
| PascalContext (ViT-S+MoE)    | 5           | Seg mIoU $77.12$, parts $64.73$, saliency $85.24$, normals $14.04^\circ$, boundary $70.00$ F | $29M$ params, $214$ MFLOPs    |
| PascalContext (Swin-T PEFT)  | 4           | $\Delta m\ +0.91\%$ (TS$σ$BN), $+1.63\%$ (TS$σ$BN+LoRA $r=16$) | $3.08$M (TS$σ$BN), $4.25$M (+LoRA) |

In every setting—including both random and pretrained initialization and both CNNs and transformers—TS$σ$BN matches or outperforms more complex soft-sharing, mixture-of-experts, and PEFT-MTL methods, typically with less than $1\%$ increase in parameters and no additional FLOPs beyond gating multiplications [2512.20420].

## 5. Analysis, Ablations, and Robustness

Ablation studies confirm several aspects of TS$σ$BN:

- **Gating Nonlinearity:** Direct affine scaling (TSBN, i.e., without sigmoid) yields moderate gains (e.g., $+4.4\%$ on NYUv2), but is unstable under high learning-rate multipliers. TS$σ$BN with sigmoid gating remains robust up to $\times 10^2$ multipliers, failing only at $\times 10^3$.
- **Discriminative Learning Rates:** Increasing $\alpha_{\sigma BN}$ to $10^2$ maximizes average multi-task gain before inducing hard-partitioning effects.
- **Loss-Scale Perturbations:** TS$σ$BN exhibits minimal variance in performance when individual task losses are artificially scaled, indicating stability against task-dominance issues without explicit loss weighting.

## 6. Interpretability and Capacity Allocation

TS$σ$BN provides a direct lens into network capacity allocation and filter specialization. The learned gate matrix $I\in\mathbb{R}^{T\times F}$, with $I_{t,i}=\sigma(\alpha_{t,i})$, quantifies per-task, per-channel utilization:

- **Total task capacity:** $C_t = \frac{1}{F}\sum_i I_{t,i}$
- **Shared/independent capacity:** Projecting $I_t$ into the shared subspace yields $C_t^{shared}$ and $C_t^{indep}$, with experiments indicating most capacity is shared.
- **Task relationships:** Pairwise cosine similarity of $I_t$ vectors reveals stable attribute clusters (e.g., hair, makeup, glasses on CelebA), matching semantic groupings.
- **Filter specialization:** A filter $i$ is specialized to task $t$ if $\sigma(\alpha_{t,i}) / \sum_u \sigma(\alpha_{u,i}) > 0.5$. Specialized filters concentrate in deeper layers, and pruning them most degrades the corresponding task's performance, validating their inferred importance.

A plausible implication is that this interpretability could support diagnosis and refinement of MTL architectures or serve as a quantitative indicator of interfering or synergistic task relationships.

## 7. Design Considerations, Limitations, and Scope

TS$σ$BN achieves simplicity by requiring no new convolutional layers, experts, routing, or complex solvers—only per-task scaling gates in normalization layers. The total parameter increase is under $1\%$ per task, and the only additional computation per sample is a channel-wise multiply-add.

Limitations are as follows:

- Gates are static for each task; input-dependent (dynamic) routing is not addressed.
- Hard gating (large $|\alpha|$) may over-partition the backbone as tasks diverge, potentially fragmenting capacity.
- Extensions to non-homogeneous domains (such as video vs. single-frame image tasks) may require separate normalization statistics.

TS$σ$BN demonstrates that per-task sigmoid-gated normalization suffices to resolve cross-task interference, dynamically allocate shared capacity, and provide interpretable representations—without the complexity of expert modules or dynamic router networks [2512.20420].

Source: https://www.emergentmind.com/topics/task-specific-sigmoid-batch-normalization-ts-bn