Causal Instance Normalization

• Causal instance normalization is an adaptive strategy that recalculates feature statistics at test time to mitigate extraneous variable effects.
• The method improves classification accuracy by reducing train-test feature distribution mismatches, with gains of 10-15% on datasets like CIFAR-10C, JIGSAWS, and StrokeRehab.
• It employs batch-based or instance-based normalization to condition out nuisance variance, fostering invariant deep representations for robust predictions.

Causal instance normalization is best understood as a causal-inspired interpretation of adaptive, instance-based feature normalization in which feature statistics are recomputed so that variation induced by an extraneous variable $z$ is reduced before prediction of the task label $y$. In the evidence considered here, the central motivating result is not a formal Pearl-style causal model, but an empirical and methodological argument: when an extraneous variable changes the data distribution while remaining irrelevant to the target, normalization statistics fixed from the training set can induce a train-test mismatch, whereas statistics adapted to the current test instance or test group can improve robustness and often increase invariance (Kaku et al., 2020).

1. Conceptual scope and causal reading

The underlying problem is classification when observations depend not only on the task-relevant label $y$, but also on an extraneous variable $z$ that should not determine the prediction. In the studied settings, $z$ is surgeon identity in JIGSAWS, patient identity in StrokeRehab, and corruption type or severity in CIFAR-10C. The key issue is that $z$ changes the distribution of the raw inputs and therefore shifts intermediate feature distributions. If the model is trained under one set of extraneous-variable values and tested under another, the internal feature statistics can change substantially, degrading classification (Kaku et al., 2020).

A causal-style interpretation follows naturally from this setup. A reasonable causal reading is that $z$ is a nuisance cause of observed feature variation, whereas $y$ is the prediction target. Under that reading, deep features should ideally encode information about $y$ while being invariant to $z$. Adaptive normalization then acts like a mechanism for conditioning out, or factoring out, the influence of $y$0 on feature statistics. This suggests that “causal instance normalization” is less a formally identified causal estimator than a representation-level strategy for nuisance removal motivated by causal invariance.

The same source also makes the limits of that interpretation explicit. It does not provide a formal causal graphical model or explicit causal-identification argument. Its contribution is empirical and methodological rather than formal causal theory. Accordingly, any encyclopedic use of the term should distinguish between causal motivation and causal proof.

2. Extraneous variables, feature-statistic shift, and normalization mismatch

The main empirical claim is that extraneous variables can degrade the performance of deep-learning models because they induce a mismatch between the feature statistics assumed by the normalization layer and the statistics encountered at inference. The paper frames this as a train-test shift problem: the model is trained on data with certain values of $y$1, inference occurs on data with different or unknown values of $y$2, and deep models—especially those with normalization layers—can suffer because the learned feature-statistic assumptions no longer hold (Kaku et al., 2020).

This problem is described as distinct from standard domain adaptation because access to target-domain unlabeled data during training is not assumed. That distinction matters conceptually. The method does not attempt to align source and target domains during training. Instead, it changes how internal feature statistics are estimated at inference.

Batch normalization provides the central contrast. Its standard form is

$y$3

followed by learned affine parameters,

$y$4

During training, $y$5 and $y$6 are computed from the current mini-batch and running averages are maintained. During inference, batch normalization uses the running training-set estimates of $y$7 and $y$8. For standard datasets such as CIFAR-10 and Fashion-MNIST, where training and test distributions are similar, this works well. For datasets affected by extraneous variables, however, the training-set statistics are no longer representative, normalized test features do not have mean $y$9 and variance $y$0, and this mismatch correlates with reduced accuracy.

Instance normalization changes the averaging regime. Statistics are computed per example rather than globally from training data, which makes normalization adaptive to the current input instance. In the presence of varying extraneous variables, this is the critical property: normalization no longer depends on fixed training-set statistics that may be invalid under nuisance-variable shift.

3. Adaptive feature normalization as the operative mechanism

The main method is adaptive feature normalization, defined by estimating normalization statistics at inference time from the current test data rather than fixing them from training (Kaku et al., 2020). Two variants are described.

Batch-based adaptive normalization applies when the extraneous variable $y$1 is known at test time. Test samples are grouped by $y$2, feature means and standard deviations are estimated from the group, and those statistics are used to normalize held-out samples from the same group. The examples given are grouping by surgeon in JIGSAWS, by patient in StrokeRehab, and by corruption type in CIFAR-10C.

Instance-based adaptive normalization applies when $y$3 is unknown or not easy to group by. Statistics are estimated from each individual test instance, making normalization adaptive to that instance’s own nuisance context. The practical caveat is equally explicit: if one averages only over a single instance, the statistics are noisier. For that reason, the paper suggests either estimating test statistics from a batch of examples that share the same extraneous-variable value, or using instance-based averaging during training as well, which corresponds to instance normalization.

The generic normalization form remains

$y$4

with optional affine post-transformation,

$y$5

The paper also compares mean and standard deviation $y$6 with mean-square statistics $y$7, and reports that both behave similarly in most experiments.

Architecturally, the method is implemented by swapping normalization layers in standard CNNs. DenseNet variants are used for JIGSAWS and StrokeRehab. VGG-19, ResNet-50, and DenseNet-121 are used for CIFAR-10, CIFAR-10C, and Fashion-MNIST. For sensor data, the setup includes input windows of 200 time steps, dense blocks with convolution, normalization, and ReLU, trained with cross-entropy and Adam. For image data, standard architectures are used with batch norm layers replaced by the normalization variant under study.

4. Experimental evidence and invariance behavior

The empirical evidence is organized around three datasets with strong extraneous-variable effects, plus two homogeneous baselines. On the baselines, CIFAR-10 and Fashion-MNIST, non-adaptive batch-based normalization performs best or near-best, and adaptive methods do not help much and can be slightly worse. This is important because it restricts the claim: adaptive normalization is not universally better; it helps when there is nuisance-variable shift (Kaku et al., 2020).

Dataset	Extraneous variable	Reported result
JIGSAWS	surgeon identity	adaptive normalization improves accuracy by about 10%
StrokeRehab	patient identity	adaptive normalization improves accuracy by about 15%
CIFAR-10C	corruption type / severity	adaptive normalization improves accuracy by about 10%

For JIGSAWS, the task is classification of surgical gestures from kinematic sensor data, with training on one set of surgeons and testing on unseen surgeons. Instance-based adaptive normalization often performs best. In the main table, instance-based adaptive normalization is around 72.7%, versus about 67.2% for non-adaptive batch-based normalization. Across multiple splits, instance-based adaptive normalization consistently wins. The reported interpretation is that surgeon-specific movement patterns strongly contaminate the feature statistics.

For StrokeRehab, the task is classification of movement primitives from IMU sensor data, with disjoint patients across train, validation, and test. Adaptive batch-based normalization is especially strong, reaching roughly 61.6%, compared with 48.3% for non-adaptive batch-based normalization. Instance-based normalization is also better than non-adaptive, though the gap varies. The stated interpretation is that patient-specific impairment patterns induce large distribution shifts.

For CIFAR-10C, the task is classification of corrupted CIFAR-10 images, trained on clean CIFAR-10 and tested on corrupted versions. Example numbers are around 81.9% for adaptive batch-based $y$8 versus 72.6% for non-adaptive batch-based $y$9. The improvements hold across ResNet-50, VGG-19, and DenseNet-121. The stated interpretation is that corruption type changes low-level feature statistics.

The same study also evaluates whether learned features encode the extraneous variable by training a separate decoder to predict the extraneous variable from penultimate-layer features. Lower decoding accuracy means greater invariance. Adaptive normalization generally reduces decoding accuracy, indicating more invariant features; this is especially clear for JIGSAWS and CIFAR-10C. StrokeRehab is more mixed, partly because non-adaptive normalization already performs poorly there. Deeper layers are more invariant than earlier ones. This suggests that the benefits of causal-style normalization are not limited to classification accuracy but extend to reduced nuisance information in representation space.

5. Relation to instance-centered hybrid normalization: ILN

A related but non-causal line of work is "U-Net Training with Instance-Layer Normalization" (Zhou et al., 2019). That paper proposes Instance-Layer Normalization (ILN), a learned hybrid normalization layer built from instance normalization and layer normalization, with a sigmoid-gated interpolation followed by cascading Group Normalization with 16 groups (GN16). Its motivation is not nuisance causality but the claim that different normalization strategies are beneficial at different layers and that fixed normalization per layer can be suboptimal.

ILN computes an IN-normalized feature map and an LN-normalized feature map, mixes them with a learnable gate, and then applies GN16 to re-normalize the mixed feature map. The gate is defined through

$z$0

and the mixed feature map is

$z$1

The paper positions ILN against Batch-Instance Normalization (BIN), identifying two issues in BIN: the Clip function is not differentiable at input values of $z$2 and $z$3, and the combined feature map is not with a normalized distribution, which is harmful for signal propagation in DCNN. ILN addresses those issues by replacing Clip with Sigmoid and adding GN16 after the mixture.

Empirically, ILN is evaluated on Right Ventricle and Left Ventricle segmentation using U-Net. The main takeaway is that ILN is best on 5 of 6 splits in the table, with the only exception being LV-3, where IN is slightly better. The paper states that this shows noticeable accuracy improvements, roughly around 2% DSC in some cases. An additional observation is that different layers converge to different gate values, indicating diverse normalization behavior across layers.

For the present topic, the key relationship is negative as much as positive. ILN is related to instance normalization because IN is a core branch, and it is conceptually similar to later approaches that adapt normalization based on input conditions or task context. However, it is not a causal normalization method in the strict sense. It does not discuss causality, interventions, causal effects, or disentanglement of nuisance factors in a causal framework, and its statistics are deterministic normalization statistics from spatial and channel aggregates rather than causal variables.

6. Limits, misconceptions, and current significance

Several misconceptions are explicitly ruled out by the evidence. The first is that batch normalization is intrinsically inferior. The stronger claim is narrower: non-adaptive normalization can fail under nuisance-variable shift, whereas adaptive normalization—especially instance-based when needed—better matches train and test statistics (Kaku et al., 2020). On homogeneous benchmarks such as CIFAR-10 and Fashion-MNIST, standard batch-based normalization works well and adaptive alternatives can be slightly worse.

The second misconception is that instance normalization, by itself, constitutes a causal method. The evidence supports a causal-inspired interpretation of normalization as nuisance removal, but it does not prove causal identifiability or derive normalization from a structural causal model. The relevant paper explicitly supports a practical and conceptual claim: if an extraneous variable causally affects observed feature statistics but is not relevant to the label, then normalizing features using statistics adapted to the current nuisance context can help remove spurious variation and improve robustness. This is aligned with the motivation one would associate with causal instance normalization, but it remains an empirical and methodological position rather than a formal causal theorem.

The third misconception is that any learned mixture of normalization schemes should be read causally. ILN provides a useful counterexample. It shows that normalization can be adaptively combined through sigmoid-gated interpolation plus GN16, and that such combinations can improve segmentation performance, but its goal is training stability and layerwise flexibility rather than causal invariance (Zhou et al., 2019).

Taken together, these results place causal instance normalization in a precise conceptual niche. It denotes a family of instance-based or adaptively estimated normalization strategies motivated by the desire to reduce representation dependence on nuisance variables such as surgeon identity, patient identity, or corruption type. Its strongest current support is empirical: correcting the mismatch in feature statistics can improve classification and can make representations less informative about extraneous variables. Its principal limitation is equally clear: the causal interpretation is supported as motivation and as a reading of the observed invariance effects, not as a formally identified causal model.