Beyond receptive fields: sequence-pooled normalization can supply most of a sequence labeler's context

Abstract: A convolutional sequence labeler's receptive field is routinely treated as the extent of the model's usable context: it sets dilation schedules, bounds streaming horizons, and underwrites locality claims. However, we show that this can be false: when a normalization layer computes statistics from the current input along the sequence at inference, those statistics open a sequence-spanning path that bypasses the convolutional receptive field to provide global context. We derive this from the layer's Jacobian (the criterion needs no experiment), and what the path carries has a closed form. On a synthetic labeling process with computable optima, the global summary that a sequence-spanning normalization encodes already supplies almost all of what a larger receptive field would buy where labels come in long runs: a network reaching 9 positions comes within 0.009 of the whole-sequence optimum, against a near-chance bound for its reach. Closing the path, by taking the same statistics per position, multiplies what enlarging the receptive field is worth by up to an order of magnitude on simulated genomes at every difficulty level tested and on real 1000 Genomes haplotypes. The same path also confounds attribution: ablating a trained network's receptive-field-enlarging blocks severs part of the path, overstating their contribution 8.3-16.1-fold relative to retraining from scratch. The substitution of normalization for receptive field fades as labels switch more often. Where labels run long, neither the receptive-field justification nor the ablation is wrong about its numbers, but both credit the wrong component.

• The paper demonstrates that sequence-pooled normalization creates a sequence-wide information path invisible to convolutional receptive-field calculations, allowing a 9-position model to approach the whole-sequence optimum within 0.009 on a synthetic labeling task.
• Experiments on simulated genomes, 1000 Genomes haplotypes, and Conv-TasNet show that removing sequence-wide pooling can increase the value of receptive-field expansion by 6.3–20.6 times, while causal normalization can still violate locality.
• The paper finds that standard block ablations can overstate the importance of long-range components by up to 16.1 times, recommending a control that removes the sequence axis from normalization statistics before attributing performance to receptive-field growth.

The claim

A convolutional sequence labeler's receptive field is routinely treated as the extent of its usable context: it motivates dilation schedules, bounds streaming horizons, and underwrites locality claims. This paper shows that this treatment fails whenever a normalization layer computes statistics from the current input along the sequence at inference. Such a layer opens a sequence-spanning path from every input position to every output position — a path that passes through no convolution and is invisible to any receptive-field calculation. Where labels come in long runs, that single path supplies almost all of what enlarging the receptive field would buy, and it silently corrupts the standard ablation methodology used to attribute function to reach-enlarging components.

The paper's central quantitative finding is stark: on a synthetic labeling process with exactly computable optima, a network with a 9-position receptive field but sequence-pooled normalization reaches within 0.009 of the whole-sequence optimum, against a near-chance bound for its nominal reach. On simulated genomes and real 1000 Genomes haplotypes, closing the path by taking the same statistics per position multiplies the value of enlarging the receptive field by up to an order of magnitude.

Mechanism: the exposure criterion

For a normalization layer $z_t = \gamma(x_t - \mu_S)/\sigma_S + \beta$ with statistics pooled over an index set $S$, differentiating gives, for every $s \in S$,

$$\frac{\partial z_t}{\partial x_s} = \frac{\gamma}{\sigma_S}\Big[\delta_{ts} - \frac{1 + \hat x_t \hat x_s}{|S|}\Big],$$

verified against automatic differentiation to machine precision. Three consequences follow. First, the off-diagonal terms are generically nonzero whenever $S$ spans the current input's sequence at inference, regardless of what the convolutions reach. Second, each term is individually negligible — about $3\times10^{-5}$ of the diagonal for the paper's GroupNorm configuration — but they do not cancel among themselves; they sum to exactly minus the diagonal term, because adding a constant to all pooled inputs leaves $z_t$ unchanged. A one-position-at-a-time sensitivity analysis therefore dismisses as noise a path whose aggregate magnitude equals the direct one. Third, which layers have the path is readable from the layer definition alone: GroupNorm, InstanceNorm, and layer norm over the sequence meet the criterion; BatchNorm at evaluation (running constants) does not; per-token transformer normalization does not; Conv-TasNet's cumulative layer norm pools only over the past, making it causal but not local.

The paper calls this the exposure criterion: statistics computed from the current input, along the sequence, at inference. It is a property of the layer, checkable before any experiment.

What the path carries

Existence says nothing about worth, so the paper derives a closed form on a Gaussian–Markov process with per-position binary labels from a two-state chain with switch probability $p$, writing $n = pL$ for expected label switches. An oracle given only the sequence's class proportion $\bar\pi$ predicts the majority label everywhere, gaining approximately

$S$0

with nothing fitted; the approximation holds to within 4% for $S$1. The class proportion is precisely what a sequence-pooled statistic encodes to first order — on this process the pooled mean is an affine function of $S$2 plus emission noise, confirmed against raw draws with residual standard deviations within 1–2% of theory. Below $S$3 the approximation must fail, since the approximate variance exceeds what a $S$4-valued quantity can carry; the finite-$S$5 sum remains exact there.

This yields four falsifiable predictions: (P1) reach is worth far less while the path is open; (P2) only pooling over length at inference produces the effect; (P3) the gap decays as $S$6 grows; (P4) the effect is absent where attention already spans the sequence or where inference-time statistics are frozen.

Magnitude across processes and architectures

All comparisons hold architecture, parameter count, data, and seeds fixed, varying only the pooling axes. On the simulated genomic process (a coalescent local-ancestry task ported from a published benchmark, with measured $S$7 matching the benchmark to within 0.0029), a 227-fold receptive-field enlargement buys +0.0320 to +0.0524 accuracy with sequence-pooled statistics versus +0.2011 to +0.3741 without — ratios of 6.3–8.3 at every difficulty level, with no trend across them. At the easiest divergence level, the sequence-pooled model reaches 0.963 from 9 positions; the per-position control needs 129 positions to match it.

On real 1000 Genomes CEU/GIH haplotypes the ratio is 12.1, and the sequence-pooled curve is flat within noise: 0.8536 at reach 9 versus 0.8717 at 2049. In the published Conv-TasNet separator run on the synthetic process, gLN yields a ratio of 20.6, with the model reaching 0.9246 at a receptive field of 7 positions against an exact optimum of 0.9522. Notably, the causal cLN variant still shows a ratio of 6.5: causality is preserved but locality is not, so streaming deployments are not exempt.

Prediction (P2) isolates the axis cleanly: pooling over channels alone matches no normalization (+0.3741 vs +0.3740); pooling over length alone reproduces the full effect (+0.0424, matching GroupNorm's +0.0452); BatchNorm, which pools over length during training but substitutes running constants at evaluation, behaves as though it had no path (+0.3687). Prediction (P4) holds in both absence cases: a transformer differs by only 0.0021–0.0025 between normalizations while sitting within 0.0047 of the exact optimum, and BatchNorm shows no substitution despite training-time pooling.

The dose–response follows $S$8 empirically, with fitted exponents of $S$9 and $s \in S$0 on the two processes — consistent with unity, and clearly separating $s \in S$1 from the $s \in S$2 form the closed form would naively suggest for the ratio (the numerator carries its own $s \in S$3-dependence). The exponent transfers across processes; the constant does not, varying 22% within a process but differing between them.

Two further checks strengthen the account. Per-position networks attain 98–99% of their local information bound across all 35 cells of receptive field and switch density (mean absolute gap 0.004), so the comparison pits a model at its information limit against one with an extra path, not a good model against a bad one. And a U-Net, whose pooled extent shrinks with depth — the architecture Pfrommer et al. name as the practical risk — preserves the effect: autograd-based receptive-field measurement returns the full 4096 positions at every depth when statistics span the sequence, sequence-pooled accuracy is flat to within 0.017 while the pooled count falls 32-fold, and the reach-worth ratio of 16.1 at sparse switches collapses to 2.0 at dense ones, as predicted.

Consequence for attribution

Block ablation removes a trained network's reach-enlarging blocks and re-evaluates; retraining builds the same reduced architecture from scratch. Where each block enters residually ($s \in S$4), removal is exactly the identity map, requiring no surgery — yet the estimate is biased by a large factor. With sequence-pooled normalization, ablating costs 0.3511 accuracy on the genomic process where retraining costs 0.0423: a ratio of 8.3, rising to 16.1 on real haplotypes. Ablation is also 6.7 times noisier across seeds.

The mechanism is that each removed block takes its normalization layers with it, severing part of the sequence-spanning path and charging the blocks for context a fresh network obtains anyway. A controlled comparison separates this from ordinary compensation effects: holding everything else fixed, ablation and retraining differ by only 1.1–1.3× where no statistic spans the sequence (the band adaptation and compensation produce), versus 9.1–9.8× where they do. The factor-of-eight difference between bands is attributable to the path alone.

The error extends beyond magnitude to trend. At a shallower cut removing blocks with dilation ≥ 8, ablation reports retention of 0.527 ± 0.157 — roughly half the model's above-chance skill appears to depend on the removed blocks — while retraining reports 0.925 ± 0.019. Read by ablation, reliance on long-range blocks appears to grow sharply as the task hardens (retention falling 0.79 → 0.28); read by retraining, it barely moves (0.97 → 0.90). An analysis built on the ablation curve would report a difficulty dependence that does not exist, with eightfold wider error bars.

The recommended diagnostic costs one additional training run: repeat the attribution with the length axis removed from the normalization statistics. If the conclusion moves, the original number was measuring the normalization path rather than the component.

Which architectures are exposed

Applied to released source code rather than papers, the criterion partitions thirteen published models across four domains with long-label-run tasks. Only two are exposed: ASFormer (InstanceNorm1d without running statistics in its attention modules) and pyannote.audio's SincNet frontend. Enformer and Basenji — both motivated explicitly by the genomic distance they integrate — are unexposed at inference, contrary to the author's expectation. Exposure follows neither domain nor reported reach, and must be read from code: one diarization model specifies an input layer normalization its released code instantiates but never applies, so citing the equation would misclassify it. Within a single lineage, Mamba normalizes per position and is unexposed, while S4 is unexposed under default layer normalization yet exposed under instance or group options — one configuration string decides whether the path exists. These are readings of source, not measurements of effect size.

An uncomfortable corollary follows: practitioners who move from BatchNorm to GroupNorm or InstanceNorm for reasons unrelated to context acquire the path as a side effect, invalidating a receptive-field justification through a change that looks like housekeeping. The escape via BatchNorm's running statistics is contingent on that implementation detail; leaving BatchNorm in training mode at inference, or test-time adaptation that recomputes statistics, restores exposure.

Limitations and open questions

The paper scopes its claims carefully. The effect is conditional on both halves of the exposure criterion and on labels coming in long runs; the sweep crosses the regime boundary, and at high switch rates the two routes are worth about the same. Several violations can be named in advance: class-balanced sequences give the summary no variance to carry; tasks depending on segment order gain little from a whole-sequence statistic (cLN's cumulative version excepted); and inference-length mismatch with training is untested. The derived closed form covers binary labeling only — multiclass requires rederiving for a $s \in S$5-dimensional proportion vector, and regression has no class proportion at all, though the exposure criterion itself is untouched. The real-data arm uses one chromosome and one population pair, with three replicates sharing whatever is particular to that chromosome. All measurements use three seeds, few enough that ablation-based retention should be read as an order of magnitude. What share of Conv-TasNet's speech-separation performance comes from gLN remains unmeasured, requiring training on a separation corpus against separation metrics. The constant $s \in S$6 in the empirical $s \in S$7 regularity varies across settings in a way the paper does not explain.

Conclusion

For a convolutional sequence labeler whose normalization statistics are pooled along the sequence at inference, the receptive field computed from the convolutions is not a good description of the context the model uses, and where labels come in long runs it is not even the dominant contributor. Both routine practices built on that description — justifying architectures by the span they reach, and attributing behavior to reach-enlarging components by removing them — overstate the receptive field's contribution by roughly an order of magnitude in the measured regime. Whether a given model sits in that regime takes two checks and no experiment: whether the normalization statistics are computed from the current input along the sequence at inference, read from the layer's definition, and whether the labels come in long runs, read from the task.