Log-Spaced Continuous Position Bias

• Log-CPB is a method that replaces fixed tabular biases with a neural meta-network using log-transformed coordinates for consistent spatial encoding.
• It enables seamless scaling in vision Transformers by allowing accurate bias extrapolation when transferring to larger windows or higher resolutions.
• Empirical results show Log-CPB maintains high top-1 accuracy and improves metrics on dense prediction tasks, outperforming linear alternatives.

Log-Spaced Continuous Position Bias (Log-CPB) is a method introduced for encoding relative spatial relations between tokens in window-based vision Transformers. Unlike static tabular position biases, Log-CPB employs a small neural meta-network that accepts log-spaced, continuous distance coordinates, enabling seamless and stable extrapolation across window sizes—crucial for transferring pre-trained models to higher resolutions or larger windows without performance degradation. It was proposed in the context of Swin Transformer V2 to address the challenges associated with resolution gaps between pre-training and downstream fine-tuning (Liu et al., 2021).

1. Motivation and Conceptual Framework

Window-based vision Transformer architectures, such as Swin Transformer, require an inductive bias that reflects the spatial proximity of image patches. In the original Swin Transformer, this was realized as a learned bias $B_{ij}$ for each pair of patches at grid distance $(\Delta x, \Delta y)$, stored as a discrete lookup table of size $(2M-1)\times(2M-1)$ for window size $M$. Transitioning to different window sizes at downstream tasks (e.g., during fine-tuning or deployment at higher resolutions) necessitated ad hoc resizing (e.g., via bicubic interpolation), which often led to suboptimal generalization and substantial performance drops at extreme window scales.

Log-CPB replaces this table with a two-layer MLP $\mathcal G$, which takes as input log-spaced, continuous relative coordinates, enabling smooth extrapolation far beyond the window sizes used at pre-training. The log-spacing transformation ensures stability and consistent scaling even across large relative distances, due to the slow growth of the logarithm function.

2. Mathematical Formulation

Let $M$ denote the window width or height ($M \times M$ patches per window). For any two patches within a window, the grid offset is $(\Delta x, \Delta y) \in \{-(M-1), ..., M-1\}^2$.

Log-Spaced Coordinate Transformation:

$$\hat\Delta x = \mathrm{sign}(\Delta x)\,\log\bigl(1 + |\Delta x|\bigr), \qquad \hat\Delta y = \mathrm{sign}(\Delta y)\,\log\bigl(1 + |\Delta y|\bigr)$$

Meta-Network Bias Prediction:

$$B(\Delta x, \Delta y) = \mathcal G(\hat\Delta x, \hat\Delta y)$$

Here, $(\Delta x, \Delta y)$0 is a 2-layer MLP with hidden dimension $(\Delta x, \Delta y)$1 (commonly 128 or 256), using ReLU activation: $(\Delta x, \Delta y)$2

$(\Delta x, \Delta y)$3

with $(\Delta x, \Delta y)$4, $(\Delta x, \Delta y)$5. For multi-head attention, $(\Delta x, \Delta y)$6 either outputs an $(\Delta x, \Delta y)$7-vector per head or separate MLPs per head.

Integration into Attention Mechanism:

$(\Delta x, \Delta y)$8

or in scaled dot-product attention: $(\Delta x, \Delta y)$9 $(2M-1)\times(2M-1)$0 with $(2M-1)\times(2M-1)$1.

3. Integration within Transformer Modules

Construction of the position bias matrix $(2M-1)\times(2M-1)$2 is performed for each forward pass, or pre-computed and cached at test-time. The process involves the following steps:

1. For each patch pair $(2M-1)\times(2M-1)$3, compute $(2M-1)\times(2M-1)$4.
2. Apply the log-spaced transform to obtain $(2M-1)\times(2M-1)$5.
3. Pass the result through the 2-layer MLP $(2M-1)\times(2M-1)$6 to compute $(2M-1)\times(2M-1)$7.
4. Add the resulting $(2M-1)\times(2M-1)$8 to the unnormalized attention logits prior to softmax.

At inference, $(2M-1)\times(2M-1)$9 can be stored as a fixed parameter tensor with shape $M$0, eliminating further MLP computations. No modifications are required elsewhere in the Transformer block.

4. Hyperparameters, Design Choices, and Ablation

The primary hyperparameters and design decisions include:

• MLP Depth: 2 layers
• Hidden Dimension ($M$1): Typically 128 or 256; experiments demonstrate 256 is effective.
• Activation: ReLU
• Coordinate Transform: $M$2 with sign preserved, no additional scaling.
• Number of Heads ($M$3): Equal to attention heads; $M$4 may be shared or distinct across heads.
• Bias Output: Either vector-valued for all heads jointly or per-head small MLPs.

Ablations:

• Replacing log-spaced CPB with linear-spaced CPB (using raw $M$5 as input) or the original table leads to inferior scaling. Log-CPB outperforms linear CPB by 0.3–0.6% top-1 accuracy on ImageNet and improves box/mask mAP and mIoU by up to 3.3 points under large window-size transitions.

5. Empirical Performance and Scaling Results

With models such as Swin-T trained at $M$6 resolution using $M$7 windows:

• The original scheme’s top-1 accuracy on ImageNet-V2 falls from 81.7% at window size $M$8 to 68.7% at $M$9.
• Log-CPB maintains performance across this transition, dropping only from 81.8% to 79.1% without fine-tuning.
• With fine-tuning on larger window sizes, Log-CPB achieves up to 84.2% top-1, a 1.0% gain over linear CPB and up to 1.0% over the original parameterization.

In dense prediction tasks (COCO, ADE20K), Log-CPB yields 0.3–0.9 higher box AP and 1.5–3.3 higher mIoU when doubling window size, supporting its efficacy in extreme regime shifts.

6. Implementation Characteristics and Computational Complexity

Complexity:

• A naive approach recomputing $\mathcal G$0 for all $\mathcal G$1 possible patch-pairs is prohibitive.
• Efficiently, one precomputes a $\mathcal G$2 table of unique $\mathcal G$3, resulting in $\mathcal G$4 MLP calls, then indexes into this table to assemble the $\mathcal G$5 matrix, amortizing computation.
• At inference, the bias tensor for all heads and patch pairs can be statically stored, removing further computational overhead.

Memory:

• The overhead is identical to the original tabular approach, requiring a $\mathcal G$6 bias tensor.
• Additional parameters due to $\mathcal G$7 are negligible (on the order of a few thousand floats).

Stability and Training:

• The log transformation keeps MLP input ranges stable; $\mathcal G$8 grows slowly and remains in roughly $\mathcal G$9 for $M$0.
• Training uses standard weight initialization (Xavier or Kaiming); no altered schedule is needed.
• When sharing an MLP, vector-valued outputs for all heads can be efficiently computed; per-head MLPs can be batched.

A plausible implication is that this approach enables immediate and robust scaling of window-based vision Transformer models across a wide range of image resolutions and spatial granularity, with no need for interpolation or retuning of position biases (Liu et al., 2021).