RSIT: Test for Repeat and Shift Invariance

• RSIT is a benchmark for evaluating model invariance under repeat and shift transformations applied to inputs in CNNs and GNNs.
• It employs rigorous metrics like cosine-similarity deviation and RSIT Gap to quantify stability and performance changes over millions of augmented samples.
• By exposing both architectural weaknesses and strengths, RSIT guides improvements in deep learning models for image analysis and polymer property prediction.

The Repeat and Shift Invariance Test (RSIT) is a rigorous adversarial benchmark developed for quantifying the invariance of learned representations or prediction models with respect to input transformations that should not alter semantic content. RSIT was introduced simultaneously in the analysis of convolutional neural network (CNN) feature invariance and in the domain of polymer property prediction using graph neural networks (GNNs). Across these domains, RSIT formalizes whether a model’s output remains stable under canonical symmetries—namely translations for images and sequence repeat/cyclic shift for polymer strings—thus exposing both weaknesses and strengths in modern architectures (Lee et al., 2020, Wang et al., 27 Jul 2025).

1. Formal Statement of RSIT

RSIT assesses invariance by testing a model’s response to specific classes of input transformations corresponding to known symmetries. In image analysis, this is spatial shift; for polymers, this is repetition and cyclic shift in the sequence representation.

Image Setting

Given an input image $x$ and a CNN feature extractor $f$, shift invariance is defined by the requirement:

$f(x) = f(\mathrm{shift}(x,\,\tau))$

for any pixel offset $\tau$. RSIT quantifies the deviation of $f$ from this ideal over a large and exhaustive set of shifted inputs (Lee et al., 2020).

Polymer Setting

For a polymer string $s$ (specifically, P-SMILES), two operators are defined:

• Repeat by $k$ copies: $R_k(s) = s \Vert s \Vert \dots \Vert s$
• Cyclic shift by $r$ chars: $T_r(s) = s_{[r+1 \dots end]} \Vert s_{[1 \dots r]}$

Invariance requires that for any model $f$0, any $f$1,

$f$2

Thus, RSIT is implemented as an adversarial procedure: given a model and a ground truth label, the input is randomly repeated and cyclically shifted, and the maximal degradation in prediction or loss is measured over multiple such augmentations (Wang et al., 27 Jul 2025).

2. Mathematical Formulation and Metrics

RSIT utilizes quantitative metrics that directly measure the change in model outputs under the defined symmetries.

For CNN Features

• Cosine-similarity deviation:

$f$3

• Euclidean deviation:

$f$4

Perfect invariance yields $f$5 and $f$6 for all $f$7. Summaries such as 2D invariance maps (heatmaps) and anchor-based mean statistics are generated for comprehensive comparison across models and layers (Lee et al., 2020).

For Polymer Property Models

• Prediction Stability: The RSIT protocol measures the worst-case absolute change in prediction or increase in loss after random repeat and shift augmentations. This is operationalized as:
  • For each test sample, generate several (e.g., $f$8) random augmentations via repeat+shift, compute model outputs, and report the maximal observed deviation.
  • The core metric is the RSIT Gap:

  $f$9

where $f(x) = f(\mathrm{shift}(x,\,\tau))$0 is performance on unperturbed data and $f(x) = f(\mathrm{shift}(x,\,\tau))$1 is under adversarial augmentation (Wang et al., 27 Jul 2025).

3. Experimental Protocols

Visual Feature Invariance (CNNs)

• Dataset Generation: N=200 object patches from COCO 2017, resized to “small” (75$f(x) = f(\mathrm{shift}(x,\,\tau))$275 px) and “large” (125$f(x) = f(\mathrm{shift}(x,\,\tau))$3125 px), placed on 224$f(x) = f(\mathrm{shift}(x,\,\tau))$4224 white backgrounds, translated exhaustively over the feasible frame region (yielding millions of images).
• Shifts: Local ($f(x) = f(\mathrm{shift}(x,\,\tau))$5 near anchor) and global ($f(x) = f(\mathrm{shift}(x,\,\tau))$6 across full frame) for fine-to-coarse invariance assessment.
• Evaluation: Extract features from pre-trained CNNs (AlexNet, ResNet-50, MobileNetV2), compute pairwise cosine similarities or Euclidean norms, aggregate into invariance maps, and generate summary tables (Lee et al., 2020).

Polymer Sequence Models

• Graph Construction: Compare standard “star” strategies (“star keep”, “star remove”, “star substitution”) versus “star linking”—which creates a cyclic monomer graph for true invariance.
• Backbone Model: GIN3-512 (3-layer Graph Isomorphism Network).
• Dataset/Task Coverage: Eight DFT-predicted polymer property regression tasks (electronic, thermal, optical), datasets of 370–3,380 samples each.
• RSIT Execution: On test fold samples, Algorithm 2 runs randomized repeat+shift augmentations and records worst-case loss/prediction. Clean vs. RSIT $f(x) = f(\mathrm{shift}(x,\,\tau))$7 and gap statistics are reported (Wang et al., 27 Jul 2025).

4. Theoretical Results and Empirical Findings

CNN Feature Extractors

• Local vs. Global Invariance: No standard CNN exhibited global shift invariance; cosine similarity drops at image edges for all tested architectures. Local invariance is improved by anti-aliased models (e.g., Bin-5 filter), but this has marginal effect on global behavior.
• Bias and Artifacts: Stronger invariance to horizontal than vertical shifts, not explained by data augmentation practices alone. Local grid-like periodicity in invariance linked to pooling layer configuration.
• Layer-wise Structure: fc7 in AlexNet is more invariant than fc6 or fc8, but shift information is still present and linearly decodable even at deep fully connected layers.
• Feature Arithmetic: Lower fully connected layers encode spatial information allowing semantic manipulation via vector arithmetic, demonstrating non-invariance (Lee et al., 2020).

Polymer Properties and GNNs

• Star Linking: The “star linking” strategy, proven via symmetry arguments and mean pooling, enforces exact invariance to repeat and shift of P-SMILES (Thm. 1 & 2). Empirically, RSIT Gap is $f(x) = f(\mathrm{shift}(x,\,\tau))$8, indicating zero observed degradation. Other “star” strategies show significant performance loss under RSIT (average drop in $f(x) = f(\mathrm{shift}(x,\,\tau))$9 ranges 0.107–0.233).
• Limitations: Invariance proofs for localized attention require minimal monomer size; otherwise, monomer tiling is required. The expressive power of GIN and local attention is bounded by the Weisfeiler-Lehman test and fails to distinguish nontrivial “twin” polymers differing only in ring structure; backbone embedding is added to address this but is not part of the strict invariance mechanism (Wang et al., 27 Jul 2025).

Strategy	R² (clean)	R² (RSIT)	RSIT Gap
star keep	0.866–0.956	0.745–0.928	0.233
star remove	0.857–0.939	0.765–0.864	0.199
star substitution	0.868–0.963	0.796–0.922	0.107
star linking	0.870–0.966	0.870–0.966	0.000

5. Implementation Details and Reproducibility

• Frameworks: For CNNs, PyTorch and torchvision for feature extraction, NumPy for similarity computations, matplotlib/seaborn for visualization. Data and code are available at https://github.com/jakehlee/visualize-invariance and https://jakehlee.github.io/visualize-invariance (Lee et al., 2020).
• Sample procedure:

$\tau$2

• Polymer GNNs: Models are implemented with 3-layer GIN, with graph construction strategies detailed above. Five-fold cross-validation under standard training regimes is employed. RSIT augmentations are performed for each test sample, with explicit adversarial tracking of the maximal loss change (Wang et al., 27 Jul 2025).
• Computational scale: Full CNN invariance analysis required extracting features for $\tau$0M images, tens of GPU-hours, and $\tau$1 GB temporary storage if images are not generated on-the-fly (Lee et al., 2020).

6. Practical Implications and Limitations

RSIT exposes systematic weaknesses in invariant representation learning. Existing CNN feature extractors are only locally invariant; anti-aliasing mitigates small shifts but fails for large translations, necessitating alternate strategies (spatial pooling, augmentation, or modules such as spatial transformers) for truly shift-insensitive global descriptors (Lee et al., 2020).

For polymer property prediction, failure of GNNs to respect repeat and shift invariance directly impacts prediction reliability. The star-linking construction, both theoretically and empirically, provides a remedy, but this invariance does not cover edge cases involving monomers with minimal size or cyclic rings, which remain out of reach for conventional message-passing and graph attention mechanisms. Additional embeddings or higher expressive models are necessary to handle such cases (Wang et al., 27 Jul 2025).

RSIT serves both as a diagnosis tool for architecture brittleness and as a standard for benchmarking the invariance of any new method targeting data domains with inherent symmetry. Its use is recommended wherever input symmetries are known a priori, such as global image descriptors or periodic sequences.