Trans-CNN: Transformation-Aware CNNs

• Trans-CNNs are neural network architectures that modify standard CNNs to enforce transformation symmetry and integrate transformer attention for enhanced context awareness.
• They include variants such as TI-CNNs for strict invariance, translation-insensitive models using smooth basis expansions, and hybrid convolution-transformer designs.
• These approaches improve robustness, parameter efficiency, and overall accuracy in applications ranging from medical imaging to NLP and geolocalization.

A Trans-CNN refers broadly to a class of network architectures or convolutional neural network (CNN) modifications that enforce or exploit transformation symmetries or that integrate attention-based transformer modules with convolutional backbones. Multiple distinct lines of research have used the "Trans-CNN" term, including works on transformationally identical CNNs with strict symmetry constraints, translation invariant/insensitive CNNs, and hybrid CNN/transformer or CNN-enhanced transformer encoders. Each direction addresses foundational issues in equivariance, invariance, and representation fusion in deep learning.

1. Transformationally Identical CNNs: Symmetry-Invariant Architectures

Transformationally identical CNNs (TI-CNNs, or Trans-CNNs) (Lo et al., 2018) are mathematically constructed to ensure exact equivariance—and with proper readout, invariance—to a chosen group of type-1 transformations (rigid rotation, reflection, integer translation without interpolation). In this setting, the convolutional kernel $K$ is constrained such that $K = T\{K\}$ for all $T$ in the symmetry group of interest. The critical property is commutativity: $$\operatorname{conv}(T\{V_i\},\,K) = T\{\operatorname{conv}(V_i, K)\}$$ for any input $V_i$ and transformation $T$. This property is preserved across sequences of convolution, activation, cross-channel convolution, and optional pooling, provided all kernels in every convolutional layer satisfy the $K = T\{K\}$ constraint.

When the flatten/dense layer (used as the first MLP input) also shares weights among transformation-related positions, the full model becomes strictly invariant: $F(T\{V_i\}) = F(V_i)$ where $F$ represents the CNN-to-MLP pipeline. TI-CNNs enforce these constraints via parameter tying at init, and by averaging or summing gradients among tied parameters during training. For instance, for 2D Dih$_4$ (eight symmetries of the square), only $K = T\{K\}$0 of the $K = T\{K\}$1 kernel parameters are learned; all other elements share values accordingly.

Sanity checks demonstrate that, up to floating-point precision, $K = T\{K\}$2 holds for all $K = T\{K\}$3 in the chosen group, regardless of network depth, stride, pooling, or use of activation functions. Empirical studies (e.g., lung-nodule classification) reveal matched or improved generalization—and parameter efficiency—over standard CNNs, eliminating the need for explicit data augmentation over these transformation groups (Lo et al., 2018).

Symmetry Group	Free Parameters per $K = T\{K\}$4 Kernel	Example TI-CNN Effect
Dih$K = T\{K\}$5	$K = T\{K\}$6	Exact invariance to rotation/reflection
90$K = T\{K\}$7 rot	$K = T\{K\}$8	Invariance to rotation by multiples of 90
Translation TI	$K = T\{K\}$9 (per direction, depending on $T$0)	Invariance to integer translation

TI-CNNs guarantee output consistency for all transformation-related inputs and are particularly suited to domains where rotational, reflectional, or translational invariance are essential.

2. Translation Insensitive and Translation Invariant CNNs

Distinct from explicit finite-group equivariance, a line of research enforces translation invariance or a controlled translation sensitivity through the use of smooth, parameterized basis expansions for CNN filters (Sundaramoorthi et al., 2019). The core insight is that expressing each learned filter as a truncated expansion in Hermite–Gaussian functions (or, equivalently, as coefficient-weighted Gaussian derivatives) ensures that convolution commutes with shifts in the absence of subsampling: $T$1 for any shift $T$2 and input $T$3.

When all such covariant layers are followed by global average pooling, the resulting feature is exactly translation invariant. Introducing subsampling (strided convolution or pooling) necessarily breaks strict invariance, but if the filters are smooth (i.e., constrained via the Hermite–Gaussian basis), the output exhibits a quantifiable Lipschitz translation insensitivity: $T$4 where $T$5 scales with the basis scale $T$6 and weight norms. This enables graceful degradation rather than brittle failure under small shifts—an improvement over vanilla CNNs and CNNs with only fixed anti-aliasing.

Empirical evaluations (e.g., CIFAR-10, CIFAR-10-ZP on ResNet-18/50) confirm that such Trans-CNNs significantly reduce prediction instability to small shifts, with $T$7 (probability of class flips across 8 one-pixel shifts) dropping from $T$8 in standard CNNs to $T$9–$$\operatorname{conv}(T\{V_i\},\,K) = T\{\operatorname{conv}(V_i, K)\}$$0 in Trans-CNNs, with minimal computational or test-accuracy overhead (Sundaramoorthi et al., 2019).

3. Hybrid Convolution–Transformer Architectures

A prominent family of "Trans-CNN" architectures unifies the advantages of convolutions (locality, strong spatial inductive bias) with those of transformers (global context modelling, non-local dependencies) via explicit architectural fusion.

a. Parallel and Serial Hybrid Encoders in Vision

ParaTransCNN (Sun et al., 2024) employs a parallelized U-Net encoder, consisting of a ResNet-style CNN branch (local extraction) and a pyramid Vision Transformer (multi-scale, global context), each operating on the raw image or previous layer features. Outputs at several spatial scales are fused via channel attention modules (squeeze-and-excitation mechanism), then merged in the decoder through standard U-Net skip connections. Empirical studies on AVT (aortic vessel tree), ACDC (cardiac MRI), and Synapse multi-organ CT demonstrate consistent improvements in segmentation accuracy, especially for small or complex anatomical structures—e.g., the pancreas DSC increases from $$\operatorname{conv}(T\{V_i\},\,K) = T\{\operatorname{conv}(V_i, K)\}$$165\% (baseline) to 69.8\% (Sun et al., 2024).

Ablation shows that the pyramid transformer and channel attention fusion are essential for these gains.

b. Transformer-Head Guidance over CNN Features

In TransGCNN (Wang et al., 2022), a lightweight dual-branch transformer head is applied atop VGG-16 CNN feature maps. The global (full-field) and part-level (striped partition) attention blocks are merged and produce spatial importance weights via SE blocks. These weights reweight the original CNN features through a Feature Coupling Unit, yielding a descriptor sensitive to both local and global cues. This approach outperforms all-CNN baselines (SAFA) and direct transformer backbones (EgoTR) in cross-view geolocalization, with higher accuracy, half the parameter count, and nearly double the frame rate compared to purely transformer solutions.

c. 3D Medical Imaging: Local-Global Attention

WSC-Trans (Hua et al., 2022) extends the hybrid paradigm to 3D medical image segmentation. A 3D DenseBlock-based CNN preserves low-level spatial detail, while a bottleneck window/shifted-window/channel self-attention module (adapted from Swin Transformer) enhances the representation of long-range context and small structures. A fused bottleneck representation (concatenation followed by $$\operatorname{conv}(T\{V_i\},\,K) = T\{\operatorname{conv}(V_i, K)\}$$2 convolution) enables accurate segmentation of intricate anatomical structures, with superior Dice, Jaccard, and Hausdorff metrics versus U-Net family and SwinUNETR competitors (Hua et al., 2022).

4. Transforming CNNs into Attention-Enabled Models

Transformed CNNs (T-CNNs) (d'Ascoli et al., 2021) constitute a procedure wherein an existing pre-trained CNN (e.g., ResNet-RS) is "lifted" by substituting its spatial convolutions (specifically 3×3 filters in late residual blocks) with functionally identical multi-head positional self-attention layers. This is accomplished by initializing the attention heads so their receptive fields and weights exactly reproduce the original convolution, using the construction established by Cordonnier et al. (2019):

• For each offset $$\operatorname{conv}(T\{V_i\},\,K) = T\{\operatorname{conv}(V_i, K)\}$$3 in the 3×3 stencil, the $$\operatorname{conv}(T\{V_i\},\,K) = T\{\operatorname{conv}(V_i, K)\}$$4-th attention head uses a positional embedding and gating to focus attention exactly at $$\operatorname{conv}(T\{V_i\},\,K) = T\{\operatorname{conv}(V_i, K)\}$$5, with $$\operatorname{conv}(T\{V_i\},\,K) = T\{\operatorname{conv}(V_i, K)\}$$6 and $$\operatorname{conv}(T\{V_i\},\,K) = T\{\operatorname{conv}(V_i, K)\}$$7 set from the original conv weights.
• The architecture employs Gated Position Self-Attention (GPSA) blocks, with a gating parameter $$\operatorname{conv}(T\{V_i\},\,K) = T\{\operatorname{conv}(V_i, K)\}$$8 controlling the balance between convolution-like and content-based attention.

Fine-tuning the self-attention parameters for only 50 epochs yields up to +2.2 absolute top-1 accuracy on ImageNet-1k (ResNet50-RS: 78.8%→81.0%) and +11 points on ImageNet-C robustness. This workflow is more compute-efficient than training a hybrid model from scratch and reveals intermediate "unlearning to relearn" phase transitions during fine-tuning. Later GPSA layers spontaneously adopt more global receptive fields, contributing to improved robustness to corruptions (d'Ascoli et al., 2021).

5. CNN-Enhanced Transformers for Text: Convolutional QKV in Self-Attention

CNN-Trans-Enc (Benarab et al., 2022) applies a different "trans-CNN" concept to downstream NLP. Instead of refining the backbone Transformer (BERT), the architecture freezes BERT and builds an additional Transformer-Encoder whose Q, K, V projections are generated via small CNNs operating across the stack of BERT [CLS] layer embeddings. Specifically:

• For an input, the 12 BERT [CLS] representations are stacked as a $$\operatorname{conv}(T\{V_i\},\,K) = T\{\operatorname{conv}(V_i, K)\}$$9 matrix.
• Separate 2D CNNs window across groups of 4 adjacent layers and 5 embedding positions to generate Q, K, V maps.
• These convolutional QKV maps enter a 2-layer Transformer-Encoder whose output feeds a classifier.

This configuration leverages BERT-layer diversity, promoting inductive bias, parameter sharing, and dimensionality reduction on Q/K/V representations. Empirically, CNN-Trans-Enc closes $V_i$0 of SOTA on IMDB, $V_i$1 on SST-5, and establishes new SOTA on YELP-5 ($V_i$2) and Amazon-Polarity ($V_i$3) in 5-fold cross-validation—outperforming both simple BERT-last classifiers and vanilla Transformer-Encoders built on the same frozen embeddings (Benarab et al., 2022).

Dataset	BERT-last+Softmax	Vanilla Trans-Enc	CNN-Trans-Enc (Ours)
IMDB	93.21%	94.60%	95.70%
YELP-5	70.10%	75.33%	82.23%
SST-5	49.15%	53.01%	56.03%
Amazon-2	93.20%	96.90%	98.05%
AG-News	92.05%	93.40%	95.49%
DBPedia-14	99.10%	99.43%	99.51%

The CNN-for-QKV approach exploits cross-layer patterns, reduces overparameterization risk, and yields a generalizable attention mechanism, particularly effective when BERT is frozen.

6. Conceptual Unification and Distinctions

The "Trans-CNN" label has been assigned to diverse constructs:

• Structural invariance/equivariance: Enforce symmetry at the kernel and readout levels to guarantee invariance/equivariance to group actions (TI-CNNs) (Lo et al., 2018, Sundaramoorthi et al., 2019).
• Hybrid convolution/transformer fusion: Architectures using both convolutional and transformer-based modules in parallel, serial, or attention-head forms to blend local and non-local information (Sun et al., 2024, Wang et al., 2022, Hua et al., 2022, d'Ascoli et al., 2021).
• Downstream attention enhancement: Use of convolutions to generate inductive-bias-rich Q/K/V representations for Transformer attention over fixed upstream embeddings (Benarab et al., 2022).

The choice among these approaches depends on the target domain, the need for strong equivariance/invariance, data structure, and computational considerations. All approaches reflect a broader research trend: leveraging the interplay between local translation-invariant processing (CNNs) and global or context-sensitive mechanisms (transformers and group-theoretic parameter-tying) to resolve core representational limitations in high-dimensional learning tasks.

7. Impact and Future Directions

Trans-CNN methodologies have enabled practical models with strict or graceful invariance to transformations, high robustness to input perturbations, and improved sample efficiency—especially in domains with strong symmetry priors (medical imaging, geolocalization, document classification). The hybridization with transformers allows selective exploitation of locality/globality, often at cost-accuracy frontiers unattainable by either class alone. Ongoing and future work will likely refine the theoretical basis for smoothness/insensitivity, extend group-theoretic approaches to broader transformation classes, and further integrate parameter-efficient attention mechanisms in both vision and language contexts (Lo et al., 2018, Sundaramoorthi et al., 2019, d'Ascoli et al., 2021, Wang et al., 2022, Benarab et al., 2022, Hua et al., 2022, Sun et al., 2024).