T2C-adapters: Transformer-to-CNN Enhancement

• The paper demonstrates that T2C-adapters effectively couple transformer-derived global context with CNN local features, enhancing performance on tasks requiring both long-range and fine-grained information.
• The design encompasses various archetypes such as cross-attention fusion, projection-style merging, and pseudo-token augmentation, each tailored for specific domains like speech, segmentation, and counting.
• Empirical results show that uni-directional adapter setups and selective late-stage insertion achieve significant performance gains and parameter efficiency compared to full fine-tuning.

Searching arXiv for the cited T2C-adapter papers and closely related work. Transformer-to-CNN Enhancement Adapters (T2C-adapters) denote a family of modules that couple transformer-derived contextual representations with convolutional feature pathways. In the available literature, the label does not refer to a single canonical operator. Instead, it encompasses cross-attentional projection from transformer tokens into CNN feature maps, token-to-grid reconstruction followed by gated residual merging, appended pseudo tokens at the CNN-to-transformer boundary, and CNN-side residual adapters that complement transformer adaptation. This suggests that T2C-adapters are best understood as a design family for combining global dependency modeling with convolutional locality, rather than as a uniquely standardized block (Pandey et al., 2021, d'Ascoli et al., 2021, Chen et al., 2022, Shi et al., 9 Jul 2025, Inoue et al., 2024).

1. Terminological scope and lineage

The literature uses the T2C label in at least two closely related ways. In the strict sense, a T2C-adapter transfers information from a transformer stream into a CNN stream, as in the layer-wise Transformer-to-CNN enhancement adapters in QUANet and the projection-style fusion modules used for cell instance segmentation. In a broader adapter interpretation, the same label is applied to hybrid modules that preserve or refine CNN behavior through transformer-style mechanisms, including the GPSA-based recasting of convolutions in Transformed CNNs, the CNN feature-extractor adapters in CHAPTER, and token-to-grid reconstruction patterns derived from CTA-Net and ConvFormer. The available corpus therefore suggests semantic heterogeneity rather than a settled nomenclature (Shi et al., 9 Jul 2025, Pandey et al., 2021, d'Ascoli et al., 2021, Chen et al., 2022, Meng et al., 2024, Lin et al., 2023).

Work	Adapter locus	Core mechanism
"CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Models" (Chen et al., 2022)	Speech CNN feature extractor	Residual Conv1d adapters
"Transformed CNNs: recasting pre-trained convolutional layers with self-attention" (d'Ascoli et al., 2021)	Selected CNN conv layers	GPSA initialized as conv
"Transformer Assisted Convolutional Network for Cell Instance Segmentation" (Pandey et al., 2021)	Multi-stage CNN backbone	Cross-attentive token projection
"Text-promptable Object Counting via Quantity Awareness Enhancement" (Shi et al., 9 Jul 2025)	Transformer/CNN decoder streams	CA + CE, Transformer $\rightarrow$ CNN
"ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks" (Inoue et al., 2024)	CNN-to-transformer boundary	Appended pseudo tokens

A second historical regularity is that T2C-adapters emerge where task performance depends on both long-range context and local structure. In speech, the motivation is adaptation of acoustic front-ends for speaker- and emotion-centric tasks. In counting and segmentation, the motivation is injecting global semantics into dense local prediction. In image classification, the motivation is to preserve pretrained convolutional behavior at initialization while allowing self-attention to improve robustness after brief fine-tuning (Chen et al., 2022, Shi et al., 9 Jul 2025, d'Ascoli et al., 2021).

2. Architectural archetypes

One common archetype is explicit transformer-to-CNN transfer by cross-attention. In QUANet, each T2C-adapter has a Cross-Attention block and a Channel-Excitation block. Keys and values come from the Transformer stream, queries come from the CNN stream, and the adapter output $F_{T2C} = F_{CE} + F_{CA}$ is added to the CNN feature. The transfer is uni-directional only, because the reverse direction is reported to “break the contextual cues originally stored in the Transformer stream.” The same section of the model then fuses density maps at image level as $D^{*} = w_1 D^{*}_{CNN} + w_2 D^{*}_{Trans}$ (Shi et al., 9 Jul 2025).

A second archetype is projection-style fusion at multiple CNN stages. In the cell instance segmentation model, a transformer-assisted feature extractor replaces the vanilla convolutional feature extractor by fusing each backbone stage with same-depth transformer tokens. After a $1\times1$ convolution aligns the CNN feature map to the transformer embedding dimension, the adapter computes a single-head cross-attention-like projection and uses residual fusion before FPN, RPN, ROI heads, and the mask head. This design keeps the downstream region-proposal pipeline unchanged while altering the feature hierarchy consumed by it (Pandey et al., 2021).

A third archetype is reconstruction-and-gating. The CTA-Net-derived T2C pattern projects transformer tokens $T \in \mathbb{R}^{N \times d}$ back to spatial feature maps $F_{t2c} \in \mathbb{R}^{C \times H \times W}$, applies depthwise and pointwise convolutions for locality injection, and merges the result with the CNN feature map through an SE-style gate. In parallel, LMF-MHSA supplies a multi-scale attention mechanism with reduced parameterization. The resulting adapter is explicitly designed as a reusable block for injecting transformer global context into CNN stages (Meng et al., 2024).

A fourth archetype operates at the CNN-to-transformer boundary rather than between parallel streams. In ELP-adapters, the P-adapter appends learnable pseudo features $P \in \mathbb{R}^{m \times d}$ to the CNN encoder output $X_0 \in \mathbb{R}^{n \times d}$, for example through the suffix form $g_{\mathrm{suf}}(X_0) = [X_0; P]$. The augmented sequence then traverses frozen transformer layers, after which the pseudo tokens are removed. The paper identifies this as the key T2C component because it enhances the CNN-derived sequence through transformer self-attention with only $m \cdot d$ additional parameters (Inoue et al., 2024).

A fifth archetype reverses the usual direction of emphasis by adapting the convolutional front-end itself. CHAPTER inserts residual CNN adapters into selected convolutional blocks of the HuBERT-style feature extractor while also retaining transformer-layer Houlsby adapters. In its minimal form, the CNN block is “Conv1d $F_{T2C} = F_{CE} + F_{CA}$0 LayerNorm $F_{T2C} = F_{CE} + F_{CA}$1 GELU,” followed by residual addition. This design operationalizes the view that efficient adaptation should address both the transformer stack and the acoustic front-end (Chen et al., 2022).

A sixth, more indirect, pattern is distillation-mediated enhancement. TCC does not introduce an explicit plug-in T2C block; instead, it uses class-aware feature consistency distillation and consistency-aware cross distillation between a ViT student and a CNN student. The adapter interpretation derived from TCC uses pseudo prototypes, class-aware feature maps, and bidirectional KL regularization as a route for transferring global ViT structure into CNN features under semi-supervised segmentation (Zheng et al., 2022).

3. Representative mathematical formulations

The mathematical forms reported for T2C-adapters are heterogeneous. In CHAPTER, the residual update is written as

$F_{T2C} = F_{CE} + F_{CA}$2

with a canonical bottleneck form

$F_{T2C} = F_{CE} + F_{CA}$3

and a minimal adapter

$F_{T2C} = F_{CE} + F_{CA}$4

Here $F_{T2C} = F_{CE} + F_{CA}$5, the convolution is temporal Conv1d with stride $F_{T2C} = F_{CE} + F_{CA}$6, and $F_{T2C} = F_{CE} + F_{CA}$7 reconstructs the original channel dimension by replication or concatenation before residual addition (Chen et al., 2022).

In projection-based fusion for cell instance segmentation, the adapter is written as

$F_{T2C} = F_{CE} + F_{CA}$8

where $F_{T2C} = F_{CE} + F_{CA}$9 is the flattened CNN feature after channel projection, $D^{*} = w_1 D^{*}_{CNN} + w_2 D^{*}_{Trans}$0 is the transformer token set at the same depth, and the softmax is taken along the token dimension. The residual term $D^{*} = w_1 D^{*}_{CNN} + w_2 D^{*}_{Trans}$1 projects transformer information back into the CNN spatial lattice (Pandey et al., 2021).

In Transformed CNNs, the core equivalence is established through Gated Positional Self-Attention,

$D^{*} = w_1 D^{*}_{CNN} + w_2 D^{*}_{Trans}$2

with an initialization that sets $D^{*} = w_1 D^{*}_{CNN} + w_2 D^{*}_{Trans}$3, uses $D^{*} = w_1 D^{*}_{CNN} + w_2 D^{*}_{Trans}$4 heads for a $D^{*} = w_1 D^{*}_{CNN} + w_2 D^{*}_{Trans}$5 convolution, and assigns each head to a kernel offset $D^{*} = w_1 D^{*}_{CNN} + w_2 D^{*}_{Trans}$6. With $D^{*} = w_1 D^{*}_{CNN} + w_2 D^{*}_{Trans}$7 and $D^{*} = w_1 D^{*}_{CNN} + w_2 D^{*}_{Trans}$8 loaded from the corresponding convolutional kernel slice, the multi-head sum reconstructs the original convolution exactly at $D^{*} = w_1 D^{*}_{CNN} + w_2 D^{*}_{Trans}$9 (d'Ascoli et al., 2021).

QUANet formalizes T2C coupling through additive feature fusion: $1\times1$0

$1\times1$1

followed by residual enhancement of the CNN stream. The model is optimized jointly with

$1\times1$2

$1\times1$3

where $1\times1$4 (Shi et al., 9 Jul 2025).

ELP’s P-adapter is algebraically simpler but conceptually important: $1\times1$5 The pseudo tokens co-attend with CNN-derived features in all transformer layers, after which the first or last $1\times1$6 tokens are removed. This is a T2C formulation by sequence augmentation rather than by explicit cross-attention or spatial gating (Inoue et al., 2024).

4. Empirical behavior across domains

In speech, CHAPTER reports that fewer than $1\times1$7 of upstream parameters are trained per task, with approximately $1\times1$8M trainable parameters versus $1\times1$9M in HuBERT, corresponding to approximately $T \in \mathbb{R}^{N \times d}$0. Against a Houlsby-only baseline, speaker identification accuracy improves from $T \in \mathbb{R}^{N \times d}$1 to $T \in \mathbb{R}^{N \times d}$2, emotion recognition on fold 1 improves from $T \in \mathbb{R}^{N \times d}$3 to $T \in \mathbb{R}^{N \times d}$4, and speaker diarization DER improves from $T \in \mathbb{R}^{N \times d}$5 to $T \in \mathbb{R}^{N \times d}$6. The same study reports that adding adapters only to the top 3 CNN layers reaches $T \in \mathbb{R}^{N \times d}$7 ER accuracy with approximately $T \in \mathbb{R}^{N \times d}$8M parameters, and that gains are strongest for speaker-related tasks and emotion, while ASR gains are more modest (Chen et al., 2022).

ELP reports a complementary speech result at the CNN-to-transformer boundary. With a WavLM backbone, ELP-adapters are comparable to or better than full fine-tuning on all evaluated tasks while requiring $T \in \mathbb{R}^{N \times d}$9 fewer learnable parameters. Total learnable parameters are $F_{t2c} \in \mathbb{R}^{C \times H \times W}$0M for ELP versus $F_{t2c} \in \mathbb{R}^{C \times H \times W}$1M for full fine-tuning, and the P-adapter alone contributes $F_{t2c} \in \mathbb{R}^{C \times H \times W}$2M parameters. The reported ablations indicate that the P-adapter yields consistent small gains across tasks at negligible parameter cost, whereas E-adapters are most critical for ASR and L-adapters dominate ASV and SER (Inoue et al., 2024).

In dense counting, QUANet isolates the empirical contribution of T2C-adapters through decoder ablations. Removing T2C-adapters degrades validation MAE from $F_{t2c} \in \mathbb{R}^{C \times H \times W}$3 to $F_{t2c} \in \mathbb{R}^{C \times H \times W}$4 and test MAE from $F_{t2c} \in \mathbb{R}^{C \times H \times W}$5 to $F_{t2c} \in \mathbb{R}^{C \times H \times W}$6. Removing CE yields test MAE $F_{t2c} \in \mathbb{R}^{C \times H \times W}$7, removing CA yields $F_{t2c} \in \mathbb{R}^{C \times H \times W}$8, CNN$F_{t2c} \in \mathbb{R}^{C \times H \times W}$9Transformer transfer yields $P \in \mathbb{R}^{m \times d}$0, and bidirectional transfer yields $P \in \mathbb{R}^{m \times d}$1, all inferior to the default uni-directional Transformer$P \in \mathbb{R}^{m \times d}$2CNN design. The full model attains FSC-147 test MAE $P \in \mathbb{R}^{m \times d}$3 and RMSE $P \in \mathbb{R}^{m \times d}$4, while the paper further reports CARPK MAE $P \in \mathbb{R}^{m \times d}$5, PUCPR+ MAE $P \in \mathbb{R}^{m \times d}$6, ShanghaiTech SHA RMSE $P \in \mathbb{R}^{m \times d}$7, and SHB MAE $P \in \mathbb{R}^{m \times d}$8. GAME ablations show that the Transformer-only decoder is stronger at coarse grids, the CNN-only decoder is stronger at fine grids, and the combined DAC-decoder performs best across $P \in \mathbb{R}^{m \times d}$9–$X_0 \in \mathbb{R}^{n \times d}$0 (Shi et al., 9 Jul 2025).

In image classification and robustness, Transformed CNNs show that converting pretrained convolutions into GPSA layers can improve both accuracy and corruption robustness after short fine-tuning. For ResNet50-RS, top-1 accuracy on ImageNet-1k rises from $X_0 \in \mathbb{R}^{n \times d}$1 to $X_0 \in \mathbb{R}^{n \times d}$2, and ImageNet-C top-1 rises from $X_0 \in \mathbb{R}^{n \times d}$3 to $X_0 \in \mathbb{R}^{n \times d}$4. The paper reports analogous gains for ResNet101-RS and demonstrates that $X_0 \in \mathbb{R}^{n \times d}$5 epochs of fine-tuning suffice to obtain these improvements (d'Ascoli et al., 2021).

In segmentation and instance segmentation, several results support the same architectural intuition. The transformer-assisted cell instance segmentation model raises mIoU from $X_0 \in \mathbb{R}^{n \times d}$6 to $X_0 \in \mathbb{R}^{n \times d}$7 for DetectoRS + EfficientNet-b5 and from $X_0 \in \mathbb{R}^{n \times d}$8 to $X_0 \in \mathbb{R}^{n \times d}$9 for Cascade Mask R-CNN X152 + EfficientNet-b5, surpassing the heavier default-backbone baselines. ConvFormer, used as a plug-and-play module for segmentation frameworks, improves SETR on ACDC from $g_{\mathrm{suf}}(X_0) = [X_0; P]$0 to $g_{\mathrm{suf}}(X_0) = [X_0; P]$1 Dice Avg and improves TransFuse on ISIC from $g_{\mathrm{suf}}(X_0) = [X_0; P]$2 to $g_{\mathrm{suf}}(X_0) = [X_0; P]$3 Dice. CTA-Net, although presented as a hybrid aggregation network rather than as a named T2C module, reports Top-1 Acc $g_{\mathrm{suf}}(X_0) = [X_0; P]$4, Params $g_{\mathrm{suf}}(X_0) = [X_0; P]$5M, and FLOPs $g_{\mathrm{suf}}(X_0) = [X_0; P]$6B on small-scale datasets, and its RRCV/LMF-MHSA design is explicitly formulated as a T2C-adapter pattern in the supplied technical guide (Pandey et al., 2021, Lin et al., 2023, Meng et al., 2024).

5. Optimization, placement, and efficiency

A recurrent design choice is selective placement. Transformed CNNs recommend converting later stages, because early stages have too many tokens and induce the attention bottleneck at large spatial resolutions. In CHAPTER, late feature-extractor layers are especially important for emotion cues, and top-$g_{\mathrm{suf}}(X_0) = [X_0; P]$7 placement preserves most of the gains with fewer parameters. In QUANet, adapters are inserted layer-wise between corresponding Transformer and CNN decoder stages, operating at patch level rather than only at the final fusion stage. Across these settings, the literature repeatedly favors late or stage-aligned insertion over indiscriminate early-layer attachment (d'Ascoli et al., 2021, Chen et al., 2022, Shi et al., 9 Jul 2025).

Training protocols likewise reflect parameter-efficiency. CHAPTER freezes all original backbone weights in both the feature extractor CNN and the transformer layers, and trains only CNN adapters, Houlsby adapters, and the downstream head. QUANet fine-tunes the last six layers of a DINOv2 ViT-B/14 vision encoder while keeping the BERT-base text encoder frozen, using AdamW with learning rate $g_{\mathrm{suf}}(X_0) = [X_0; P]$8, batch size $g_{\mathrm{suf}}(X_0) = [X_0; P]$9, image size $m \cdot d$0, and $m \cdot d$1 epochs. Transformed CNNs use AdamW, batch size $m \cdot d$2, warm-up to $m \cdot d$3, cosine decay, and a larger learning rate for $m \cdot d$4. ELP freezes the backbone CNN, MHSA, and FFN weights, and trains the E-, L-, and P-adapters, the downstream head, and LayerNorm parameters (Chen et al., 2022, Shi et al., 9 Jul 2025, d'Ascoli et al., 2021, Inoue et al., 2024).

Efficiency gains are substantial but not uniform. CHAPTER reports modest training overhead, negligible latency increase relative to the backbone, and a sub-$m \cdot d$5 FLOPs increment, because the backbone remains frozen and only adapter weights are stored per task. ELP obtains approximately $m \cdot d$6 parameter savings relative to full fine-tuning, with the P-adapter contributing only $m \cdot d$7 parameters when $m \cdot d$8 and $m \cdot d$9. By contrast, attention-heavy conversions can be compute-expensive: Transformed CNNs increase FLOPs for ResNet50-RS from $F_{T2C} = F_{CE} + F_{CA}$00G to $F_{T2C} = F_{CE} + F_{CA}$01G, which is why late-stage conversion and local initialization are emphasized. QUANet reports overall model size $F_{T2C} = F_{CE} + F_{CA}$02 MB and $F_{T2C} = F_{CE} + F_{CA}$03 fps, noting that the DAC-decoder accounts for less than $F_{T2C} = F_{CE} + F_{CA}$04 of total parameters (Chen et al., 2022, Inoue et al., 2024, d'Ascoli et al., 2021, Shi et al., 9 Jul 2025).

6. Limitations, misconceptions, and research directions

A common misconception is that T2C-adapters always denote direct bidirectional exchange between transformer and CNN streams. QUANet provides explicit counterevidence: uni-directional Transformer$F_{T2C} = F_{CE} + F_{CA}$05CNN transfer outperforms CNN$F_{T2C} = F_{CE} + F_{CA}$06Transformer and bidirectional designs, and the reverse direction is reported to “break the contextual cues originally stored in the Transformer stream.” A second misconception is that gains arise merely from adding parameters. CHAPTER directly compares against larger transformer-only adapters and reports that CHAPTER, at approximately $F_{T2C} = F_{CE} + F_{CA}$07M parameters, outperforms Houlsby128 at approximately $F_{T2C} = F_{CE} + F_{CA}$08M on SID, SD, and ER, attributing the gain to feature-extractor adaptation rather than parameter count alone (Shi et al., 9 Jul 2025, Chen et al., 2022).

The literature also indicates clear task sensitivity. For tasks dominated by linguistic context, such as ASR, CNN-side adaptation yields smaller improvements than for speaker, emotion, or dense local prediction tasks. In counting, the adapters help but do not fully resolve heavy stacking, occlusion, or ambiguous part-whole semantics. In semi-supervised segmentation, TCC-derived designs depend on pseudo-label quality and unlabeled data volume, and the original framework does not introduce explicit projection modules. In ConvFormer, larger receptive-field tendency does not necessarily improve performance; Dice on ACDC peaks around $F_{T2C} = F_{CE} + F_{CA}$09–$F_{T2C} = F_{CE} + F_{CA}$10, which indicates that global context is not uniformly beneficial for every pixel (Chen et al., 2022, Shi et al., 9 Jul 2025, Zheng et al., 2022, Lin et al., 2023).

Open directions follow directly from these limitations. The supplied material repeatedly points to multi-scale adapters, careful late-stage placement, and lightweight gating as practical ways to preserve efficiency. It also suggests that future T2C designs will need more precise alignment strategies for tokens and feature maps, stronger but controlled cross-stream supervision, and task-dependent choices about whether the adapter should act as cross-attention, token augmentation, convolutional refinement, or distillation. The strongest results to date indicate that T2C-adapters are most effective when they respect the division of labor already visible in the underlying architectures: transformers contribute global semantics or long-range dependence, while CNNs retain responsibility for local geometry, spatial detail, or acoustic front-end structure (Meng et al., 2024, Inoue et al., 2024, Pandey et al., 2021).