---
title: 'Conv-Former: A Convolutional Transformer Hybrid'
url: https://www.emergentmind.com/topics/conv-former
type: topic
---

# Conv-Former: A Convolutional Transformer Hybrid

Searching arXiv for the named paper and closely related ConvFormer variants to ground the article in current arXiv records.
arXiv search query: 2208.08900 Conviformers ConvFormer
“Conv-Former” denotes a heterogeneous line of arXiv architectures that combine convolutional operators with Transformer or Conformer-style blocks rather than a single standardized model. Across the papers collected under this label, convolution is used as a token mixer, as a generator of $Q/K/V$ projections, as a feed-forward refinement operator, as a dynamic gate on convolutional weights, or as a replacement for selected attention layers. The resulting models span fine-grained plant classification, point-cloud segmentation and scene flow, sequential recommendation, single-image super-resolution, monocular 3D human pose estimation, medical image segmentation, speech enhancement, and automatic speech recognition [2208.08900], [2208.02879], [2308.02925], [2401.05633], [2304.02147], [2309.05674], [2211.08564], [2106.15813], [2304.00171], [2407.03718]. This suggests that “Conv-Former” is best understood as a recurring hybrid design motif: preserve or reintroduce convolutional inductive bias while retaining some of the long-range modeling, residual structure, and modularity associated with Transformer-family networks.

## 1. Scope and nomenclature

The term appears in several closely related spellings—“Conviformer,” “ConvFormer,” “PointConvFormer,” “DF-Conformer,” and “Multi-Convformer”—and these names refer to architectures that are similar in intent but not identical in mechanism. In the vision literature, “Conviformers: Convolutionally guided Vision Transformer” introduces a convolutional transformer for fine-grained categorization of plants from herbarium sheets and pairs it with the PreSizer preprocessing method [2208.08900]. In recommendation, “ConvFormer: Revisiting Transformer for Sequential User Modeling” replaces self-attention with a Light Temporal Convolutional Network layer [2308.02925]. In super-resolution, “Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach” uses a large-kernel convolutional feature mixer and an edge-preserving feed-forward network [2401.05633]. In 3D human pose estimation, “ConvFormer: Parameter Reduction in Transformer Models for 3D Human Pose Estimation by Leveraging Dynamic Multi-Headed Convolutional Attention” retains scaled dot-product attention but makes the projections convolutional and multi-scale [2304.02147].

| Variant | Domain | Characteristic mechanism |
|---|---|---|
| “Conviformers: Convolutionally guided Vision Transformer” [2208.08900] | Fine-grained plant classification | Higher-resolution handling and PreSizer |
| “PointConvFormer: Revenge of the Point-based Convolution” [2208.02879] | Point-cloud segmentation and scene flow | Point convolution re-weighted by feature-difference attention |
| “ConvFormer: Revisiting Transformer for Sequential User Modeling” [2308.02925] | Next-item prediction | Depth-wise convolution + channel-wise convolution with FFT acceleration |
| “Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach” [2401.05633] | Lightweight SISR | Large-kernel mixer + edge-preserving FFN |
| “ConvFormer: Parameter Reduction in Transformer Models for 3D Human Pose Estimation...” [2304.02147] | Monocular 3D pose estimation | Dynamic multi-headed convolutional self-attention |
| “ConvFormer: Plug-and-Play CNN-Style Transformers for Improving Medical Image Segmentation” [2309.05674] | Medical segmentation | Pooling + CNN-style self-attention + convolutional FFN |
| “ConvFormer: Combining CNN and Transformer for Medical Image Segmentation” [2211.08564] | Medical segmentation | Hierarchical hybrid stem + Enhanced DeTrans |
| “DF-Conformer” [2106.15813], “Practical Conformer” [2304.00171], “Multi-Convformer” [2407.03718] | Speech enhancement and ASR | Linear attention, conv-only lower blocks, or multiple convolution kernels |

The nomenclature is therefore broad rather than canonical. A plausible implication is that the name signals a design philosophy—convolution-guided Transformer-family modeling—more than a fixed blueprint.

## 2. Recurrent architectural motifs

A persistent motif is the replacement or restructuring of the Transformer token mixer while preserving the residual, normalization, and feed-forward scaffolding. In sequential user modeling, ConvFormer removes the $Q/K/V$ projections and Softmax attention entirely and replaces the standard block with a depth-wise convolution (DWC) plus channel-wise convolution (CWC) sublayer, called the Light Temporal Convolutional Network layer [2308.02925]. In lightweight super-resolution, the ConvFormer layer similarly replaces multi-head self-attention with a large-kernel depth-wise convolutional gate, while the FFN is reworked into an edge-preserving module [2401.05633]. These designs make the “mixer + MLP” decomposition explicit without insisting on attention as the mixer.

A second motif is to keep attention but make it convolutional in how queries, keys, or values are formed. The 3D pose ConvFormer generates $Q$, $K$, and $V$ via small $1$D convolutions along the sequence dimension and aggregates multiple kernel sizes with learned nonnegative weights [2304.02147]. The medical segmentation ConvFormer with CNN-style self-attention projects $Q$, $K$, and $V$ by $3\times3$ convolutions over $2$D feature maps, constructs self-attention matrices as convolution kernels with adaptive sizes, and follows them with a convolutional feed-forward network [2309.05674]. The hierarchical medical segmentation ConvFormer based on Enhanced DeTrans inserts a depth-wise convolution into the Deformable Transformer feed-forward module and combines convolutional and deformable-attention branches in a residual-shaped hybrid stem [2211.08564].

A third motif reverses the direction of influence: attention modulates convolution rather than replacing it. PointConvFormer preserves the point-convolution operator and uses an attention score based on feature difference between points in the neighborhood to modify the convolutional weights at each point:
$$
x'_p = \sum_{p_i\in N(p)} w(p_i-p)^T \,\psi\bigl([x(p_i)-x(p),\,p_i-p]\bigr)\,x(p_i).
$$
In the paper’s interpretation, this preserves the invariances from point convolution while using attention to select relevant points in the neighborhood for convolution [2208.02879].

A fourth motif is explicit complexity control. ConvFormer for recommendation accelerates full-sequence DWC using the convolution theorem,
$$
\mathrm{DWC}(R)=\mathrm{IFFT}\bigl(\mathrm{FFT}(R)\odot \mathrm{FFT}(C_{\mathrm{padded}})\bigr),
$$
reducing the DWC from $O(D\cdot L^2)$ to $O(D\cdot L\log L)$ [2308.02925]. DF-Conformer replaces quadratic self-attention by FAVOR+ linear attention and pairs it with stacked $1$-D dilated depthwise convolution layers [2106.15813]. The optimized ASR Conv-Former replaces lower Conformer blocks with convolution-only blocks and uses an RNNAttention-Performer to reduce latency [2304.00171].

## 3. Vision, super-resolution, and medical segmentation

In fine-grained plant classification, “Conviformers” begins from the observation that fine-grained tasks require discovery of subtle differences between highly similar sub-classes and that such distinctions are often lost when images are downscaled to save memory and computational cost associated with vision transformers [2208.08900]. The model is presented as a convolutional transformer architecture that, unlike the popular Vision Transformer (ConViT), can handle higher resolution images without exploding memory and computational cost. The same work introduces PreSizer, described as a novel, improved pre-processing technique to resize images better while preserving their original aspect ratios, which proved essential for classifying natural plants, and reports SoTA on Herbarium 202x and iNaturalist 2019 [2208.08900].

In lightweight single-image super-resolution, ConvFormer is instantiated as the core layer of the CFSR network. The large-kernel mixer computes
$$
V=\mathrm{Conv}_{1\times1}(F),\qquad U=\mathrm{DwConv}_{k\times k}(F),
$$
followed by
$$
F_{\rm gate}=\mathrm{Conv}_{1\times1}(U),\qquad F_{\rm out}=V\odot F_{\rm gate},
$$
with $k=9$ chosen as a good trade-off between receptive field and cost [2401.05633]. The same paper contrasts global self-attention, local window self-attention, and the large-kernel mixer by complexity, and introduces the edge-preserving feed-forward network, whose depth-wise branch combines a learnable $3\times3$ convolution with fixed Sobel and Laplacian filters through learnable Softmax gates. With $N=2$ residual ConvFormer blocks, channel width $C=48$, and $6$ ConvFormer layers per block, the network has about $291$ K parameters and $62.6$ G FLOPs for $\times2$ SR. On Urban100, CFSR achieves PSNR/SSIM $=32.28/0.9300$ versus $31.89/0.9257$ of ShuffleMixer, a gain of $+0.39$ dB, while reducing parameters by $26\%$ and FLOPs by $31\%$ [2401.05633].

Medical image segmentation contains two distinct ConvFormer lines. The plug-and-play CNN-style Transformer variant operates directly on $2$D feature maps through Pooling, CNN-Style Self-Attention (CSA), and a Convolutional Feed-Forward Network (CFFN) [2309.05674]. Its stated motivation is attention collapse: on relatively limited well-annotated medical image data, attention maps can become similar or even identical. CSA forms an unnormalized cosine-similarity map
$$
I^{\,i,j}_{m,n}=\frac{Q_{i,j}\cdot K_{m,n}}{\|Q_{i,j}\|\|K_{m,n}\|},
$$
then multiplies it by a learnable Gaussian mask to obtain an adaptive convolution kernel $A^{i,j}$ over the value map [2309.05674]. Across SETR, TransUNet, TransFuse, FAT-Net, and Patcher, the module yields consistent performance gains; for example, SETR on ACDC improves from Dice $87.14\%$ and HD $17.24$ to Dice $91.00\%$ and HD $14.08$, and across all backbones and datasets the reported gains are $+0.6$–$3.9\%$ Dice and $1$–$3$ pt HD reduction [2309.05674].

The hierarchical ConvFormer for medical image segmentation instead adopts a U-shaped encoder-decoder architecture built from a shallow Conv stem, three residual-shaped hybrid stems, an additional multi-scale Enhanced DeTrans encoder, and a symmetric decoder [2211.08564]. Enhanced DeTrans retains multi-scale deformable self-attention but re-designs the feed-forward module by inserting a $3\times3$ or $3^3$ depth-wise convolution, while Enhanced Positional Encoding adds a learnable DWConv branch to sinusoidal positional encoding. On MM-WHS CT, this ConvFormer reports MeanDice $0.925$ with $61$ M parameters, compared with $0.915$ for UNETR and $0.911$ for CoTr. On a lymph node ultrasound dataset it reports IoU $0.845\pm0.002$ and F1 $0.916\pm0.002$, and on ISIC skin lesion segmentation it reports Jaccard $0.797\pm0.003$ and Dice $0.889\pm0.002$ [2211.08564].

## 4. Sequential, geometric, and pose modeling

For sequential user modeling in recommender systems, ConvFormer is explicitly derived from an empirical analysis of self-attention in next-item prediction. The paper identifies three essential criteria for an effective token mixer: order sensitivity, large receptive field, and lightweight architecture [2308.02925]. The resulting model is a standard two-tower next-item model in which each Transformer-style block is replaced by a depth-wise convolution along the time axis and a channel-wise convolution per time step. After $N$ stacked LighTCN layers, the final user representation is $u=R^N[L]$, the score for a candidate item is $u^T E^I_c$, and training uses a pairwise ranking loss [2308.02925]. On four public datasets—Amazon-Beauty, Sports, Toys, and Yelp—with $L=50$, $D=64$, and $N=2$ blocks, ConvFormer consistently achieves the highest Hit@5/10 and MRR on $1$-vs-$99$ tests, improves MRR by $5$–$10\%$ relative over the best Transformer baseline and by $3$–$6\%$ over FMLP-Rec, and, in the FFT-accelerated ConvFormer-F version, gives a $3$–$5\times$ speedup over SASRec when $L$ is large [2308.02925].

PointConvFormer targets point-cloud segmentation and scene-flow estimation by combining point convolution, where filter weights are only based on relative position, with Transformer-style feature-based attention [2208.02879]. Its theoretical motivation is tied to generalization: attention filters out neighbors whose feature difference is large, while the point-convolution part preserves translation- and rotation-invariant geometric priors. On ScanNet semantic segmentation at a $2$ cm grid, PointConvFormer reports $74.5\%$ mIoU / $145$ ms / $9.4$ M parameters, compared with $70.3\%$ / $83$ ms / $5.4$ M for a PointConv bottleneck re-implementation and $71.9\%$ / $115$ ms / $37.9$ M for MinkowskiNet42. On SemanticKITTI it reports $67.1\%$ mIoU, exceeding RandLA-Net at $57.1\%$, MinkowskiNet at $61.1\%$, and SPVNAS at $64.7\%$. For scene flow, replacing PointConv with PointConvFormer in PointPWC-Net reduces EPE3D from $0.0588$ m / $0.0694$ m to $0.0416$ m / $0.0479$ m on FlyingThings3D / KITTI, described as $\sim30\%$ error reduction [2208.02879].

In monocular 3D human pose estimation, ConvFormer follows a two-stage lift pipeline: a spatial ConvFormer first models human joint relations within individual frames, then a temporal ConvFormer fuses all frame embeddings across time to predict the $3$D pose of the middle frame [2304.02147]. The central operator is dynamic multi-headed convolutional self-attention, in which each head forms $Q$, $K$, and $V$ by small $1$D convolutions and then aggregates multiple kernel sizes with learned nonnegative weights that sum to one. The temporal version introduces the temporal joints profile, in which each token already “sees” a $\kappa$-sized temporal neighborhood of joint features before attention weights are computed [2304.02147]. Parameter reduction is a primary design goal: for a $243$-frame sequence and $17$ joints, ConvFormer uses approximately $10.2$ M parameters versus approximately $31.5$ M for MHFormer, a $67.6\%$ reduction; the $143$-frame variant uses $5.24$ M versus $31.5$ M, an $83.4\%$ saving; and at $T=81$ it uses $3.43$ M versus approximately $9.6$ M for PoseFormer, a $64.3\%$ cut [2304.02147]. On Human3.6M with CPN inputs and $T=243$, it reports $43.2$ mm average MPJPE under Protocol I, $34.2$ mm under Protocol II, and a $14.3\%$ reduction in velocity error under Protocol III versus the previous SOTA. On MPI-INF-3DHP with $T=9$, it reports $96.4\%$ PCK, $69.8$ AUC, and $53.6$ mm MPJPE [2304.02147].

## 5. Speech enhancement and automatic speech recognition

In speech enhancement, DF-Conformer integrates Conv-TasNet and Conformer by using a Conformer-style mask-prediction network with linear-complexity FAVOR+ attention and $1$-D dilated depthwise convolutions [2106.15813]. The encoder is a learned $1$-D convolutional analysis filterbank with window $2.5$ ms, hop $1.25$ ms, and output dimension $D_e=256$, followed by an $L$-layer DF-Conformer mask network and an overlap-add synthesis decoder. Each block applies half-step FFN, MHSA\_FAVOR, GLU, dilated depthwise convolution, BatchNorm, Swish, pointwise dense projection, dropout, and a second half-step FFN, then LayerNorm [2106.15813]. The model was trained on $3{,}396$ hours of noisy speech data. In the reported comparisons, DF-Conformer-8 achieves SI-SNRi $14.43$ dB and ESTOI $85.4\%$ at real-time factor $0.13$, compared with TDCN++ at SI-SNRi $14.10$ dB, ESTOI $85.7\%$, and RTF $0.10$. The iterative iDF-Conformer-12 variant reaches SI-SNRi $15.93$ dB and ESTOI $88.4\%$ at RTF $0.46$ [2106.15813].

The optimized ASR Conv-Former described in “Practical Conformer” is a streamlined, memory-lite Conformer encoder intended for ultra-low-latency, on-device ASR and as the first pass in a two-stage cascaded system [2304.00171]. Its three principal interventions are replacing lower Conformer blocks with convolution-only blocks, strategically downsizing the architecture, and utilizing an RNNAttention-Performer. Relative to a $12$-layer causal Conformer with WER $6.5\%$, size $120$ M, FLOPs $247$ M, and TPU latency $21.8$ ms, the optimized Conv-Former reports WER $7.7\%$, size $56$ M, FLOPs $93$ M, and TPU latency $3.2$ ms, a $6.8\times$ latency reduction [2304.00171]. In a cascaded encoder design, the first-pass causal Conv-Former produces frame-level embeddings for a low-latency RNN-T decoder, and a second-pass non-causal Conformer operates on those embeddings when more compute is available. The second pass recovers WER to $5.8\%$, matching the large-model pipeline [2304.00171].

Multi-Convformer revisits the Conformer convolution module itself. Instead of a single depthwise convolution of fixed kernel size, it applies $P$ parallel depthwise convolutions with different kernel sizes and fuses them with gating; the best-performing choice is $P=4$ with $\{7,15,23,31\}$ [2407.03718]. The default variant, MultiConv\(_{\rm depth}\), performs best among sum, weighted-sum, concatenation, and concat-plus-depthwise-conv fusion strategies. In a $12$-layer AED model, Conformer has $17.9$ M total parameters and Multi-Convformer\(_{\rm depth}\) has $18.8$ M, a reported $+5\%$ overhead [2407.03718]. On LS-100 under the AED setting, WER improves from $6.7$ to $6.2$ on Test Clean and from $20.1$ to $18.5$ on Test Other; on TEDLIUM-2 it improves from $9.1$ to $8.7$; and on AISHELL Test the CER improves from $5.4$ to $5.1$. The paper summarizes the effect as up to $8\%$ relative WER improvements while remaining more parameter efficient than existing Conformer variants such as CgMLP and E-Branchformer [2407.03718].

## 6. Empirical patterns, limitations, and misconceptions

A consistent empirical pattern is that Conv-Former variants are used to reconcile local inductive bias with long-range context while improving the accuracy-speed-parameter trade-off. In recommendation, the claimed benefit arises from order sensitivity, large receptive field, and weight sharing [2308.02925]. In super-resolution and fine-grained classification, large-kernel or convolutionally guided designs are used to retain broad spatial context without the quadratic cost of global self-attention or without downscaling away fine detail [2401.05633], [2208.08900]. In medical segmentation, convolutional structure is explicitly used to mitigate attention collapse on small-scale training data and to preserve $2$D feature-map geometry [2309.05674]. In point clouds, the benefit is boundary-aware neighborhood selection while preserving the invariances of continuous point convolution [2208.02879].

The literature also records nontrivial trade-offs. In the optimized ASR Conv-Former, the small single-pass encoder incurs a WER increase from $6.5\%$ to $7.7\%$ before the cascaded second pass recovers accuracy [2304.00171]. In DF-Conformer, FAVOR+ can slightly degrade ESTOI and SI-SNRi in the smallest setting, even though it enables scaling to large sequence lengths [2106.15813]. In Multi-Convformer, adding too many kernels is not uniformly beneficial: the $4$-kernel set $\{7,15,23,31\}$ performs best on the cited ablation, whereas the $5$-kernel set $\{7,\dots,39\}$ degrades [2407.03718]. The hierarchical medical segmentation ConvFormer also notes increased model complexity and training time, approximately $34$–$96$ GPU-hours, and potential memory bottlenecks for very high resolutions [2211.08564].

Several misconceptions are therefore contradicted by the arXiv record. First, Conv-Former is not synonymous with an attention-free network: the 3D pose ConvFormer, CSA-based medical ConvFormer, Enhanced DeTrans ConvFormer, DF-Conformer, and Multi-Convformer all retain attention or Conformer-style attention modules [2304.02147], [2309.05674], [2211.08564], [2106.15813], [2407.03718]. Second, Conv-Former is not restricted to vision; the term is used in recommendation, speech enhancement, and ASR as well [2308.02925], [2106.15813], [2304.00171]. Third, parameter reduction is common but not universal: some variants primarily target convergence behavior, attention diversity, or high-resolution handling rather than absolute parameter minimization [2208.08900], [2309.05674].

Taken together, the cited works support a narrower but more precise interpretation. “Conv-Former” names a family of architectures that treat convolution not as a legacy component to be displaced by attention, but as a first-class mechanism for locality, order sensitivity, multi-scale aggregation, efficiency, and stabilization. The specific implementation differs sharply by domain—large-kernel mixers in SISR, convolutional projections in pose estimation, adaptive attention kernels in medical imaging, feature-difference gating in point clouds, and multi-kernel or linear-attention Conformer modules in speech—but the unifying objective is recurrent: preserve the strengths of convolution while retaining the expressive and modular benefits of Transformer-family design.

Source: https://www.emergentmind.com/topics/conv-former