ConvUtr: Convolution-Transformer Hybrid

• ConvUtr is a hybrid model combining convolutional inductive bias with Transformer-style tokenization to produce denoised, semantically rich embeddings for medical imaging.
• In MobileUtr and Mobile U-ViT, ConvUtr functions as an efficient CNN tokenizer using depthwise-separable convolutions, residual connections, and max-pooling to reduce computational cost while maintaining accuracy.
• For speech recognition, ConvUtr is implemented as a Conv-Transformer Transducer that interleaves convolutional layers with unidirectional Transformers to achieve low latency and reduced frame rate processing.

Searching arXiv for the relevant ConvUtr-related papers and closely associated work. ConvUtr is a term used in recent arXiv literature in two distinct senses. In medical image segmentation, it denotes a Transformer-like CNN patch embedding introduced in "MobileUtr: Revisiting the relationship between light-weight CNN and Transformer for efficient medical image segmentation" and later reused as a hierarchical embedding in "Mobile U-ViT: Revisiting large kernel and U-shaped ViT for efficient medical image segmentation" (Tang et al., 2023, Tang et al., 1 Aug 2025). In automatic speech recognition, the same shorthand has been used to refer to the Conv-Transformer Transducer, a streamable Transducer architecture with a unidirectional Transformer encoder interleaved with convolutional layers (Huang et al., 2020). The shared label therefore designates architectures that combine convolutional inductive bias with Transformer-style representation learning, but the underlying tasks, data modalities, and implementation details differ substantially.

1. Terminological scope and disambiguation

Within the medical imaging literature, ConvUtr is a CNN-based embedding module placed before a lightweight Transformer bottleneck. Its stated purpose is to abstract a Transformer-like lightweight CNN block as the patch embeddings of ViTs, thereby feeding the Transformer with denoised, non-redundant and highly condensed semantic information (Tang et al., 2023). In that setting, ConvUtr is not the full network; it is an encoder-side component inside MobileUtr and, later, Mobile U-ViT.

By contrast, in speech recognition the shorthand refers to the entire Conv-Transformer Transducer rather than to a submodule. That model replaces attention-based alignment with a Transducer framework, employs a unidirectional Transformer encoder, interleaves convolutional layers to model future context, and gradually downsamples acoustic input for lower frame rate (Huang et al., 2020). The overlap in naming is therefore lexical rather than architectural identity.

A further source of possible confusion is ConUNETR, which is a separate conditional Transformer network for 3D micro-CT embryonic cartilage segmentation with age tokens and slice spatial encoding (Sapkota et al., 2024). ConUNETR does not define or use ConvUtr as its core mechanism.

2. ConvUtr as a Transformer-like CNN embedding in MobileUtr

In MobileUtr, ConvUtr is a depthwise-separable, inverted-bottleneck residual block that mirrors the MHSA-FFN decomposition in ViTs by factorizing spatial mixing and channel mixing with residual connections and GELU nonlinearity (Tang et al., 2023). For the $l$-th unit in a stage, with input $X_l \in \mathbb{R}^{h \times w \times C}$, the block is defined as

$Y_l = BN(GELU(DWConv(X_l))) + X_l$

$Z_l = BN(GELU(PWConv(Y_l)))$

$X_{l+1} = BN(GELU(PWConv(Z_l))) + Y_l$

where the first pointwise convolution expands channels with ratio $r = 4$ and the second projects them back to $C$ (Tang et al., 2023). Depthwise convolution uses groups $= C$, stride $= 1$ within a stage, and stage kernels are $K_1 = 3$, $X_l \in \mathbb{R}^{h \times w \times C}$0, and $X_l \in \mathbb{R}^{h \times w \times C}$1 in MobileUtr base. BatchNorm is used throughout ConvUtr, GELU is the activation, and residual connections are applied after the depthwise operation and after the second pointwise projection.

The design rationale is explicit. Depthwise convolution is presented as a content-independent analogue of headwise spatial mixing, the two pointwise convolutions emulate the ViT FFN with $X_l \in \mathbb{R}^{h \times w \times C}$2 expansion, and BatchNorm is chosen instead of LayerNorm to better match CNN training dynamics on small-batch medical regimes (Tang et al., 2023). Max pooling between stages is deliberately preferred over strided convolution because it denoises low-resolution, noisy medical images efficiently.

This formulation situates ConvUtr between conventional CNN stems and early-attention hybrids. Compared with a single stride-$X_l \in \mathbb{R}^{h \times w \times C}$3 convolution used as patch embedding, ConvUtr adds FFN-like channel mixing and residuals to generate semantic rather than merely visual tokens. Compared with MobileViT and LeViT, the early stages remain purely CNN-based, so locality, translation invariance, and noise suppression are front-loaded before global modeling (Tang et al., 2023).

3. Hierarchical tokenization and interface with local-global modeling

The MobileUtr encoder uses three ConvUtr stages followed by $X_l \in \mathbb{R}^{h \times w \times C}$4 max pooling with stride $X_l \in \mathbb{R}^{h \times w \times C}$5 after each stage, producing an overall spatial reduction factor of $X_l \in \mathbb{R}^{h \times w \times C}$6 before tokenization (Tang et al., 2023). Given an input image $X_l \in \mathbb{R}^{h \times w \times C}$7, the ConvUtr-based embedding is formalized as

$X_l \in \mathbb{R}^{h \times w \times C}$8

and the token sequence is obtained by flattening,

$X_l \in \mathbb{R}^{h \times w \times C}$9

In MobileUtr base, the first three encoder stages use channel plan $Y_l = BN(GELU(DWConv(X_l))) + X_l$0, $Y_l = BN(GELU(DWConv(X_l))) + X_l$1, and $Y_l = BN(GELU(DWConv(X_l))) + X_l$2, with block lengths $Y_l = BN(GELU(DWConv(X_l))) + X_l$3, $Y_l = BN(GELU(DWConv(X_l))) + X_l$4, and $Y_l = BN(GELU(DWConv(X_l))) + X_l$5 (Tang et al., 2023). The reported stagewise tensor progression is:

• Stage 1: $Y_l = BN(GELU(DWConv(X_l))) + X_l$6, then $Y_l = BN(GELU(DWConv(X_l))) + X_l$7
• Stage 2: $Y_l = BN(GELU(DWConv(X_l))) + X_l$8, then $Y_l = BN(GELU(DWConv(X_l))) + X_l$9
• Stage 3: $Z_l = BN(GELU(PWConv(Y_l)))$0, then $Z_l = BN(GELU(PWConv(Y_l)))$1

A $Z_l = BN(GELU(PWConv(Y_l)))$2 convolution may optionally project channels to the Transformer embedding dimension $Z_l = BN(GELU(PWConv(Y_l)))$3 before flattening (Tang et al., 2023).

After ConvUtr, MobileUtr inserts an adaptive Local-Global-Local bottleneck as an adapter between CNN and Transformer representations. The LGL block composes LocalAgg, GlobalSP, and LocalPro, respectively described as large-kernel convolution for local aggregation, global sparse attention for long-range transmission, and transposed convolution for local redistribution (Tang et al., 2023). The kernel size for LocalAgg is set by

$Z_l = BN(GELU(PWConv(Y_l)))$4

where $Z_l = BN(GELU(PWConv(Y_l)))$5 is the average object diameter in the dataset and $Z_l = BN(GELU(PWConv(Y_l)))$6 is the number of pre-ViT downsamplings, with $Z_l = BN(GELU(PWConv(Y_l)))$7 in MobileUtr; the implementation also reports using $Z_l = BN(GELU(PWConv(Y_l)))$8 by default (Tang et al., 2023). This suggests that ConvUtr is intended not as a stand-alone tokenizer but as the first half of a staged local-to-global transition, in which convolutional inductive bias and an intermediate adapter jointly reduce the burden placed on the Transformer bottleneck.

4. Computational characteristics and empirical behavior in MobileUtr

The computational profile of ConvUtr is given explicitly. For a ConvUtr unit with input/output channels $Z_l = BN(GELU(PWConv(Y_l)))$9, spatial size $X_{l+1} = BN(GELU(PWConv(Z_l))) + Y_l$0, kernel $X_{l+1} = BN(GELU(PWConv(Z_l))) + Y_l$1, and expansion $X_{l+1} = BN(GELU(PWConv(Z_l))) + Y_l$2, the total parameter count is approximated as

$X_{l+1} = BN(GELU(PWConv(Z_l))) + Y_l$3

and the total MACs per unit are approximated as

$X_{l+1} = BN(GELU(PWConv(Z_l))) + Y_l$4

The comparison target is a vanilla ViT patch embedding via a non-overlapping $X_{l+1} = BN(GELU(PWConv(Z_l))) + Y_l$5 convolution from $X_{l+1} = BN(GELU(PWConv(Z_l))) + Y_l$6 to $X_{l+1} = BN(GELU(PWConv(Z_l))) + Y_l$7, whose parameters are $X_{l+1} = BN(GELU(PWConv(Z_l))) + Y_l$8 and whose MACs reduce to $X_{l+1} = BN(GELU(PWConv(Z_l))) + Y_l$9 (Tang et al., 2023). The reported explanation is that ConvUtr lowers cost by using depthwise spatial mixing, operating mostly at reduced resolution due to pooling, and keeping $r = 4$0 relatively small before the ViT bottleneck.

Ablation results on Synapse isolate the effect of replacing heavier stems and adapters. The reported configurations are summarized below (Tang et al., 2023).

Configuration	Params / GFLOPs	mIoU
ResNet34 + ViTs, no skip	25.65 M / 85.95	63.76
ResNet34 + LGL, no skip	22.11 M / 81.41	63.25
ConvUtr + LGL, no skip	1.32 M / 2.37	63.16
ConvUtr + Adaptive LGL, no skip	1.34 M / 2.39	64.40
Full skip design	1.39 M / 2.50	68.17

These data show that replacing ResNet34+ViT embedding with ConvUtr+LGL reduced parameters from $r = 4$1 M to $r = 4$2 M and GFLOPs from $r = 4$3 to $r = 4$4, while maintaining similar mIoU on Synapse $r = 4$5 (Tang et al., 2023). With the full adapter and skip design, performance rises to $r = 4$6 mIoU at $r = 4$7 M parameters and $r = 4$8 GFLOPs.

The same paper reports full-benchmark performance across ultrasound, dermoscopy, and CT. MobileUtr $r = 4$9 M, $C$0 GFLOPs, $C$1 FPS$C$2 attains, for example, IoU/F1 of $C$3 on BUS, $C$4 on BUSI, $C$5 on TNSCUI, and $C$6 on ISIC18 (Tang et al., 2023). On Synapse, MobileUtr achieves $C$7 mIoU and $C$8 Dice, while MobileUtr-L reaches $C$9 mIoU and $= C$0 Dice with $= C$1 HD95, compared with TransUnet’s $= C$2 mIoU and $= C$3 Dice (Tang et al., 2023).

Implementation details further clarify the intended operating regime. ConvUtr is trained with BatchNorm and GELU, uses max pooling for downsampling, reports no explicit positional encodings for its tokens, and is optimized with SGD, momentum $= C$4, weight decay $= C$5, initial learning rate $= C$6 with poly schedule, batch size $= C$7, $= C$8 epochs, input resized to $= C$9, and loss $= 1$0 (Tang et al., 2023). Max-pooling downsampling is reported as empirically better on Synapse than convolutional downsampling, with mIoU $= 1$1 versus $= 1$2 at nearly the same complexity.

5. Large-kernel ConvUtr in Mobile U-ViT

"Mobile U-ViT: Revisiting large kernel and U-shaped ViT for efficient medical image segmentation" reuses the name ConvUtr but refines the emphasis toward a large-kernel, hierarchical patch embedding with inverted bottleneck fusion (Tang et al., 1 Aug 2025). In this formulation, ConvUtr is still the CNN-based embedding used in the first three encoder stages, but it is explicitly presented as a parameter-efficient large-kernel CNN with transformer-like representation learning capacity.

The core block is again expressed as

$= 1$3

$= 1$4

$= 1$5

where $= 1$6 is GELU and BN is BatchNorm (Tang et al., 1 Aug 2025). Stagewise hyperparameters are given for Base and Large variants: $= 1$7, $= 1$8, $= 1$9; kernels $K_1 = 3$0, $K_1 = 3$1, $K_1 = 3$2; channels $K_1 = 3$3, $K_1 = 3$4, and $K_1 = 3$5 for Base/Large respectively. Max-pooling follows each stage.

The paper also supplies complexity formulas in a generalized inverted-bottleneck notation with expansion ratio $K_1 = 3$6. For a single block on spatial size $K_1 = 3$7 with channel count $K_1 = 3$8,

$K_1 = 3$9

and

$X_l \in \mathbb{R}^{h \times w \times C}$00

It contrasts this with conventional convolution complexity

$X_l \in \mathbb{R}^{h \times w \times C}$01

and states that ConvUtr reduces this to

$X_l \in \mathbb{R}^{h \times w \times C}$02

For 3D adaptation, the FLOPs generalize to

$X_l \in \mathbb{R}^{h \times w \times C}$03

with 3D depthwise separable convolutions replacing the 2D operators (Tang et al., 1 Aug 2025).

The architectural context also evolves. After the three ConvUtr stages, Mobile U-ViT introduces an LKLGL stage with large-kernel local aggregation, token pooling with ratio $X_l \in \mathbb{R}^{h \times w \times C}$04, efficient global attention, and local redistribution by transposed convolution with kernel $X_l \in \mathbb{R}^{h \times w \times C}$05 (Tang et al., 1 Aug 2025). Pooling reduces tokens by $X_l \in \mathbb{R}^{h \times w \times C}$06, lowering global attention complexity from $X_l \in \mathbb{R}^{h \times w \times C}$07 to approximately $X_l \in \mathbb{R}^{h \times w \times C}$08. The final Transformer bottleneck remains shallow, with $X_l \in \mathbb{R}^{h \times w \times C}$09 and $X_l \in \mathbb{R}^{h \times w \times C}$10 for Base/Large.

Empirical evidence again isolates the contribution of ConvUtr. Replacing a MobileNetV2 stem with ConvUtr increases Synapse slice-level Jaccard from $X_l \in \mathbb{R}^{h \times w \times C}$11 for MV2(LK)+MViT to $X_l \in \mathbb{R}^{h \times w \times C}$12 for ConvUtr+MViT, a gain of $X_l \in \mathbb{R}^{h \times w \times C}$13 points (Tang et al., 1 Aug 2025). The complete Mobile U-ViT configuration with ConvUtr, LKLGL, ViT, full skip3, and cascaded decoder reaches $X_l \in \mathbb{R}^{h \times w \times C}$14 Jaccard with only $X_l \in \mathbb{R}^{h \times w \times C}$15 M parameters and $X_l \in \mathbb{R}^{h \times w \times C}$16 GFLOPs. The broader system-level efficiency figures are $X_l \in \mathbb{R}^{h \times w \times C}$17 M parameters, $X_l \in \mathbb{R}^{h \times w \times C}$18 GFLOPs, and $X_l \in \mathbb{R}^{h \times w \times C}$19 FPS for 2D Base, and $X_l \in \mathbb{R}^{h \times w \times C}$20 M parameters, $X_l \in \mathbb{R}^{h \times w \times C}$21 GFLOPs, and $X_l \in \mathbb{R}^{h \times w \times C}$22 FPS on $X_l \in \mathbb{R}^{h \times w \times C}$23 crops for 3D Base (Tang et al., 1 Aug 2025).

This later formulation therefore preserves the main identity of ConvUtr as a lightweight convolutional tokenizer, while placing greater weight on large kernels, overlapping patches, and 3D scalability. A plausible implication is that the name came to denote a reusable design pattern rather than a single fixed implementation.

6. ConvUtr as Conv-Transformer Transducer in streaming speech recognition

In "Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition," ConvUtr denotes the full ASR architecture rather than a medical-imaging submodule (Huang et al., 2020). The system combines a Transducer framework with a unidirectional Transformer encoder interleaved with convolution blocks. Its objective is to make Transformer-style ASR streamable by replacing full-sequence alignment learning with RNN-T-style alignment, constraining right context to convolutional layers, and limiting self-attention history so that computation per decoding step remains constant.

The architecture comprises an encoder, a prediction network, and a joint network. The encoder has three blocks; each block contains three convolution layers followed by a stack of unidirectional Transformer layers, with stride $X_l \in \mathbb{R}^{h \times w \times C}$24 in time applied at the second convolution layer in each block (Huang et al., 2020). The default encoder layer allocation is $X_l \in \mathbb{R}^{h \times w \times C}$25 Transformer layers in Block 1, $X_l \in \mathbb{R}^{h \times w \times C}$26 in Block 2, and $X_l \in \mathbb{R}^{h \times w \times C}$27 in Block 3, yielding a total of $X_l \in \mathbb{R}^{h \times w \times C}$28 encoder layers. The prediction network uses an embedding layer, linear projection, and unidirectional Transformer, while the joint network concatenates encoder and predictor states and passes them through a one-hidden-layer MLP with $X_l \in \mathbb{R}^{h \times w \times C}$29 ReLU units before softmax.

The Transducer probability formulation is stated as a marginalization over alignment paths, with forward recursion

$X_l \in \mathbb{R}^{h \times w \times C}$30

$X_l \in \mathbb{R}^{h \times w \times C}$31

and sequence probability $X_l \in \mathbb{R}^{h \times w \times C}$32, so the loss is

$X_l \in \mathbb{R}^{h \times w \times C}$33

The joint network is defined by

$X_l \in \mathbb{R}^{h \times w \times C}$34

$X_l \in \mathbb{R}^{h \times w \times C}$35

$X_l \in \mathbb{R}^{h \times w \times C}$36

where $X_l \in \mathbb{R}^{h \times w \times C}$37 is a linear layer with ReLU (Huang et al., 2020).

The efficiency claims are tied to latency and frame rate. Input features are $X_l \in \mathbb{R}^{h \times w \times C}$38-dimensional log Mel filterbanks with $X_l \in \mathbb{R}^{h \times w \times C}$39 ms window and $X_l \in \mathbb{R}^{h \times w \times C}$40 ms hop. Because stride-$X_l \in \mathbb{R}^{h \times w \times C}$41 downsampling occurs in the second convolution of each block, the default cumulative factor is $X_l \in \mathbb{R}^{h \times w \times C}$42, producing an $X_l \in \mathbb{R}^{h \times w \times C}$43 ms frame rate from the original $X_l \in \mathbb{R}^{h \times w \times C}$44 ms rate; a high-frame-rate variant uses stride $X_l \in \mathbb{R}^{h \times w \times C}$45 in the third block and yields $X_l \in \mathbb{R}^{h \times w \times C}$46 ms (Huang et al., 2020). All future context comes from the convolution stack, and the total look-ahead is reported as $X_l \in \mathbb{R}^{h \times w \times C}$47 ms. Limited-history self-attention reduces total complexity from $X_l \in \mathbb{R}^{h \times w \times C}$48 to $X_l \in \mathbb{R}^{h \times w \times C}$49 and per-step cost to $X_l \in \mathbb{R}^{h \times w \times C}$50.

The reported LibriSpeech results are $X_l \in \mathbb{R}^{h \times w \times C}$51 WER on test-clean and $X_l \in \mathbb{R}^{h \times w \times C}$52 on test-other for the $X_l \in \mathbb{R}^{h \times w \times C}$53 M-parameter ConvUtr at $X_l \in \mathbb{R}^{h \times w \times C}$54 ms look-ahead and $X_l \in \mathbb{R}^{h \times w \times C}$55 ms frame rate, without external LLMs (Huang et al., 2020). The high-frame-rate variant reaches $X_l \in \mathbb{R}^{h \times w \times C}$56 and $X_l \in \mathbb{R}^{h \times w \times C}$57. Compared with a previously published Transformer Transducer at $X_l \in \mathbb{R}^{h \times w \times C}$58 M parameters, $X_l \in \mathbb{R}^{h \times w \times C}$59 ms look-ahead, and $X_l \in \mathbb{R}^{h \times w \times C}$60 ms frame rate, ConvUtr is reported as competitive while using fewer parameters and smaller look-ahead (Huang et al., 2020).

Despite the name overlap, this usage of ConvUtr is conceptually distinct from the medical-image ConvUtr block. The commonality lies in architectural philosophy: convolution is used to provide inductive bias, future context control, or efficient local aggregation, while Transformer components handle broader contextual modeling.

7. Scope, limitations, and recurring design motifs

The medical-image ConvUtr is explicitly described as best suited for $X_l \in \mathbb{R}^{h \times w \times C}$61D medical segmentation with scarce, noisy data, low contrast, and blurred edges (Tang et al., 2023). Scaling to very high resolutions or $X_l \in \mathbb{R}^{h \times w \times C}$62D volumes requires adapting strides, receptive fields, and memory budgets, and a $X_l \in \mathbb{R}^{h \times w \times C}$63D ConvUtr variant would increase cost significantly unless carefully factorized (Tang et al., 2023). BatchNorm may be sensitive to very small batch sizes, in which case synchronization or GroupNorm variants may be needed. The adaptive Local-Global-Local mechanism also requires estimating average object size $X_l \in \mathbb{R}^{h \times w \times C}$64; an inaccurate prior may suboptimalize local aggregation, which is why a conservative default such as $X_l \in \mathbb{R}^{h \times w \times C}$65 is used in practice (Tang et al., 2023).

The Mobile U-ViT variant adds a related set of heuristics: gradual kernel growth $X_l \in \mathbb{R}^{h \times w \times C}$66 is preferred to abrupt jumps, max-pooling remains preferable to convolutional downsampling in this medical setting, and the paper notes that exact initialization details are not specified (Tang et al., 1 Aug 2025). The 3D extension follows the same block logic, but the move from $X_l \in \mathbb{R}^{h \times w \times C}$67 to $X_l \in \mathbb{R}^{h \times w \times C}$68 scaling underscores why tight computational control remains central.

The speech-recognition ConvUtr has a different limitation profile. The encoder’s limited left-context window and the $X_l \in \mathbb{R}^{h \times w \times C}$69 ms right-context may degrade performance in very noisy or long-context scenarios, and the paper identifies potential future gains from Conformer-style modules, refined chunked streaming strategies, and LLM fusion (Huang et al., 2020). The authors also note incomplete disclosure of attention heads, $X_l \in \mathbb{R}^{h \times w \times C}$70, $X_l \in \mathbb{R}^{h \times w \times C}$71, and convolution kernel or dilation sizes, which constrains exact reproducibility.

Across these uses, ConvUtr consistently denotes a convolution-Transformer hybrid that shifts some of the representational burden away from quadratic or globally connected attention and into structured, low-cost convolutional operators. In medical segmentation, this takes the form of a CNN tokenizer that produces denoised, compact semantic tokens for a shallow Transformer (Tang et al., 2023, Tang et al., 1 Aug 2025). In streaming ASR, it takes the form of a Transducer architecture that uses interleaved convolutions to supply limited future context and reduce frame rate while preserving constant per-step decoding cost (Huang et al., 2020). The recurrence of the name across domains reflects a shared design orientation rather than a single canonical model.