Inverted Residual Embedding (IRE) in MobileNetV2 & ViTs

• Inverted Residual Embedding (IRE) is a MobileNetV2-inspired module that utilizes a thin bottleneck expanded to a wider intermediate space via depthwise convolutions.
• IRE replaces standard patch embeddings in transformers by integrating feature compression with convolutional locality, leading to significant FLOPS reduction.
• Variants of IRE extend its use to tasks like object detection and segmentation by leveraging bottleneck shortcut connectivity for efficient feature extraction.

Inverted Residual Embedding (IRE) denotes, in its most explicit usage, an embedding module derived from the MobileNetV2 pattern of thin bottleneck inputs and outputs, an expanded intermediate representation, depthwise spatial filtering, and linear projection; in recent Vision Transformer work, it appears as a replacement for conventional patch embedding (Sandler et al., 2018, Huang et al., 24 Sep 2025). The term is not uniform across the cited literature. MobileNetV2 provides the architectural basis but does not use the phrase verbatim, while other papers either reuse the acronym “IRE” for a different module or interpret residualized representations as an IRE only retrospectively (Huang et al., 2023, He et al., 26 Oct 2025).

1. Terminology and scope

The phrase “Inverted Residual Embedding” is best understood by separating explicit usage from conceptual usage. In one line of work, it names a concrete frontend embedding block for transformers; in another, it functions as a descriptive label for MobileNetV2-style thin–wide–thin mappings; elsewhere, the acronym “IRE” refers to a different residual mechanism altogether.

Context	Term usage	Core idea
"MobileNetV2" (Sandler et al., 2018)	Not verbatim; conceptual basis	Shortcut between thin bottlenecks, wide intermediate transform
"EfficienT-HDR" (Huang et al., 24 Sep 2025)	Exact term used	Replace standard ViT patch embedding with an inverted-residual module
"RestNet" (Huang et al., 2023)	IRE = Intra-domain Residual Enhancement	Add transformed and original features
"Far from the Shallow" (He et al., 26 Oct 2025)	Not explicit; retrospective interpretation	Residualize deep states by subtracting lower-level projections

This distribution of usage suggests that IRE is not a single fixed designation across fields. In the narrowest and most technically precise sense, it refers to a MobileNetV2-inspired embedding operator. In a broader interpretive sense, it can denote any embedding mechanism that preserves a low-dimensional bottleneck representation while locating nonlinear expressiveness in an expanded intermediate space, or that isolates a residual subspace by removing lower-level components.

2. Architectural foundation in MobileNetV2

The architectural source of IRE is MobileNetV2’s “inverted residual with linear bottleneck,” where the shortcut connections are between the thin bottleneck layers and the intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity (Sandler et al., 2018). Its block structure is thin $\rightarrow$ wide $\rightarrow$ thin: a $1\times1$ expansion convolution maps $k \rightarrow tk$, a $3\times3$ depthwise convolution operates in the expanded space, and a linear $1\times1$ projection maps $tk \rightarrow k'$. When $s=1$ and $k=k'$, the residual shortcut is

$\text{output} = x + \text{block}(x).$

A compact functional form used in the MobileNetV2 synthesis is

$\rightarrow$0

with no nonlinearity after $\rightarrow$1 (Sandler et al., 2018). The inversion relative to classical ResNet lies in the placement of the shortcut: classical residuals connect high-channel tensors, whereas inverted residuals connect bottlenecks. The high-dimensional representation is treated as an internal transform space rather than the state space preserved across blocks.

This design is accompanied by a quantitative efficiency profile. MobileNetV2 reports 72.0% top-1 with 3.4M params, 300M MAdds, and 75 ms CPU; MobileNetV2 (1.4) reports 74.7% top-1 with 6.9M params and 585M MAdds (Sandler et al., 2018). The same block family also underlies MobileNetV2 + SSDLite for detection and Mobile DeepLabv3 for semantic segmentation, indicating that the inverted residual pattern is not confined to classification.

3. Embedding-space interpretation and linear bottlenecks

The “embedding” aspect of IRE follows from MobileNetV2’s treatment of the bottleneck as a low-dimensional representation space. The input bottleneck tensor is expanded into a wider intermediate tensor, transformed there by depthwise convolution and ReLU6, and then projected back into a low-dimensional output space with a linear layer (Sandler et al., 2018). The paper’s geometric argument is that the manifold of interest is low-dimensional and that bottleneck layers should therefore remain linear, because “Experimental evidence suggests that using linear layers is crucial as it prevents non-linearities from destroying too much information” (Sandler et al., 2018).

The theoretical intuition is expressed through mappings of the form

$\rightarrow$2

where expansion precedes the nonlinearity. The cited synthesis states that if $\rightarrow$3 and ReLU leaves even a small fraction of coordinates positive, then with high probability $\rightarrow$4 is invertible on the manifold of interest (Sandler et al., 2018). In this view, the narrow bottleneck stores the semantic state, while the expanded space carries the nonlinear transformation capacity. The paper explicitly describes this as decoupling the input/output domains from the expressiveness of the transformation.

This perspective explains why the term “embedding” can be attached to inverted residual blocks even when the original paper does not use that wording. The bottleneck is the representation that is propagated, stored, and, when possible, linked by the shortcut. The expanded tensor is not the enduring representation; it is the computation space in which expressive nonlinear filtering occurs.

4. Explicit IRE in transformer embedding: EfficienT-HDR

A direct, explicit use of the term appears in "EfficienT-HDR," where Inverted Residual Embedding replaces the conventional patch embedding of a transformer backbone (Huang et al., 24 Sep 2025). The module is defined as

$\rightarrow$5

with a pointwise expansion, a $\rightarrow$6 depthwise convolution with stride $\rightarrow$7, an SE-Net channel attention block, and a pointwise projection. It is inserted after Intersection-Aware Adaptive Fusion and before stacked Context-Aware ViT blocks, so it serves as the frontend tokenization and feature-compression stage.

The motivation is tied to the quadratic complexity of self-attention, written as

$\rightarrow$8

Because the term $\rightarrow$9 grows quadratically with the number of patches, the paper uses IRE to downsample and compress features before attention (Huang et al., 24 Sep 2025). This turns embedding from a simple linear projection into what the paper describes as a module that combines feature extraction and complexity control. The reported efficiency gains are substantial: compared to the baseline, the main version reduces FLOPS by approximately 67% and increases inference speed by more than fivefold on CPU and 2.5 times on an edge device. In the ablation specifically isolating IRE, FLOPS decrease from 20.95 G without IRE to 12.10 G with IRE, while parameters increase from 1.19 M to 1.45 M. Image quality remains close: with IRE, PSNR-$1\times1$0 is 38.0760, SSIM-$1\times1$1 is 0.9867, PSNR-$1\times1$2 is 40.9750, SSIM-$1\times1$3 is 0.9891, and HDR-VDP-2 is 64.55; without IRE, the corresponding values are 38.7190, 0.9869, 40.7420, 0.9897, and 65.19 (Huang et al., 24 Sep 2025).

In this explicit instantiation, IRE is not merely a residual block transplanted into a transformer. It is a token-reduction operator that injects convolutional locality and channel attention into the embedding stage while preserving the MobileNetV2 logic of expand $1\times1$4 depthwise transform $1\times1$5 project.

5. Variants, extensions, and downstream use

The inverted-residual embedding pattern extends beyond transformer tokenization. MobileNetV2 itself shows that bottleneck-connected shortcuts generalize effectively to dense prediction pipelines: MobileNetV2 + SSDLite attains 22.1 mAP on COCO with 4.3M params and 0.8B MAdds, while an efficient Mobile DeepLabv3 configuration reports mIOU = 75.32%, 2.11M params, and 2.75B MAdds (Sandler et al., 2018). These examples indicate that bottleneck-domain residual embeddings can serve as reusable feature extractors across classification, detection, and segmentation.

A closely related detector-side variant appears in "YOLO-FireAD," whose Attention-guided Inverted Residual Block is described as very close to what one might call an Inverted Residual Embedding (Pan et al., 27 May 2025). AIR performs channel reduction with reduction rate $1\times1$6, applies a depthwise $1\times1$7 convolution, and injects convolutional additive self-attention through spatial and channel branches. In parallel, the Dual Pool Downscale Fusion Block preserves multi-scale fire patterns via learnable fusion of max-average pooling outputs. The full model reports 1.45M parameters, 4.6G FLOPs, and mAP75 higher than YOLOv8n, YOLOv9t, YOLOv10n, YOLO11n, YOLOv12n, and other YOLOv8 variants by 1.3–5.5% (Pan et al., 27 May 2025).

These cases suggest that IRE-like modules are not tied to one network family. The common thread is an economical latent transform built around depthwise convolution and a bottleneck-domain representation, with added attention or pooling when the application requires stronger local discrimination or multi-scale preservation.

6. Boundaries of the concept and common confusions

A recurrent source of confusion is the reuse of the acronym “IRE” for modules that are not inverted residual embeddings in the MobileNetV2 sense. In "RestNet," IRE means Intra-domain Residual Enhancement, defined by

$1\times1$8

where transformed features are added back to the original masked features (Huang et al., 2023). The paper explicitly states that it does not use the term “inverted residual” as in MobileNetV2 and that the module is a straight residual add rather than a bottleneck or inverted bottleneck block. This distinction is technically important: the presence of a residual connection alone does not make a module an inverted residual embedding.

Two additional papers broaden the residual-embedding idea in other directions. "Far from the Shallow" constructs a reasoning embedding by residual disentanglement, with

$1\times1$9

after first identifying feature-specific saturation layers and iteratively regressing out lower-level representations (He et al., 26 Oct 2025). The resulting lexicon, syntax, meaning, and reasoning embeddings are described as nearly orthogonal, with all off-diagonal entries falling below approximately 0.045 after residualization, compared to strong overlap of approximately 0.75 in raw hidden states. "Complementing Lexical Retrieval with Semantic Residual Embedding" trains a dense retriever to model what BM25 misses by using a residual-based hinge loss whose margin depends on lexical score differences,

$k \rightarrow tk$0

so that the embedding focuses on lexical residual cases rather than relearning exact-match behavior (Gao et al., 2020).

Taken together, these examples show that “residual embedding” and “inverted residual embedding” should not be collapsed into a single category. The MobileNetV2-derived meaning is architectural: shortcut connections are between thin bottleneck layers, with expressive computation moved into a wider intermediate space. Other residual embeddings are score-space, representation-space, or domain-adaptation residuals. This suggests that the most precise use of “Inverted Residual Embedding” is for MobileNetV2-style embedding operators and for direct descendants such as the EfficienT-HDR frontend, rather than for residual mechanisms in general.