INRe Basic Block

• INRe Basic Block is a hybrid neural network component that unifies 1×1 convolutions and MLP layers through a compression–expansion bottleneck with GELU activations.
• It employs an interleaved Conv–MLP architecture that alternates convolution and MLP operations to achieve superior parameter efficiency and improved performance in tasks like video fitting.
• Its design supports both low-level signal fitting and high-level semantic tasks by leveraging non-uniform stacking strategies and intermediate supervision for effective representation learning.

An INRe basic block refers to a modular network component that unifies convolutional and multilayer perceptron (MLP) operations under a compression–expansion bottleneck and GELU activations, designed as the principal building unit of Implicit Neural Representation Networks (INRN). These blocks are formulated to bridge low-level signal parameterization and high-level semantic vision tasks, addressing representational and depth limitations in prior implicit neural representation (INR) approaches (Song et al., 2022).

1. Structural Composition

An INRe basic block is a hybrid sub-network comprising three principal elements:

1. Hybrid Convolution + MLP Architecture: Each block alternates $1\times 1$ convolutions and MLP layers, embedding spatial inductive bias and locality alongside expressive nonlinear channel mixing.
2. Compression–Expansion Bottleneck: MLP layers are implemented in a two-layer bottleneck configuration, with a compression ratio $\alpha < 1$ to first reduce, then expand the input features, optimizing parameter efficiency and empirical accuracy.
3. GELU Activation: The Gaussian Error Linear Unit (GELU) is applied in all nonlinearities, favoring smooth gradients and mitigating dead neurons compared to ReLU.

The block operates on a $H\times W\times C_{\text{in}}$ input tensor, producing a $H\times W\times C_{\text{out}}$ output.

2. Internal Forward Pathways

There are two main architectural sequencing variants for processing input through an INRe block:

• Front-Conv MLP ("Front-conv + MLP"): Applies a $1\times 1$ convolution, flattens spatial dimensions, and passes the result through a single MLP.
• Interleaved Conv–MLP (used in principal experiments): Alternates $1\times 1$ convolutions and MLPs, such as

$$z^{(0)} = h^{\mathrm{in}} \xrightarrow{\mathrm{Conv}_{1\times1}} z^{(1)} \xrightarrow{\mathrm{MLP}_1} z^{(2)} \xrightarrow{\mathrm{Conv}_{1\times1}} z^{(3)} \xrightarrow{\mathrm{MLP}_2} h^{\mathrm{out}}$$

The interleaved variant demonstrates a superior balance of representational power and parameter efficiency, as shown by significant PSNR gains in video fitting tasks (Song et al., 2022).

3. Compression–Expansion MLP Design

Each MLP layer within the block is formulated as:

$$\mathrm{MLP}_k(u) = W^{(k)}_{\mathrm{exp}}\, \phi\left(W^{(k)}_{\mathrm{comp}} u + b^{(k)}_{\mathrm{comp}}\right) + b^{(k)}_{\mathrm{exp}},\quad k=1,2$$

with $\phi$ the GELU function, compression weights $$W^{(k)}_{\mathrm{comp}}\in{\mathbb{R}}^{d_{\mathrm{mid}}\times d_{\mathrm{in}}}$$ where $\alpha < 1$0, and expansion weights $\alpha < 1$1. The bottleneck drastically improves parameter efficiency while maintaining, or improving, classification accuracy on ImageNet (Song et al., 2022).

4. Stacking Strategies: Single-Stage and Multi-Stage

The INRe basic block underpins two network scaling paradigms:

• Single-Stage ("Pipeline") Stacking: Used for tasks such as image fitting. $\alpha < 1$2 INRe blocks are stacked sequentially with only the final output supervised, minimizing a joint $\alpha < 1$3/SSIM loss:

$\alpha < 1$4

• Multi-Stage (with Distillation) Stacking: Employed for high-level tasks (classification, detection, segmentation), dividing $\alpha < 1$5 blocks into $\alpha < 1$6 stages. Each stage's outputs undergo intermediate supervision using stage-wise distillation:

$\alpha < 1$7

and the final classification objective combines standard cross-entropy with this intermediate distillation:

$\alpha < 1$8

Non-uniform staging ratios (e.g., $\alpha < 1$9) provide empirically superior top-1 accuracy compared to uniform depths.

5. Empirical Results and Representation Efficiency

Ablation studies report:

• Hybrid Conv+MLP: Substantially outperforms pure-MLP configurations on signal fitting tasks, attributed to convolutional inductive bias.
• Interleaving: Achieves superior or competitive accuracy for a given parameter count due to finer distribution of convolutional and MLP transformations.
• Compression–Expansion: Reduces overall parameters while sometimes improving accuracy; preserves performance on ImageNet.
• GELU Activation: Statistically outperforms ReLU on both low-level and high-level benchmarks, due to smoother gradient properties and reduced risk of inactive units.
• Staging Ratios: Non-uniform block allocation per stage (e.g., $H\times W\times C_{\text{in}}$0) yields 1–2% absolute gain in top-1 accuracy on ImageNet, and marked improvements on detection/segmentation tasks (Song et al., 2022).

6. Broader Significance and Paradigm Unification

The INRe basic block is positioned as a unifying building module for implicit representation paradigms across the vision task spectrum, serving as the first INR-based element scalable to both low- and high-level semantic vision tasks. This modularity simplifies the integration of signal fitting and semantic learning within a single architectural framework, allowing flexible deployment as either stand-alone units or plug-in backbones within deep networks. This approach reformulates the role of implicit neural representations, extending their applicability beyond conventional limitations to encompass the breadth of visual learning pipelines (Song et al., 2022).