Product-Unit Residual Blocks (PURe)

• Product-Unit Residual Blocks are neural modules that replace additive interactions with learned multiplicative ones to model high-order feature relationships.
• They leverage a log-space implementation to stabilize computation and omit post-addition activations to maintain gradient flow.
• Empirical evaluations on Galaxy10, ImageNet, and CIFAR-10 demonstrate faster convergence, improved robustness, and competitive accuracy with fewer parameters.

Product-Unit Residual Blocks (PURe) are a class of neural network modules that integrate product units—neurons computing multiplicative interactions between features—into the residual learning framework. Developed to improve both the expressiveness and parameter efficiency of deep convolutional networks, PURe blocks substitute the conventional summation operations in residual blocks with units capable of directly modeling high-order feature interactions. This design enables deep models to represent complex patterns with fewer layers and parameters, enhancing both performance and robustness across challenging computer vision benchmarks (Li et al., 7 May 2025).

1. Architectural Principles

The Product-Unit Residual block departs from the design of the canonical two-layer ResNet basic block. While the first layer remains an ordinary $3\times 3$ convolution followed by Batch Normalization (BN), the second layer is replaced with a two-dimensional product unit (ConvPU), also followed by BatchNorm, and crucially, omits the post-addition activation present in classic residual blocks. The precise computational sequence is:

• Input $x$ undergoes convolution and BN
• Output is passed to a product-unit–based convolution (ConvPU), then BN
• The residual connection is added (with identity or $3\times 3$ Convᵈ as needed for channel or spatial downsampling)
• No activation is applied after the addition

This architecture can be represented as:

$w(m,n)$6

This structure eliminates the ReLU nonlinearity after the final addition, preserving the identity mapping and promoting stable gradient propagation (Li et al., 7 May 2025).

2. Mathematical Formulation of the 2D Product Unit

The defining operation of a product-unit layer is a learned, exponent-weighted multiplicative interaction over the input feature map within a convolutional kernel. For input $x : \mathbb{Z}^2 \rightarrow \mathbb{R}$ and learned exponents $w(m,n)$ over a $(2k+1)\times(2k+1)$ window, the product unit produces: $$y(i,j) = \prod_{m=-k}^{k} \prod_{n=-k}^{k} [x(i+m, j+n)]^{w(m,n)}$$ To circumvent numerical instability and optimize computation, this is implemented in log-space: $$y(i,j) = \exp\left( \sum_{m=-k}^k \sum_{n=-k}^k w(m,n) \cdot \log\operatorname{clamp}(x(i+m,j+n), \epsilon) \right)$$ where $\epsilon$ is a small, learnable positive constant ($$\epsilon = \operatorname{softplus}(\theta) + 10^{-7}$$ per output channel) ensuring strictly positive arguments (Li et al., 7 May 2025).

3. Residual Connections and Nonlinearity Handling

In PURe, the two-layer mapping is denoted $x$0, resulting in the block output $x$1. The design omits a nonlinear activation (such as ReLU) after the second layer for two reasons: (i) the product-unit already introduces strong nonlinearity, and (ii) inserting ReLU at this stage can cause collapsed gradients due to the nature of multiplicative couplings. The absence of an activation preserves the identity shortcut, further stabilizing gradient flow within deep networks (Li et al., 7 May 2025).

4. Parameter and Computational Complexity Analysis

A direct comparison highlights the efficiency of PURe blocks relative to traditional ResNet blocks with the same input ($x$2), output ($x$3), and intermediate ($x$4) channels, and kernel size $x$5. The following table summarizes the block-level resource requirements:

Block Type	Parameters	FLOPs per output pixel
ResNet Basic Block	$x$6	$x$7
PURe Block	$x$8	$x$9 (logs) $3\times 3$0 (log-domain) $3\times 3$1 (exps)

In practical regimes ($3\times 3$2, $3\times 3$3), the additional terms for the product unit are negligible compared to the dominant operations. The parameter counts for the main ConvPU (second layer) match those of the standard convolutional counterpart, plus one trainable threshold per output channel (Li et al., 7 May 2025).

5. Expressiveness and Representational Power

Product units implement monomial mappings: each output is a weighted product of input features. This enables a single PURe block to express high-order polynomial interactions that would otherwise require stacking multiple (linear+ReLU) convolutional layers. The log-exp implementation preserves the essential polynomial expressiveness while maintaining computational tractability. Multiplicative couplings confer additional invariances, such as ratio-invariant or "light-invariant" responses, which are impractical to model with standard convolutional architectures unless very deep or highly overparameterized. Empirical ablation studies on Galaxy10 and CIFAR-10 datasets confirm that replacing only the second convolution with ConvPU improves accuracy with minimal increase in parameter count, evidencing enhanced representational power per parameter (Li et al., 7 May 2025).

6. Empirical Benchmarking

Comprehensive evaluation of PURe models on Galaxy10 DECaLS, ImageNet, and CIFAR-10 demonstrates robust gains in performance, convergence, and noise resilience:

• Galaxy10 DECaLS: PURe34 ($3\times 3$4M params) achieved $3\times 3$5 best test accuracy, outperforming much deeper ResNet152 ($3\times 3$6M, $3\times 3$7), and converged to $3\times 3$8 validation accuracy in $3\times 3$9s versus $x : \mathbb{Z}^2 \rightarrow \mathbb{R}$0s for ResNet152. Accuracy drop under Poisson noise was smaller for PURe34 ($x : \mathbb{Z}^2 \rightarrow \mathbb{R}$1 vs. $x : \mathbb{Z}^2 \rightarrow \mathbb{R}$2).
• ImageNet: PURe34 ($x : \mathbb{Z}^2 \rightarrow \mathbb{R}$3M) attained $x : \mathbb{Z}^2 \rightarrow \mathbb{R}$4 top-1 and $x : \mathbb{Z}^2 \rightarrow \mathbb{R}$5 top-5 accuracy, exceeding ResNet50 ($x : \mathbb{Z}^2 \rightarrow \mathbb{R}$6M; $x : \mathbb{Z}^2 \rightarrow \mathbb{R}$7, $x : \mathbb{Z}^2 \rightarrow \mathbb{R}$8) and slightly surpassing ResNet101 ($x : \mathbb{Z}^2 \rightarrow \mathbb{R}$9M; $w(m,n)$0, $w(m,n)$1) while employing fewer parameters and resources.
• CIFAR-10: PURe272 ($w(m,n)$2M) recorded $w(m,n)$3 test accuracy, matching the much larger ResNet1001 ($w(m,n)$4M; $w(m,n)$5).

These results support the conclusion that PURe models not only achieve competitive accuracy with reduced depth and parameter count, but also demonstrate accelerated convergence and heightened robustness to signal-dependent noise (Li et al., 7 May 2025).

7. Significance and Implications

The introduction of product-unit residual blocks establishes a viable alternative to additive convolutional models in modern deep networks. By enabling multiplicative feature interactions within the residual learning paradigm, PURe architectures facilitate efficient learning and robust generalization with constrained computational resources. This suggests potential for scalable and reliable deep learning models in scenarios constrained by parameter or computational budgets. The observed robustness to noise and accelerated training dynamics highlight product-unit–based architectures as promising candidates for broad deployment in computer vision and related fields requiring high performance under challenging conditions (Li et al., 7 May 2025).