Unidirectional Ordering-based Smoothing (UOS)

• UOS is a technique that enforces fixed, unidirectional ordering in neural architectures to preserve early latent features while reducing over-smoothing and spectral bias.
• In graph neural networks, UOS partitions node embeddings by hop distance and uses soft gating (cumax/SOFTOR) to selectively update new information.
• For neural kernel compression, UOS applies a discrete permutation along filter indices, enabling faster MLP convergence and improved parameter efficiency.

Unidirectional Ordering-based Smoothing (UOS) is a structural and algorithmic technique for imposing order and controlled information flow in neural architectures. It enforces that representations are composed or aggregated in a fixed, one-way order—most notably by aligning the ordering of latent components (neurons or tensor coordinates) with relevant structural or semantic hierarchies (such as graph hop distance or neural kernel filter index). UOS addresses inductive bias and over-smoothing challenges in graph neural networks (GNNs) and, in a complementary application, suppresses spectral bias in neural representations for neural networks (NeRN) acting as parameter-efficient function approximators for convolutional kernel weights. It has demonstrated state-of-the-art accuracy and convergence in both GNN message passing and model compression applications (Song et al., 2023, Xie et al., 9 Sep 2025).

1. Conceptual Motivation and Background

UOS was first introduced to resolve two persistent difficulties in deep graph learning and neural representations: over-smoothing and heterophily in GNNs, and spectral bias in MLP-based NeRN kernel reconstruction.

In standard GNN architectures, all neighbor messages are repeatedly aggregated into the same latent vector space through stacking $K$ message passing layers. As depth increases, node embeddings from different classes become indistinguishable—a phenomenon known as over-smoothing. Moreover, if the local graph is heterophilic, naïve aggregation mixes features from dissimilar classes, reducing predictive performance (Song et al., 2023).

Separately, in implicit neural representations of model weights, coordinate-based MLPs exhibit pronounced spectral bias: they fit low-frequency signal components rapidly, while high-frequency details are learned slowly or omitted, leading to lossy kernel reconstructions and subpar parameter compression (Xie et al., 9 Sep 2025).

UOS addresses these issues by enforcing a fixed, unidirectional alignment: latent variables (neurons, kernel entries) are explicitly ordered so that only specific sub-blocks absorb new information at each stage, preserving locality and suppressing unwanted cross-talk across hops or axes.

2. Mathematical Formulation

2.1 In Graph Neural Networks

Given a graph $\mathcal{G}=(V,E)$ and $D$-dimensional node embeddings $h_v^{(k)}$, UOS partitions each $h_v^{(k)}$ into contiguous neuron blocks corresponding to information from specific hop distances. At each GNN layer $k$:

• The rightmost block, indices $\ge P_v^{(k)}$, is updated with aggregated messages $m_v^{(k)}$ from $k$-hop neighbors.
• The left block, $<P_v^{(k)}$, remains fixed, carrying forward prior-hop context.

Mathematically:

$$h_v^{(k)} = \tilde g_v^{(k)} \circ h_v^{(k-1)} + (1 - \tilde g_v^{(k)}) \circ m_v^{(k)}$$

$\tilde g_v^{(k)}\in [0,1]^D$ is a soft step-gating function indicating the division point $P_v^{(k)}$. A right-to-left cumulative softmax ("cumax") parameterizes the gating function, and a per-layer differentiable-OR (SOFTOR) enforces monotonicity:

$$\tilde g_v^{(k)} = \tilde g_v^{(k-1)} + (1 - \tilde g_v^{(k-1)})\circ\hat g_v^{(k)}$$

The sequence $0 = P_v^{(0)} \le \cdots \le P_v^{(K)} = D$ is strictly non-decreasing, guaranteeing older (closer-hop) information is not overwritten by new (farther-hop) context (Song et al., 2023).

2.2 In Neural Representations for Model Compression

In the context of NeRN for CNN kernel reconstruction, UOS applies a discrete permutation $\pi$ to kernel coordinates along a single axis (e.g., filter index) to minimize local variations:

$$\mathcal S_{\rm UOS}^{(l,c)}(\pi) = \sum_{k=1}^{F-1}\|w_{l,\pi(k+1),c} - w_{l,\pi(k),c}\|_2$$

$$\pi_l^* = \arg\min_{\pi\in S_F}\;\sum_{c=1}^C \sum_{k=1}^{F-1} \|w_{l,\pi(k+1),c} - w_{l,\pi(k),c}\|_2$$

This permutation ensures that the kernel weight signal is maximally low-frequency along the selected axis, allowing the MLP to rapidly fit the largest eigenmode of the neural tangent kernel (NTK) and suppress spectral bias (Xie et al., 9 Sep 2025).

3. Algorithmic Workflow and Implementation

Graph Neural Networks

Each layer proceeds by computing neighbor aggregation, determining the split point (with "cumax"), updating the gating vector (SOFTOR), and combining previous and new representations. The gating parameters are learned per node and per hop, ensuring fine-grained adaptivity.

Neural Kernel Compression

For a CNN of $L$ layers, UOS proceeds as follows (Xie et al., 9 Sep 2025):

1. For each layer $l$, compute the greedy permutation $\pi_l$ that minimizes pairwise kernel differences along the filter axis.
2. Apply $\pi_l$ to permute kernel parameter coordinates before feeding them to the MLP.
3. Encode coordinates with a Random Fourier Feature map, where the kernel (bandwidth) parameter $\sigma_\ell$ is set adaptively:

$\sigma_\ell = \sigma_0/\sqrt{p_\ell}$

with $p_\ell$ the number of parameters in layer $\ell$.

1. At inference, the outputs are un-permuted to recover original ordering.

The computational cost of the permutation is $O(F_\ell^2 C_\ell)$ per layer and is trivial for common architectures; lookup and projection overheads are negligible at runtime.

Application Domain	Ordering Axis	Gating/Permutation Mechanism
GNN Message Passing	Neuron index (hop distance)	Soft gating with cumax/SOFTOR
CNN Kernel Compression	Filter index (single dimension)	Greedy permutation for smoothness

4. Impact on Smoothness, Spectral Bias, and Over-smoothing

UOS fundamentally alters the spectral structure of the composed representations:

• GNNs: By freezing early-hop blocks, node embeddings retain distinct "ego" features regardless of depth, directly preventing classic Laplacian over-smoothing. The per-neuron, per-hop gating schedule enables sharp control over the passage and mixing of contextual signals—a property vital for robust performance on heterophilic graphs. Empirically, UOS-augmented GNNs maintain or improve accuracy as depth increases to 32–64 layers, outperforming both homophily- and heterophily-specialized baselines (Song et al., 2023).
• Neural Kernel Compression: UOS-ordered kernel signals exhibit maximal low-frequency energy concentration. As a result, MLP-based reconstructions converge faster, with higher accuracy and reduced spectral bias. Ablation studies confirm that unidirectional filter-wise permutation outperforms multi-directional and cosine-similarity based permutations, yielding both superior kernel-MSE and classification accuracy under strong compression (e.g., $91.50\%$ at $35\%$ compression ratio on CIFAR-10) (Xie et al., 9 Sep 2025).

5. Comparison with Conventional Approaches

Conventional GNNs and coordinate-based MLPs aggregate and mix all available signals irrespective of their structural provenance, leading to a loss of locality, exacerbation of over-smoothing, and high-frequency attenuation consistent with spectral bias:

• Standard GNN: Aggregates all neighbor hops throughout the entire embedding vector, conflating multi-scale structure and losing distinguishability in deep models.
• NeRN Compression without UOS: Random or multi-directional permutations, or no permutation at all, leaves high-frequency kernel components poorly represented and slows MLP convergence.

By contrast, UOS mechanisms:

• Restrict the mixing or fitting of new information to only a suffix block, preserving early components.
• Provide per-node and per-layer adaptivity for GNNs.
• Align permutation and encoding bandwidths to the actual smoothness and parameter count for kernels in compression regimes.

6. Empirical Evidence and Ablation Results

Substantial empirical results confirm the effectiveness of UOS (Song et al., 2023, Xie et al., 9 Sep 2025):

• GNNs: Ordered GNN/UOS achieves state-of-the-art classification on both homophilic and heterophilic benchmarks (Cora, Citeseer, PubMed, Texas, Cornell, Squirrel, Chameleon), with accuracy stable even at previously prohibitive depths.
• Kernel Compression: SBS (UOS + UOS-RFF) consistently outperforms prior baselines, including NeRN, multi-directional ordering, and sinusoidal/learned positional encodings. Convergence rates are markedly improved, and top-1 accuracy is systematically higher across CIFAR-10, CIFAR-100, and ImageNet.
• Ablations: Cross-paper of {none, UOS} × {PE, RFF} configurations confirms that the best performing regime is unidirectional UOS+RFF. Optimal RFF bandwidth (σ) empirically decreases with increasing parameter count, as predicted by NTK theory.

7. Practical Considerations and Extensions

UOS introduces minimal computational or memory overhead:

• The required permutation indices account for less than 4% of the MLP parameters in typical compression use cases.
• The sole pre-processing cost is the $O(F^2C)$ construction of the permutation; runtime overheads are dominated by standard neural network operations.
• The method strictly requires unidirectional (single-axis) permutation for maximal eigenvalue concentration; multi-directional variants are less effective, as demonstrated in extensive ablations (Xie et al., 9 Sep 2025).
• UOS is compatible with any coordinate encoder (e.g., sinusoidal or learned embeddings); however, Random Fourier Features with bandwidth adaptation are empirically strongest.
• In GNNs, UOS modifies only the combine step and is plug-and-play with existing architecture families (GCN, GAT, SAGE), as well as with signed or neural-attention messages.

A plausible implication is that analogous unidirectional ordering strategies may yield further gains in other hierarchical or structured neural modeling tasks by aligning latent axes with task-pertinent semantics and controlling the spectral concentration of both learned signals and function approximators.