---
title: Unidirectional Ordering-based Smoothing (UOS)
url: https://www.emergentmind.com/topics/unidirectional-ordering-based-smoothing-uos
type: topic
---

# Unidirectional Ordering-based Smoothing (UOS)

Unidirectional Ordering-based Smoothing (UOS) is a structural and algorithmic technique for imposing order and controlled information flow in neural architectures. It enforces that representations are composed or aggregated in a fixed, one-way order—most notably by aligning the ordering of latent components (neurons or tensor coordinates) with relevant structural or semantic hierarchies (such as graph hop distance or neural kernel filter index). UOS addresses inductive bias and over-smoothing challenges in graph neural networks (GNNs) and, in a complementary application, suppresses spectral bias in neural representations for neural networks (NeRN) acting as parameter-efficient function approximators for convolutional kernel weights. It has demonstrated state-of-the-art accuracy and convergence in both GNN message passing and model compression applications [2302.01524] [2509.07373].

## 1. Conceptual Motivation and Background

UOS was first introduced to resolve two persistent difficulties in deep graph learning and neural representations: over-smoothing and heterophily in GNNs, and spectral bias in MLP-based NeRN kernel reconstruction.

In standard GNN architectures, all neighbor messages are repeatedly aggregated into the same latent vector space through stacking $K$ message passing layers. As depth increases, node embeddings from different classes become indistinguishable—a phenomenon known as over-smoothing. Moreover, if the local graph is heterophilic, naïve aggregation mixes features from dissimilar classes, reducing predictive performance [2302.01524].

Separately, in implicit neural representations of model weights, coordinate-based MLPs exhibit pronounced spectral bias: they fit low-frequency signal components rapidly, while high-frequency details are learned slowly or omitted, leading to lossy kernel reconstructions and subpar parameter compression [2509.07373].

UOS addresses these issues by enforcing a fixed, unidirectional alignment: latent variables (neurons, kernel entries) are explicitly ordered so that only specific sub-blocks absorb new information at each stage, preserving locality and suppressing unwanted cross-talk across hops or axes.

## 2. Mathematical Formulation

### 2.1 In Graph Neural Networks

Given a graph $\mathcal{G}=(V,E)$ and $D$-dimensional node embeddings $h_v^{(k)}$, UOS partitions each $h_v^{(k)}$ into contiguous neuron blocks corresponding to information from specific hop distances. At each GNN layer $k$:

- The rightmost block, indices $\ge P_v^{(k)}$, is updated with aggregated messages $m_v^{(k)}$ from $k$-hop neighbors.
- The left block, $<P_v^{(k)}$, remains fixed, carrying forward prior-hop context.

Mathematically:

\[
h_v^{(k)} = \tilde g_v^{(k)} \circ h_v^{(k-1)} + (1 - \tilde g_v^{(k)}) \circ m_v^{(k)}
\]

$\tilde g_v^{(k)}\in [0,1]^D$ is a soft step-gating function indicating the division point $P_v^{(k)}$. A right-to-left cumulative softmax ("cumax") parameterizes the gating function, and a per-layer differentiable-OR (SOFTOR) enforces monotonicity:

\[
\tilde g_v^{(k)} = \tilde g_v^{(k-1)} + (1 - \tilde g_v^{(k-1)})\circ\hat g_v^{(k)}
\]

The sequence $0 = P_v^{(0)} \le \cdots \le P_v^{(K)} = D$ is strictly non-decreasing, guaranteeing older (closer-hop) information is not overwritten by new (farther-hop) context [2302.01524].

### 2.2 In Neural Representations for Model Compression

In the context of NeRN for CNN kernel reconstruction, UOS applies a discrete permutation $\pi$ to kernel coordinates along a single axis (e.g., filter index) to minimize local variations:

\[
\mathcal S_{\rm UOS}^{(l,c)}(\pi) = \sum_{k=1}^{F-1}\|w_{l,\pi(k+1),c} - w_{l,\pi(k),c}\|_2
\]
\[
\pi_l^* = \arg\min_{\pi\in S_F}\;\sum_{c=1}^C \sum_{k=1}^{F-1} \|w_{l,\pi(k+1),c} - w_{l,\pi(k),c}\|_2
\]

This permutation ensures that the kernel weight signal is maximally low-frequency along the selected axis, allowing the MLP to rapidly fit the largest eigenmode of the neural tangent kernel (NTK) and suppress spectral bias [2509.07373].

## 3. Algorithmic Workflow and Implementation

### Graph Neural Networks

Each layer proceeds by computing neighbor aggregation, determining the split point (with "cumax"), updating the gating vector (SOFTOR), and combining previous and new representations. The gating parameters are learned per node and per hop, ensuring fine-grained adaptivity.

### Neural Kernel Compression

For a CNN of $L$ layers, UOS proceeds as follows [2509.07373]:

1. For each layer $l$, compute the greedy permutation $\pi_l$ that minimizes pairwise kernel differences along the filter axis.
2. Apply $\pi_l$ to permute kernel parameter coordinates before feeding them to the MLP.
3. Encode coordinates with a Random Fourier Feature map, where the kernel (bandwidth) parameter $\sigma_\ell$ is set adaptively:
   \[
   \sigma_\ell = \sigma_0/\sqrt{p_\ell}
   \]
   with $p_\ell$ the number of parameters in layer $\ell$.
4. At inference, the outputs are un-permuted to recover original ordering.

The computational cost of the permutation is $O(F_\ell^2 C_\ell)$ per layer and is trivial for common architectures; lookup and projection overheads are negligible at runtime.

| Application Domain      | Ordering Axis                   | Gating/Permutation Mechanism           |
|------------------------|---------------------------------|----------------------------------------|
| GNN Message Passing    | Neuron index (hop distance)     | Soft gating with cumax/SOFTOR          |
| CNN Kernel Compression | Filter index (single dimension) | Greedy permutation for smoothness      |

## 4. Impact on Smoothness, Spectral Bias, and Over-smoothing

UOS fundamentally alters the spectral structure of the composed representations:

- **GNNs**: By freezing early-hop blocks, node embeddings retain distinct "ego" features regardless of depth, directly preventing classic Laplacian over-smoothing. The per-neuron, per-hop gating schedule enables sharp control over the passage and mixing of contextual signals—a property vital for robust performance on heterophilic graphs. Empirically, UOS-augmented GNNs maintain or improve accuracy as depth increases to 32–64 layers, outperforming both homophily- and heterophily-specialized baselines [2302.01524].

- **Neural Kernel Compression**: UOS-ordered kernel signals exhibit maximal low-frequency energy concentration. As a result, MLP-based reconstructions converge faster, with higher accuracy and reduced spectral bias. Ablation studies confirm that unidirectional filter-wise permutation outperforms multi-directional and cosine-similarity based permutations, yielding both superior kernel-MSE and classification accuracy under strong compression (e.g., $91.50\%$ at $35\%$ compression ratio on CIFAR-10) [2509.07373].

## 5. Comparison with Conventional Approaches

Conventional GNNs and coordinate-based MLPs aggregate and mix all available signals irrespective of their structural provenance, leading to a loss of locality, exacerbation of over-smoothing, and high-frequency attenuation consistent with spectral bias:

- **Standard GNN**: Aggregates all neighbor hops throughout the entire embedding vector, conflating multi-scale structure and losing distinguishability in deep models.
- **NeRN Compression without UOS**: Random or multi-directional permutations, or no permutation at all, leaves high-frequency kernel components poorly represented and slows MLP convergence.

By contrast, UOS mechanisms:

- Restrict the mixing or fitting of new information to only a suffix block, preserving early components.
- Provide per-node and per-layer adaptivity for GNNs.
- Align permutation and encoding bandwidths to the actual smoothness and parameter count for kernels in compression regimes.

## 6. Empirical Evidence and Ablation Results

Substantial empirical results confirm the effectiveness of UOS [2302.01524, 2509.07373]:

- **GNNs**: Ordered GNN/UOS achieves state-of-the-art classification on both homophilic and heterophilic benchmarks (Cora, Citeseer, PubMed, Texas, Cornell, Squirrel, Chameleon), with accuracy stable even at previously prohibitive depths.
- **Kernel Compression**: SBS (UOS + UOS-RFF) consistently outperforms prior baselines, including NeRN, multi-directional ordering, and sinusoidal/learned positional encodings. Convergence rates are markedly improved, and top-1 accuracy is systematically higher across CIFAR-10, CIFAR-100, and ImageNet.
- **Ablations**: Cross-study of {none, UOS} × {PE, RFF} configurations confirms that the best performing regime is unidirectional UOS+RFF. Optimal RFF bandwidth (σ) empirically decreases with increasing parameter count, as predicted by NTK theory.

## 7. Practical Considerations and Extensions

UOS introduces minimal computational or memory overhead:

- The required permutation indices account for less than 4% of the MLP parameters in typical compression use cases.
- The sole pre-processing cost is the $O(F^2C)$ construction of the permutation; runtime overheads are dominated by standard neural network operations.
- The method strictly requires unidirectional (single-axis) permutation for maximal eigenvalue concentration; multi-directional variants are less effective, as demonstrated in extensive ablations [2509.07373].
- UOS is compatible with any coordinate encoder (e.g., sinusoidal or learned embeddings); however, Random Fourier Features with bandwidth adaptation are empirically strongest.
- In GNNs, UOS modifies only the combine step and is plug-and-play with existing architecture families (GCN, GAT, SAGE), as well as with signed or neural-attention messages.

A plausible implication is that analogous unidirectional ordering strategies may yield further gains in other hierarchical or structured neural modeling tasks by aligning latent axes with task-pertinent semantics and controlling the spectral concentration of both learned signals and function approximators.

Source: https://www.emergentmind.com/topics/unidirectional-ordering-based-smoothing-uos