HOPSE: Higher-Order Positional & Structural Encoders

• HOPSE is a framework that leverages algebraic and orthogonal operator mappings to encode complex positional and structural relationships in various data domains.
• Its methodologies, including orthogonal polynomial encodings and Hasse-graph decomposition, overcome traditional message-passing limitations while preserving invariants.
• Empirical evaluations on sequences, trees, and topological tasks demonstrate improved convergence, accuracy, and scalability across large combinatorial complexes.

Higher-Order Positional and Structural Encoders (HOPSE) are a class of machine learning architectures designed to encode position and structure in data domains where the underlying geometric or combinatorial relationships go significantly beyond simple sequences or graphs. HOPSE frameworks systematically generalize positional encoding to diverse, higher-order domains (including sequences, grids, trees, graphs, and arbitrary combinatorial complexes), supporting efficient, expressive, and permutation-equivariant representations that preserve global and local structural invariants and can scale to large datasets. They have been developed in response to the documented limitations and scalability barriers of pairwise- or message-passing-centric deep learning models on relational and topological tasks (Carrasco et al., 21 May 2025, Kogkalidis et al., 2023, Wijesinghe et al., 22 Feb 2025, Aggarwal, 2024).

1. Algebraic and Combinatorial Foundations

HOPSE is grounded in an algebraic characterization of the structure's position set and a mapping of this algebra into a family of orthogonal (unitary or permutation) operators:

• For any inductively defined structure, a position is identified with an element of a free algebra $(\mathcal{P}, \circ, e)$ generated by primitive moves (e.g., “step,” “branch”), supporting operations such as composition, inversion, and identity, and potentially monoid or group laws.
• The core mechanism is a homomorphism $\Phi : (\mathcal{P}, \circ, e) \to O(d)$, mapping each syntactic position/composite move into an orthogonal matrix operator. This ensures algebraic invariants—associativity, inversion, identity (and commutativity when present)—are preserved in the embedding computation (Kogkalidis et al., 2023).

This algebraic structure supports the positional encoding and attention mechanisms in Transformer-style and related architectures by providing content-independent, structure-preserving mappings which are easily composable over higher-order domains.

2. Instantiations: Sequences, Grids, Trees, and Complexes

HOPSE provides direct instantiations for a range of structural domains, including:

• Sequences: Classical 1D position where $\mathcal{P} = \mathbb{Z}$, encoded by powers of a single orthogonal generator $W \in O(d): \Phi(p) = W^p$.
• Grids: Multidimensional structures, e.g., 2D grids, represented as direct sums of 1D generators, $\Phi(h, w) = H^h \oplus W^w$ where $H, W \in O(d/2)$.
• Trees and Composites: General $k$-ary trees, $\mathcal{P}$ the free group on $k$ generators, operate through products of multiple orthogonal matrices, e.g., $\Phi(\rho) = W_1^{n_1} \cdots W_k^{n_k}$ for branching.
• Combinatorial Complexes: Arbitrary cellular, simplicial, or incidence-rich complexes, decomposed via Hasse graphs indexed by neighborhood functions, where encodings are constructed for each combinatorial rank (0-cells, 1-cells, etc.) and concatenated or summed according to the targeted cell type (Carrasco et al., 21 May 2025).

3. Encoder Architectures and Methodologies

Several concrete HOPSE methodologies have been introduced:

• Orthogonal Polynomial Encodings: PoPE encodes position using Legendre orthogonal polynomials $\Phi : (\mathcal{P}, \circ, e) \to O(d)$0, exploiting their non-periodicity, orthogonality, and smooth decay of cross-correlations, overcoming failures and inductive biases in traditional sinusoidal or rotary (RoPE) schemes. This approach is especially effective in decoupling spatial and frequency aliasing and supports rapid convergence and improved performance in attention-based sequence models (Aggarwal, 2024).
• Algebraic Operator Encodings: By lifting the domain's algebraic specification into operator groups, the HOPSE framework can inject position/structure into attention by operating on token representations via orthogonal actions, thus preserving path invariants and ensuring structure-equivariant attention weights (Kogkalidis et al., 2023).
• Message-Passing-Free Complex Encoders: For combinatorial complexes, HOPSE leverages Hasse-graph decomposition, where each incidence or adjacency relation is encoded as a simple graph and processed by spectral or random-walk positional encoders (e.g., Laplacian eigenvectors, random-walk statistics), followed by shallow MLP fusion. This design eliminates the super-polynomial overhead of message-passing architectures and grants linear scaling in the number of cells for each combinatorial rank (Carrasco et al., 21 May 2025).
• Decomposition and Aggregation: Positional and structural encodings are aggregated across multiple neighborhoods and cell ranks, using operations such as concatenation or summation, followed by fusion via rank-specific MLPs.

4. Theoretical Properties: Invariance, Expressivity, and Complexity

HOPSE mechanisms guarantee several desirable theoretical properties:

• Permutation Equivariance: Layer composition preserves permutation equivariance across cells of equal rank; any permutation applied to input features is commuted by the encoder (Carrasco et al., 21 May 2025).
• Structural Invariance: Through their algebraic construction, attention and downstream predictions are structure- and path-equivariant, ensuring invariance to permutations, relabelings, or rewrites that preserve the combinatorial or algebraic relations among positions (Kogkalidis et al., 2023).
• Expressivity: Theoretical results demonstrate that HOPSE/GenHopNet models with higher-order aggregation (e.g., $\Phi : (\mathcal{P}, \circ, e) \to O(d)$1-hop, closed-walk statistics, Laplacian eigenvectors) surpass the distinguishing power of standard 1-WL tests, and can match or exceed the power of Combinatorial-Complex WL tests for higher-order complexes (Wijesinghe et al., 22 Feb 2025, Carrasco et al., 21 May 2025).
• Computational Efficiency: Message-passing-free HOPSE scales linearly with the number of cells, as opposed to message-passing schemes whose cost grows factorially or superlinearly with the number and rank of cells. For graphs, spectral encoders may have cubic preprocessing cost (Laplacian eigendecomposition) but admit scalable approximations.

5. Empirical Evaluation and Benchmarks

HOPSE models have been evaluated across a range of domains:

• Sequence Models: PoPE encodings outperform sinusoidal and RoPE baselines in BLEU score and convergence rate on standard translation tasks (e.g., Multi30k EN$\Phi : (\mathcal{P}, \circ, e) \to O(d)$2DE), reaching BLEU of 40.7 vs. 35.6–36.9 for sinusoidal baselines, and converging $\Phi : (\mathcal{P}, \circ, e) \to O(d)$3–$\Phi : (\mathcal{P}, \circ, e) \to O(d)$4 faster (Aggarwal, 2024).
• Synthetic Structural Tasks: Algebraic operator encodings on sequence and tree tasks achieve $\Phi : (\mathcal{P}, \circ, e) \to O(d)$5 accuracy in reverse, copy, repeat, and tree-regression tasks where competing baselines underperform markedly (e.g., sinusoidal encoding $\Phi : (\mathcal{P}, \circ, e) \to O(d)$6 for reverse) (Kogkalidis et al., 2023).
• Topological Learning: On combinatorial complex classification and regression (e.g., ZINC, TUDatasets, MANTRA homology), HOPSE matches or surpasses message-passing baselines while achieving up to $\Phi : (\mathcal{P}, \circ, e) \to O(d)$7 training speedup and reliable preservation of topological invariants (e.g., Betti numbers, topology class distinction) (Carrasco et al., 21 May 2025).
• Graph Self-Supervised Learning: GenHopNet integrates $\Phi : (\mathcal{P}, \circ, e) \to O(d)$8-hop, structural, and positional learning and consistently outperforms the strongest GNN/GSSL baselines on graph classification, regression, and isomorphism tasks, e.g., $\Phi : (\mathcal{P}, \circ, e) \to O(d)$9 on CSL vs. $\mathcal{P} = \mathbb{Z}$0 for 3-WL GNNs (Wijesinghe et al., 22 Feb 2025).
• Encoder Disentanglement: Disentangling positional and semantic streams and reserving a dedicated positional channel preserves positional structure into deep layers, outperforming RoPE and learned entangled AP on Flash-Holmes linguistic probing (winning on 49/65 phenomena) and capturing explicit low-dimensional, low-frequency structural manifolds (Lequeu et al., 28 May 2026).

6. Implications for Architecture Design and Applications

Empirical and theoretical findings from HOPSE research suggest several architectural principles:

• Decoupling: Separating positional/structural substreams from semantic representations elicits clear head specialization (structure vs. semantic attention), preserves global ordering, and mitigates task-induced positional information collapse (Lequeu et al., 28 May 2026).
• Basis Choice: The use of orthogonal, algebraically rich bases (e.g., Legendre polynomials, Laplacian eigenvectors) yields non-periodic, well-conditioned encodings, eliminating aliasing and scaling robustly to long-range dependencies (Aggarwal, 2024, Wijesinghe et al., 22 Feb 2025).
• Compositionality: Block-diagonal composition and direct sum constructs allow transparent extension to multidimensional, hierarchical, and composite domains without ad hoc engineering.
• Permutation and Structure-Invariant Representations: Structural equivariance and preservation of positional relations are ensured by algebraic and homomorphic design, facilitating strong out-of-distribution generalization on tasks involving structural ambiguities and invariants (Kogkalidis et al., 2023, Carrasco et al., 21 May 2025).
• Scalability: Message-passing-free construction via Hasse-graph decomposition and spectral aggregation enables application to large, high-rank combinatorial complexes that are intractable for message-passing GNNs (Carrasco et al., 21 May 2025).

A plausible implication is that HOPSE will continue to provide foundational design blueprints as deep learning extends to more complex data regimes, including multi-modal, hierarchical, and topological domains.

7. Limitations and Directions for Further Research

Known limitations and open questions include:

• Preprocessing Overhead: Some spectral methods require expensive eigendecomposition; recent approaches use pre-trained positional structural encoders (e.g., GPSE) or approximate solvers to mitigate this.
• Hyperparameter and Channel Selection: The selection and tuning of encoding channels (e.g., which spectral features, which polynomial bases) may require domain-specific adaptation and is not yet fully automated or end-to-end trainable (Carrasco et al., 21 May 2025).
• Integration with Message-Passing: Combining HOPSE encodings with higher-order message-passing (HOMP) or Transformer architectures for further expressivity and flexibility is an area of ongoing work.
• Scalability Beyond Small Complexes: Further scaling to dynamic, heterogeneous, or temporally evolving complexes requires advances in online/approximate computation and memory management.

In summary, Higher-Order Positional and Structural Encoders offer a flexible, algebraically principled, and computationally efficient toolkit for representing rich structural and positional information in data, with empirical and theoretical properties that generalize and unify much of the recent progress in positional encoding, graph neural networks, and topological deep learning (Aggarwal, 2024, Kogkalidis et al., 2023, Lequeu et al., 28 May 2026, Wijesinghe et al., 22 Feb 2025, Carrasco et al., 21 May 2025).