Permutation Equivariant Training (PET)

• Permutation Equivariant Training (PET) is a framework that defines neural models capable of maintaining structural consistency when processing unordered sets and graphs.
• The approach uses transmission layers and parameter-sharing to achieve universality, as exemplified by architectures like DeepSets and PointNetST.
• Empirical validations show PET models outperform non-equivariant baselines in tasks such as point cloud classification and graph message passing.

Permutation Equivariant Training (PET) refers to the design and training of neural architectures whose outputs transform consistently with permutations applied to their inputs. PET is crucial in domains where the ordering of elements—such as set membership, graph nodes, agents, or feature dimensions—is inherently arbitrary, but computations over those elements must still preserve structural relationships. PET has become foundational across set learning, graph representation, multi-agent systems, quantum learning, normalizing flows, and auction design.

1. Mathematical Foundations of Permutation Equivariance

Permutation equivariant functions satisfy a commutation property with permutation group actions. Precisely, for a permutation group $S_n$ acting on input $X \in \mathbb{R}^{n \times k}$ (as $PX$), a function $f$ is permutation equivariant if: $f(PX) = P f(X), \quad \forall P \in S_n$ For higher-order structures (e.g., matrices, tensors), equivariance generalizes to simultaneous row and column permutations, so for $A \in \mathbb{R}^{n \times n}$: $f(PAP^T) = P f(A) P^T$ Universal approximation of such functions was initially unresolved for practical architectures. Classical results established that DeepSets and PointNet are universal for permutation-invariant functions, but universality for equivariant functions required new characterizations (Segol et al., 2019).

2. Theory and Universality of Equivariant Architectures

The universality of neural set architectures for equivariant functions is established through algebraic characterization of all equivariant polynomials:

• Main Theorem: Every equivariant polynomial $$f:\mathbb{R}^{n \times k} \to \mathbb{R}^{n \times l}$$ has the form

$$f(X) = \sum_{\alpha:|\alpha| \leq n} v_\alpha q_\alpha^T$$

where $v_\alpha$ comprises per-row monomials, and $q_\alpha$ are functions of power-sum multisymmetric polynomials $s_j(X) = \sum_{i=1}^n x_i^{\alpha_j}$.

• DeepSets and a modified PointNet (PointNetST: adds a linear transmission layer, $X \mapsto \mathbf{1}\mathbf{1}^T X$) are shown to be universal for continuous permutation equivariant functions (Segol et al., 2019).
• Transmission layers (aggregating over the set) are necessary: PointNet without transmission is provably non-universal for equivariant maps.
• Minimal width bounds for universality are established, and all necessary parameter-sharing structures are determined from a combinatorial analysis.

3. Practical Architecture Design: Transmission and Parameter Sharing

The explicit decomposition of permutation equivariant polynomials guides the construction of efficient, universal PET models:

• DeepSets architecture: $f(X)_i = g(x_i) + h(\sum_j x_j)$, with $g$, $h$ as MLPs.
• PointNetST: A standard PointNet ($g(x_i)$), augmented with a linear transmission:

$$\text{transmission layer:} \quad X \mapsto \mathbf{1}\mathbf{1}^T X$$

• Higher-order architectures with explicit higher-order transmission achieve universality for tensor-valued and relation-valued outputs, but at increased computational cost.
• The explicit characterization yields layers (polynomial, or implemented as ReLU networks) that are both theoretically universal and efficiently implementable—no requirement for high-order tensors or group sorting.

4. Empirical Validation and Model Comparisons

Numerical experiments across a taxonomy of tasks demonstrate the practical implications:

• Classification (e.g., knapsack problem), regression (quadratic, spectral functions) on sets/point clouds.
• Models compared: DeepSets, PointNet, PointNetST, PointNetSeg, PointNetQT (quadratic transmission), GraphNet (message-passing), and MLP baselines.
• Findings:
  • Universal equivariant models (DeepSets, PointNetST, PointNetSeg, PointNetQT) perform comparably and at state-of-the-art level on all benchmarks, matching theoretical predictions.
  • PointNet (non-universal) consistently underperforms, especially for complex tasks—demonstrating the necessity of the transmission layer for expressive PET.
  • Baseline MLPs with matching parameter count generalize poorly, as they cannot leverage permutation equivariance.
  • GraphNet achieves strong, but generally slightly weaker, results compared to DeepSets/PointNetST.
  • DeepSets can approximate message-passing (graph convolution) layers, indicating broad expressive power for equivariant tasks.

Model	Universality	Performance	Parameter Efficiency
DeepSets	Yes	SOTA	High
PointNet	No	Lower	High
PointNetST	Yes (simple)	SOTA	Highest
Baseline MLP	No (PE)	Poor	Low

5. Implications for PET in Broader Settings

Practical guidance:

• Architectural minimalism: A single linear transmission layer suffices for universality in equivariant settings; additional complexity (high-order tensors etc.) is unnecessary for most set-based tasks.
• PET model selection: For tasks requiring permutation equivariance, DeepSets or PointNetST architectures are preferred due to universality and implementation simplicity.
• Comparative expressiveness: PET-universal architectures yield models whose performance matches or exceeds that of more complex equivariant models, with lower sample complexity and improved generalization.
• Layer design: The explicit structure enables theoretically founded parameter sharing and tractable construction for layers, avoiding redundancy.

6. Mathematical Tools for Analysis and Design

The explicit structure of permutation equivariant networks is grounded in:

• Multisymmetric polynomials: All permutation invariant polynomials are generated by power-sum multisymmetric polynomials.
• Density arguments: Continuous equivariant functions are approximable by polynomial models constructed from the above.
• Minimal universality width (with ReLU):

$$\omega \leq k_{\mathrm{out}} + k_{\mathrm{in}} + \binom{n+k_{\mathrm{in}}}{k_{\mathrm{in}}}$$

where $n$ is set size, $k_{\mathrm{in}}, k_{\mathrm{out}}$ are input/output dimensions.

7. PET: Analysis, Limitations, and Future Directions

• Advantages: PET enforces desirable symmetries, improves sample efficiency, simplifies learning, and enables provably universal, robust architectures for set-structured data.
• Limitations: The analysis focuses on continuous set-valued mappings; composition with positional encodings, or non-set-structured data, requires care.
• Connection to Related Work: Theoretical tools draw from classical algebraic invariance ([Briand, Golubitsky & Stewart]), recent set network studies ([Zaheer et al. 2017], [Keriven et al. 2019], [Sannai et al. 2019]), and universal approximation theory.
• Practical extension: Variants with higher-order transmission allow for even richer function spaces but at computational cost.

---

Permutation Equivariant Training underpins universal, efficient, and theoretically principled learning on sets and other arbitrarily ordered data. By characterizing all equivariant polynomial maps and providing efficient universal architectures (DeepSets, PointNetST), PET facilitates both state-of-the-art empirical performance and strong theoretical guarantees for symmetry-respecting machine learning (Segol et al., 2019).