Partial GroupMax Network

• Partial GroupMax Network is a neural architecture that approximates functions exhibiting partial convexity by enforcing convexity on selected input variables while modeling complex dependencies on others.
• It integrates a nonconvex feedforward track with a convex GroupMax path that uses max-affine operations to guarantee convexity and support theoretical universal approximation.
• Applications include truthful mechanism design, optimal pricing, and stochastic optimization, underscoring its practical impact in economics and machine learning.

A Partial GroupMax Network is a neural network architecture designed to approximate functions that are convex with respect to a selected subset (or “partial” set) of the input variables, while allowing complex, even nonconvex, dependence on the remaining variables. This property, termed “partial convexity,” is especially relevant in applications such as optimal pricing, mechanism design, stochastic optimization, and conditional function approximation, where theoretical and practical requirements often dictate convexity only along certain input directions. The Partial GroupMax Network has recently found practical application in learning truthful economic mechanisms without discretization and is supported by a universal approximation theorem for partial convex functions.

1. Architectural Foundations and Partial Convexity

The Partial GroupMax Network is constructed by extending standard feedforward and max-affine networks to respect convexity with respect to a designated subset of the inputs. For an input partitioned as $x = (\tilde{x}, y)$, the function to be approximated is required to be convex in $y$ for any fixed $\tilde{x}$. The architecture splits the computation into two tracks:

• Nonconvex Track: Processes $\tilde{x}$ using a general feedforward network with arbitrary (e.g., ReLU) nonlinearities, capturing nonconvex features.
• Convex Track (GroupMax Path): Processes $y$ through layers that combine parameterized affine transformations (whose parameters may depend nonlinearly on $\tilde{x}$) with a “GroupMax” operation, which takes the maximum over partitions (“groups”) of the hidden units.

Within each layer $i$, the network computes hidden states according to recursive relations, such as: $$z_{i+1} = \rho( [W^{(z)}_{i+1} \otimes (W^{(zu)}_{i+1} u_i + b^{(z)}_{i+1})]^+ z_i + W^{(y)}_{i+1} (y \circ (W^{(yu)}_{i+1} u_i + b^{(y)}_{i+1})) + W^{(u)}_{i+1} u_i + b_{i+1} )$$ where $u_i$ represents the output of the nonconvex track at layer $i$, “$\otimes$” denotes elementwise scaling, and “$\circ$” the Hadamard product. This design ensures that the output remains convex in $y$ even when the parameters are conditioned on the nonconvex features $u_i$.

In the final layer, the output $h^\theta(\tilde{x}, y)$ is constructed as a maximum over (possibly many) affine functions of $y$, whose coefficients are determined by (possibly nonlinear) transformations of $\tilde{x}$. This conditional max-affine structure guarantees convexity in $y$ for arbitrary $\tilde{x}$.

2. Universal Approximation Properties

A central theoretical result for the Partial GroupMax Network is a universal approximation theorem, extending the classical result for convex functions to the case of partial convexity. Specifically, for any function $f(\tilde{x}, y)$ that is continuous in $(\tilde{x}, y)$ and convex in $y$ on a compact domain, there exists a sequence of Partial GroupMax Networks $h^\theta(\tilde{x}, y)$ such that: $$\sup_{(\tilde{x}, y) \in K} |f(\tilde{x}, y) - h^\theta(\tilde{x}, y)| \leq \epsilon$$ for any $\epsilon > 0$ and compact $K$, with $h^\theta$ convex in $y$ for each fixed $\tilde{x}$.

The proof leverages the fact that any continuous convex function can be uniformly approximated by maxima over finitely many affine functions (“cuts”), and that the GroupMax recursion effectively grows the number of representable cuts exponentially with network depth and width.

3. Training Procedure and Optimization Techniques

Training Partial GroupMax Networks employs standard backpropagation with additional considerations to preserve the partial convexity property and handle conditional parameterization:

• Activation Functions: The GroupMax operator is differentiable almost everywhere (via max), which allows for gradient-based optimization. Care must be taken at points of nondifferentiability (maxima ties); in practice, automatic differentiation libraries handle subgradients at these points.
• Parameterization: Affine transform parameters for the convex block can be nonlinear functions of $\tilde{x}$. The convexity constraint in $y$ is enforced by imposing positivity (e.g., via softplus) on the coefficients multiplying $y$ after conditioning.
• Loss Functions: For regression, standard MSE losses are used, while in applications such as mechanism design (e.g., TEDI), the expected utility objective involves sampling and integration over (possibly continuous) action spaces.
• Gradient Estimation with Covariance Trick: In settings where the network is used inside mechanisms involving optimization (e.g., argmax of utility), unbiased gradient estimates are obtained using the covariance trick, adding a covariance correction term when differentiating through expectations involving the network output.
• Continuous Sampling via Langevin Dynamics: To efficiently sample from distributions defined by network outputs (especially when replacing softmax over large discrete spaces), approximate samples are obtained via Langevin dynamics, updating at every optimization step.

4. Empirical Performance and Comparative Results

Extensive experiments validate the efficacy of Partial GroupMax Networks in various settings:

• Function Approximation: The architecture effectively approximates convex and partially convex functions, often outperforming or matching specialized architectures such as Input Convex Neural Networks (ICNN) and classical feedforward networks in preserving convexity and accuracy.
• Mechanism Design: Within the TEDI algorithm, Partial GroupMax Networks parameterize buyer pricing rules that must be convex in the buyer’s own allocation (to ensure truthfulness). Empirical evaluations in auction settings demonstrate that mechanisms trained with this network are both expressive and competitive—often surpassing alternatives (including discretization-based approaches and ablations employing other convexity-enforcing networks).
• Computational Efficiency: The architecture allows for learning pricing rules and mechanisms on higher-dimensional inputs without the exponential blowup associated with discretization.

A table summarizing experimental features:

Feature	Partial GroupMax Network	ICNN / PICNN	Standard MLP
Ensures partial convexity	Yes	Yes	No
Universal approximation	Yes	Yes	Yes (not restricted)
Suitable for menu mechanisms	Yes	Yes	No (for truthfulness)

5. Core Applications and Significance

The principal application of Partial GroupMax Networks, as of the most recent literature, is the learning of truthful mechanisms—specifically, pricing rules for menu-mechanism representations in auctions and market design. The architecture enforces the key convexity constraints required by incentive compatibility (Rochet’s theorem), meaning that, for each player, the pricing function in their allocation variable is convex and satisfies relevant boundary conditions.

Other applications include:

• Stochastic Optimization and Control: Approximating convex cost-to-go or value functions that may exhibit partial convexity.
• Dynamic Programming: Representing BeLLMan functions in high-dimensional spaces, where explicit cut-generation or polyhedral approximations are impractical.
• Automated Mechanism Design: As showcased in TEDI, learning optimal, expressive, and dimension-insensitive direct mechanisms without discretizing the outcome space.

6. Comparative Analysis and Limitations

Compared to other convexity-preserving networks (such as ICNN, PICNN, or parameterized max-affine networks), the Partial GroupMax Network offers several advantages:

• Structural Guarantee: Convexity is preserved by construction through the group-wise max operation and positive affine parameterization.
• Expressivity: The conditional generation of cuts (affine approximations) enables the modeling of complicated dependency on nonconvex features while preserving convexity where mandated.
• Scalability: Avoids the curse of dimensionality inherent to outcome discretization.

A potential limitation, as identified in both theoretical and empirical analysis, is that while the network can approximate any partial convex function arbitrarily well, its interpolation error may be slightly higher in pure function regression tasks (without convexity constraints) than generic (non-convex) deep networks. However, in domains where convexity is required, this is not a drawback but a necessary property.

7. Future Directions and Open Questions

Areas identified for further investigation include:

• Systematic Hyperparameter Optimization: Determining optimal depth, width, and group size to efficiently trade representational richness against computational cost.
• Advanced Partial Convexity Modes: Extending architectures to more nuanced forms of convexity (such as blockwise or piecewise convexity) or adapting to alternating convex-concave structures required by broader classes of mechanism design or control problems.
• Integration with Other Learning Architectures: Combining the explicit cut structure of GroupMax networks with regularization techniques or hybrid models to further enhance approximation accuracy while retaining convexity.
• Robustness Analysis: Studying training stability, convergence, and sensitivity to input distribution shifts, particularly in large-scale applications.

In summary, the Partial GroupMax Network is an architecture characterized by its ability to represent any function that is convex in chosen input directions, with a conditional dependency on the remaining variables, supporting both theoretical and practical requirements across optimization, economics, and machine learning. Its recent deployment in automated truthful mechanism design underscores both its practical utility and the growing importance of structure-preserving function approximation in modern computational frameworks.