Additivity-Constrained Output Layer

• Additivity-Constrained Output Layer is a mechanism that exactly enforces linear equality constraints (e.g., ∑x_i = C) for valid probability and resource allocations.
• It leverages null-space parametrization and direct affine projection to guarantee efficient and exact constraint fulfillment during both forward and backward passes.
• Extensions such as entropy-regularized and box-constrained layers broaden its applications to classification, structured prediction, and optimization tasks.

An additivity-constrained output layer is a neural network output mechanism that enforces strict linear equality constraints—most typically constraints of the form $\sum_i x_i = C$—directly on the output vector. These layers guarantee that network predictions always reside within a specified affine subspace or simplex, addressing requirements for mass conservation, probability normalization, or sum-to-budget constraints in domains such as classification, allocation, and resource management. Unlike penalty-based or soft constraint methods, additivity-constrained output layers enforce these constraints exactly and efficiently in both the forward and backward passes, with several robust computational strategies available depending on the structure of the constraint set.

1. Mathematical Formulation and Motivation

The canonical additivity constraint requires the network output $x \in \mathbb{R}^n$ to satisfy $A x = b$ for a fixed $A \in \mathbb{R}^{m \times n}$, $b \in \mathbb{R}^m$ with $m < n$ and $\operatorname{rank}(A) = m$. The most common instantiation is the sum-to-one (simplex) constraint, enforced by $A = [1, \ldots, 1]$ (a $1 \times n$ row vector), $b = [1]$. This constraint ensures outputs such as class probabilities or resource fractions are valid. The enforcement must be exact, differentiable, and computationally tractable. Applications span from probabilistic classification to structured prediction, allocation, and portfolio optimization (Konstantinov et al., 2023, Zeng et al., 2024, Berzal, 7 Nov 2025).

2. Exact Null-space and Affine Projection Parametrizations

Two equivalent strategies deliver exact satisfaction of linear constraints in output layers, as detailed in (Konstantinov et al., 2023):

• Null-space parametrization: The output vector is expressed as $x = R w + u$ where $R$ spans the null space of $A$ ($A R = 0$), $w \in \mathbb{R}^{k=n-m}$ are unconstrained latent variables, and $u$ is any particular solution to $A u = b$. The mapping $h \mapsto x = R h + u$ is linear, differentiable, and requires only a one-time SVD or null-space computation. Gradients propagate as $dL/dh = R^T (dL/dx)$, so the backward pass matches the forward complexity.
• Direct affine (orthogonal) projection: The output is obtained by projecting an unconstrained vector $y$ onto the affine subspace via $x = y - M (A y - b)$, where $M = A^T (A A^T)^{-1}$. This is the L2-optimal projection onto $A x = b$, with forward and backward passes scaling as $O(n m)$ and requiring only simple matrix-vector products and small matrix inversions.

These methods support arbitrary $A$, and the sum-to-one projection emerges as a special case ($x_i = y_i - (\sum_j y_j - 1)/n$). Implementation in standard frameworks is succinct and requires only inexpensive one-off precomputations (Konstantinov et al., 2023).

3. Entropy-Regularized and Box-Constrained Additivity Layers

When box constraints ($0 \leq y_i \leq u_i$) or additional regularization are required, or for large-scale/differentiable pipeline integration, entropy-regularized projection becomes beneficial. The GLinSAT framework (Zeng et al., 2024) formulates the projection as an entropy-regularized linear program: $$\min_y \ \ -x^T y + \frac{1}{\theta} \sum_{i=1}^n \left[ \frac{y_i}{u_i} \ln \frac{y_i}{u_i} + (1 - \frac{y_i}{u_i}) \ln(1 - \frac{y_i}{u_i}) \right] \qquad \text{s.t.} \quad \sum_i y_i = C,\ 0 \leq y_i \leq u_i$$ The dual variable $\lambda$ associated with the sum constraint leads to a one-dimensional unconstrained convex optimization, solved efficiently per batch element using accelerated gradient descent. The primal solution has elementwise closed form $$y_i^* = u_i \cdot \sigma( \theta u_i (x_i + \lambda^*) )$$ where $\sigma$ denotes the sigmoid. Both forward and backward passes require only $O(n K)$ time per sample, with $K = O(\sqrt{\theta n / \epsilon})$ iterations for convergence, and the backward pass is readily obtained via implicit differentiation.

This construction smoothly interpolates between pure simplex projection ($u_i = 1$, large $\theta$) and softer normalization. Crucially, it enables enforcement of not just additivity but also box constraints in a fully differentiable, GPU-friendly manner (Zeng et al., 2024).

4. Softmax, Simplex, and the Probabilistic Interpretation

In the context of multi-class classification, the most widely used additivity-constrained output is the softmax layer, which ensures $y_i \geq 0$ and $\sum_i y_i = 1$. From a statistical standpoint, this arises naturally as the canonical link for the multinomial Generalized Linear Model, with the logit/softmax transformation producing an output interpretable as a categorical probability distribution (Berzal, 7 Nov 2025). The associated negative log-likelihood (categorical cross-entropy) matches the maximum-likelihood principle.

The softmax layer is a restriction of the general additivity-constrained framework to the probability simplex, $y \in \Delta^{n-1}$. While it guarantees normalization, it does not directly enforce upper bounds beyond one and inevitably distributes mass among all components, sometimes undesirably so for sparse selections or budgeted allocation (Zeng et al., 2024). Compared to the Euclidean projection onto the simplex (which requires sorting), softmax is faster but less expressive in handling generalized box or linear constraints (Zeng et al., 2024, Berzal, 7 Nov 2025).

5. Computational Complexity and Implementation

All major additivity-constrained output architectures provide efficient, scalable forward and backward passes. Null-space and orthogonal projection layers (for generic $A x = b$) achieve $O(n m)$ complexity per instance for both evaluation and gradient propagation, given one-off computation of $R$ and $u$ or $M$ matrices (Konstantinov et al., 2023). The entropy-regularized layer (GLinSAT) delivers $O(K n)$ time per forward, with $K$ sublinear in $(\theta n/\epsilon)^{1/2}$ for typical accuracy $\epsilon$. Importantly, the layers require only elementwise and dot-product operations (no sorting or active-set logic), making them highly amenable to hardware acceleration (Zeng et al., 2024).

In standard frameworks such as PyTorch, both exact (null-space or projection) and entropy-regularized layers are compactly implemented. The backward passes exploit the linearity of the mappings and, for entropy-based approaches, use implicit differentiation based on the stationarity conditions, maintaining $O(n)$ scaling per sample (Konstantinov et al., 2023, Zeng et al., 2024).

6. Practical Use Cases and Comparisons

Additivity-constrained output layers are indispensable in settings requiring explicit conservation or normalization, such as classification (probability outputs), combinatorial optimization (assignment, allocation), portfolio selection, power unit commitment, and constrained route planning (Zeng et al., 2024). Comparison with classic approaches highlights key trade-offs:

Method	Constraint Satisfaction	Complexity
Softmax	$\sum y_i = 1$, $y_i>0$	$O(n)$
Euclidean Proj.	$\sum y_i = 1$, $y_i \geq 0$	$O(n \log n)$
Null-space/Affine	$A x = b$	$O(n m)$
GLinSAT	$\sum y_i = C,\ 0 \leq y_i \leq u_i$	$O(K n)$

GLinSAT generalizes the simplex and sum-to-constant constraints to optionally include upper bounds and continuous relaxations, supporting applications where allocation must obey explicit capacity, fairness, or risk control (Zeng et al., 2024). Classical softmax is best considered a special case, optimal for canonical probabilistic classification but limited for general resource allocation and combinatorial objectives (Berzal, 7 Nov 2025).

7. Extensions and Theoretical Connections

Additivity-constrained output layers align closely with developments in constrained learning, structured prediction, and statistical modeling. The underlying mathematics draws from convex optimization, duality, and manifold geometry (e.g., simplex, affine subspaces). Enforcing $A x = b$ as an exact output constraint obviates the need for penalization, breaking new ground in learning with hard output restrictions (Konstantinov et al., 2023).

The choice between null-space, projection, or entropy-regularized layers is dictated by the structure of $A$, the presence of additional inequality constraints, and the practical need for smoothness or strictness. In more general settings, extensions to simplex-valued (Dirichlet, logistic-normal) or product-of-simplices (multi-assignment) can be realized through composition of basic projection layers or generalized GLinSAT modules (Zeng et al., 2024, Berzal, 7 Nov 2025).

A plausible implication is that the evolution of such constrained output layers will facilitate increasingly sophisticated integration of machine learning and combinatorial optimization, particularly in decision-making applications that demand strict feasibility and differentiability.