The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints

Abstract: Modern deep learning architectures are increasingly multi-task and multi-modal, using a pretrained foundation model combined with task-specific, fine-tuned models. Empirically, exploiting similarity across different problems, instead of solving them individually, can significantly improve overall performance. While the generalization and sample complexity properties of multitask learning have been widely studied, the parametric complexity of joint approximation in comparison to separate approximation remains less well understood. The question is particularly relevant in modern deep learning, where models are increasingly required to satisfy structural constraints such as equivariance, conservation laws, or orthogonality. We prove lower and upper bounds on the description-length for separate and joint approximation classes, respectively, in uniform norm. We build a class of orthogonal functions by composing a shared hard feature, realized by a Rademacher-Haar wavelet series, with Sawtooth-Walsh readouts to enforce orthogonality of output coordinates. The dyadic tree structure of the Rademacher-Haar wavelet concentrates the approximation hardness in the common feature component, while the readouts act as task-specific heads. Using an information-theoretic framework, we obtain a sharp gap between the optimal approximation rates achievable by joint and separate coding. Finally, we realize this separation in a neural network model using Heaviside activations via reduction to triangle-wave approximation. Our results show that even under an orthogonality constraint joint approximation requires strictly fewer bits in compositional architectures, provided the tasks share a latent hard feature. This provides theoretical insight into the description-length-efficiency of compositional multi-output architectures and clarifies how neural networks can retain expressivity under geometric constraints.

• The paper establishes an information-theoretic gap where joint encoding of a shared latent feature yields significantly lower uniform-norm distortion compared to separate encoding under orthogonality constraints.
• It employs a constructive setup using Rademacher-Haar wavelet series and Sawtooth-Walsh functions to model a 'hard' latent feature and enforce orthogonal task-specific readouts.
• The analysis bridges theory and practice by showing that multi-head neural architectures can achieve superior efficiency through shared representations despite strict geometric and quantization limitations.

Information-Theoretic Advantages of Shared Representations under Orthogonality Constraints

Introduction and Problem Formulation

This paper develops a rigorous approximation-theoretic analysis of the efficiency of joint versus separate representation learning in multi-output regression problems subject to orthogonality constraints (2606.16028). Specifically, it addresses the parametric and information-theoretic impact of sharing a latent representation ("shared feature") across multiple unrelated targets when the neural architecture must produce orthogonal outputs. This scenario appears in multi-task and multi-head deep learning pipelines, where a common latent backbone is paired with task-specific orthogonal heads, especially in settings emphasizing equivariance, conservation laws, or geometric constraints.

The authors formalize two coding paradigms: separate approximation, where each output is encoded independently, and joint approximation, where all outputs share an encoded latent feature. They quantify and prove a strict uniform-norm gap in the minimal achievable distortion for these two regimes, even when the output heads are enforced to be orthogonal and the feature to be "hard" (maximal description complexity). These results are grounded in an explicit, constructive function class built from Rademacher-Haar wavelet series (defining the "hard" feature extractor) and Sawtooth-Walsh functions (forming orthogonal head readouts).

Constructive Setup: Rademacher-Haar and Sawtooth-Walsh Architectures

The backbone of the analysis is a class of vector-valued functions

$F_g = (f_1 \circ g, \ldots, f_M \circ g)$

where $g$ is a scalar-valued Rademacher-Haar wavelet series, and each $f_i$ is a non-linear, pairwise orthogonal Sawtooth-Walsh readout. The crucial technical insight is that the approximation complexity – measured by minimal description length for a given error in the uniform norm – is concentrated in the shared latent feature $g$. The orthogonality constraint is imposed exclusively on the heads via their construction as orthogonal readouts, rather than by decomposing the feature space.

Figure 1: An example of the Rademacher-Haar wavelet series (left) and its unresolved tail for $L=2$ (right), which captures the high-complexity feature difficult to approximate individually.

This design ensures that each output is "individually hard" to approximate due to the complexity of $g$ (visualized by the unresolved dyadic-tree tail in Figure 1), yet $g$ is exactly the same for every output. Thus, under joint coding, $g$ is encoded once, and task-specific information is relegated to the low-complexity heads.

Main Theoretical Results: Quantitative Approximation Gap

The central theorem demonstrates a separation in minimal distortion (in bits) between optimal separate and optimal joint approximation of the class $\mathcal{F}$ described above. For $M$ outputs and total bit budget $g$0 (where $g$1 is per-head bit allocation), the ratio of separate to joint error satisfies

$g$2

asymptotically yielding a factor-$g$3 gap (for large $g$4) in favor of joint coding.

The analysis leverages the feature that for Rademacher-Haar series, the unresolved tail (left out by any truncation to $g$5 bits) is "hard" at every dyadic localization. This property is preserved in all output coordinates via the Sawtooth-Walsh nonlinearity, enforcing orthogonality on outputs. Thus, without shared representation, the separate coding must redundantly encode the entire hard tail $g$6 times.

The constructive rate bounds are made explicit in terms of the wavelet series truncation, with the tail error quantifiable at $g$7 per unresolved level (where $g$8 is the truncation level), providing sharp, non-asymptotic description-length bounds.

Realization in Neural Network Architectures

To bridge the codec-centric analysis to concrete neural network models, the paper constructs explicit Heaviside-activated networks that realize both the shared feature and the orthogonal heads. For a fixed bit budget $g$9, a shallow network encodes the $f_i$0-bit truncation of $f_i$1, and each head is realized exactly via Heaviside compositions matched to the dyadic partition of the wavelet series.

Figure 2: Recursive construction of sawtooth functions and characteristic dyadic sets for approximating the absolute value with Heaviside networks. The left panel shows the sawtooth, the middle panel dyadic characteristic functions, and the right panel their sum approximating $f_i$2.

Figure 3: Architecture of the recursive Heaviside network for absolute value approximation. Each layer doubles the number of segments via two skip connections and four neurons with Heaviside activation.

For pointwise multiplication needed in the readout, the authors use recursive triangle-wave approximations (using Heaviside nonlinearity and skip connections) to implement the desired product in $f_i$3 depth and constant width, achieving exponential convergence in depth.

Given quantized weights (matching the information-theoretic description complexity), the joint architecture achieves order-$f_i$4 (in bits) uniform error, whereas any collection of $f_i$5 independent networks under the same bit constraint attains only order-$f_i$6 per output, so is worse by a factor of at least $f_i$7.

Implications and Connection to Multitask Learning Theory

The results provide a first-principles, non-statistical (information-theoretic) argument for the advantage of shared latent representations in multi-output architectures, even under strict output orthogonality. This shows that the performance gain is not an artifact of redundant outputs or mere function class averaging but arises fundamentally from the joint encoding of hard latent structure. This analysis is orthogonal to earlier statistical justifications (sample complexity, inductive bias) and holds at the level of worst-case approximation rates.

Moreover, the construction clarifies that constraints such as orthogonality can be imposed entirely in the head readouts without diminishing the parametric gain from shared feature learning. This lends theoretical support for shared trunk architectures (e.g., multi-head foundation models, operator learning architectures like DeepONet or Fourier Neural Operator) employed in modern deep learning practice.

Outlook and Future Directions

A key implication is that the compositionally hard class constructed here captures essential worst-case uniform-norm limitations of non-shared architectures, extending classical principal component analysis and model reduction principles to a nonlinear, orthogonality-constrained deep learning regime. An open direction is to characterize more general geometric constraints (e.g., equivariance, conservation) under which such bit-economy persists, especially for smooth feature classes or operator-valued function spaces.

Future research may target whether similar rate separations manifest in infinite-dimensional or statistical learning settings, and whether the same efficiency holds for architectures with smooth activations (such as ReLU or Tanh), under finite-precision constraints (Grohs et al., 9 Jun 2026). Identifying the precise mechanism driving the persistence or breakdown of the joint advantage across architectures and constraint classes remains a central mathematical question.

Conclusion

This work establishes that, under strict orthogonality and for function classes with shared, high-complexity latent structure, joint approximation achieves strictly superior uniform-norm description-length efficiency over separate approximation, with the ratio scaling linearly in the number of outputs. The mechanism is explicit and constructive, realized both information-theoretically and in neural architectures with quantized parameters. The results provide foundational theoretical justification for multi-head and multitask design paradigms widely used in deep learning, elucidating the dominant role of latent feature complexity in multi-output approximation scenarios.

---

References:

(2606.16028) The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints (Grohs et al., 9 Jun 2026) Limitations of Learning Tanh Neural Networks with Finite Precision