Behavior Learning (BL): Learning Hierarchical Optimization Structures from Data

Abstract: Inspired by behavioral science, we propose Behavior Learning (BL), a novel general-purpose machine learning framework that learns interpretable and identifiable optimization structures from data, ranging from single optimization problems to hierarchical compositions. It unifies predictive performance, intrinsic interpretability, and identifiability, with broad applicability to scientific domains involving optimization. BL parameterizes a compositional utility function built from intrinsically interpretable modular blocks, which induces a data distribution for prediction and generation. Each block represents and can be written in symbolic form as a utility maximization problem (UMP), a foundational paradigm in behavioral science and a universal framework of optimization. BL supports architectures ranging from a single UMP to hierarchical compositions, the latter modeling hierarchical optimization structures. Its smooth and monotone variant (IBL) guarantees identifiability. Theoretically, we establish the universal approximation property of BL, and analyze the M-estimation properties of IBL. Empirically, BL demonstrates strong predictive performance, intrinsic interpretability and scalability to high-dimensional data. Code: https://github.com/MoonYLiang/Behavior-Learning ; install via pip install blnetwork.

• The paper presents a novel framework that infers hierarchical optimization structures using modular, penalty-based blocks, ensuring interpretable utility modeling.
• It unifies predictive accuracy with intrinsic interpretability by composing symbolic utility functions and enforcing optimization constraints.
• Empirical results confirm BL’s competitive performance and enhanced detection in high-dimensional settings, validating its scalable, interpretable approach.

Learning Hierarchical Optimization Structures: The Behavior Learning (BL) Framework

Motivation and Paradigm

The paper introduces Behavior Learning (BL), a general-purpose machine learning framework for inferring interpretable and identifiable optimization structures from data, motivated by the utility maximization paradigm central to behavioral science. BL addresses the persistent performance–interpretability trade-off in predictive modeling by parameterizing a compositional utility function built from modular penalty-based blocks. Each block encapsulates a utility maximization problem (UMP) consisting of objective, inequality, and equality terms, and is amenable to explicit symbolic representation in the style of classical behavioral modeling. The architecture supports hierarchical compositions, thereby enabling the modeling of both single and multi-level optimization structures.

Figure 1: Visualization of the BL framework, illustrating UMP-based behavioral modeling, compositional utility induction, interpretability, and architectural variants.

Theoretical Foundations

BL operates by inducing a conditional Gibbs distribution via parameterized compositional utility defined as a hierarchical composition of $\mathcal B$ blocks, each instantiated as:

$$\mathcal{B}(x, y;\theta) = \lambda_0^{\!\top} \phi\big(U_{\theta_U}(x, y)\big) - \lambda_1^{\!\top} \rho\big(\mathcal{C}_{\theta_C}(x, y)\big) - \lambda_2^{\!\top} \psi\big(\mathcal{T}_{\theta_T}(x, y)\big)$$

with interpretable polynomial feature maps, and non-negative learnable weights. The framework admits universal approximation of continuous conditional distributions (Theorem 3), and its identifiable variant (IBL) satisfies strong statistical guarantees: identifiability, consistency, asymptotic normality, and efficiency. Crucially, IBL ensures that interpretability is unique (modulo symmetry), directly supporting scientific credibility and ground-truth recovery.

Hierarchical and Compositional Interpretability

BL architectures are defined in three principal variants:

• BL(Single): A solitary $\mathcal B$ block corresponding to a single UMP with maximal symbolic interpretability.
• BL(Shallow): Stacking multiple blocks in a shallow hierarchy to capture interactions or trade-offs between multiple UMPs.
• BL(Deep): Extending to deep hierarchies, recursively aggregating block outputs, which naturally represent hierarchical optimization structures found in diverse scientific systems.

The interpretability extends to a compositional symbolic representation, where each layer or block is visualized as an agent solving its UMP, and higher layers aggregate and coordinate lower-layer outputs.

Figure 2: Hierarchical interpretation of deep BL architectures as layered agent systems, each block as a UMP, with macro-level coordination encoded in upper layers.

Empirical Evaluation

Prediction Performance

Extensive evaluation on 10 OpenML datasets demonstrates that BL consistently achieves first-tier accuracy, matching state-of-the-art black-box models and outperforming intrinsically interpretable baselines. BL(Shallow) delivers performance parity with standard MLP architectures, refuting the common assumption of inevitable predictive cost for interpretability.

Figure 3: BL attains first-tier performance in both relative gains and F1-Macro rank, with Shallow variants statistically indistinguishable from state-of-the-art baselines.

Hierarchical Interpretation Case Study

A detailed case study on the Boston Housing dataset exhibits BL's interpretability. BL(Single) recovers an explicit UMP for a representative buyer, with polynomial utility, budget, and belief terms. Deep BL architectures, such as BL[5,3,1], uncover micro-level preference types and macro-level trade-off structures, aggregating them into an effective representative agent. Extracted block coefficients align with established economics literature.

Figure 4: Detailed interpretation of BL modules for housing, showing micro-level preference types and macro-level aggregation.

High-Dimensional Generalization

BL is empirically validated on high-dimensional inputs (image and text datasets) against an energy-based MLP baseline. BL matches or exceeds E-MLP in in-distribution accuracy and achieves superior calibration. On Fashion-MNIST, BL is notably stronger in OOD detection at matched parameter budgets.

Figure 5: BL vs E-MLP on image and text datasets across varying network depth; BL exhibits competitive accuracy and enhanced OOD robustness.

Penalty Mechanism for Constraint Enforcement

The interpretable penalty formulation in BL enables near-hard constraint satisfaction under finite temperature. Direct experimentation demonstrates effective enforcement of nontrivial energy conservation constraints as a function of temperature and penalty scaling, with violations rapidly diminishing as the penalty scale increases or temperature decreases, confirming theoretical expectations.

Figure 6: BL penalty block enforces high-dimensional energy conservation with violation statistics dropping under increased penalty and reduced temperature.

Compositional Visualization: Polynomial Feature Maps

BL's polynomial feature maps support explicit computation graph visualization, enhancing the transparency of learned optimization structures and facilitating the tracing of higher-order interactions and marginal effects.

Figure 7: Visualization of polynomial feature maps as computation graphs, encoding marginal effects and higher-order interactions.

Limitations, Implications, and Future Directions

While BL is theoretically robust, scalability of identifiability guarantees in highly overparameterized settings merits further study. The choice of basis functions impacts numerical stability; alternative bases and conditioning schemes are prospective research avenues. BL's generative modeling capability, currently tied to energy-based training methods, can be further extended for interpretable generation in vision and language domains. Hybrid integration with black-box feature extractors is a viable avenue for partial interpretability in large models, especially in decision-critical applications.

Practically, BL provides a transparent modeling framework for scientific and social domains characterized by optimization and hierarchical complexity. Its capacity for scientifically credible explanations positions it as a promising vehicle for mechanistic interpretability in scientific machine learning, including applications in physics, economics, neuroscience, and policy analysis.

Conclusion

The Behavior Learning framework unifies predictive performance, intrinsic interpretability, identifiability, and scalability under a scientifically grounded, optimization-driven paradigm. BL's compositional architecture enables explicit, hierarchical explanations of complex data-generating processes, matching black-box models in accuracy and providing rigorous statistical guarantees on interpretability. Empirical evaluations substantiate its efficacy across standard and high-dimensional domains, while detailed scientific case studies demonstrate its capacity for explanatory modeling aligned with domain theory. BL thus contributes a formally principled methodology for learning and explaining optimization structures from data, with broad implications for interpretable and credible machine learning in scientific investigations.