The Standard Interpretable Model: A general theory of interpretable machine learning to deductively design interpretable methods using Lagrangian mechanics

Abstract: As Artificial Intelligence models grow in complexity, interpretability has become an indispensable tool for understanding, debugging, and controlling their computations. However, interpretability lacks general theories to deductively design interpretable methods. This gap between theories and methods results in a fragmented literature and inconsistent evaluation protocols. To fill this gap, we introduce the Standard Interpretable Model (SIM), a general theory grounded in Lagrangian mechanics that enables the deductive design of interpretable methods. Specifically, the SIM summarises, in a set of premises, what interpretability is for a target user. From these premises, the SIM systematically derives interpretability symmetries and corresponding constraints, which shape the landscape of a Lagrangian whose minima correspond to optimal interpretable models. To reach the minima, one can either update the parameter values of an opaque model to make it more interpretable or compile constraints into an interpretable architecture. We empirically show that the SIM identifies and solves limitations of existing methods (including traditional, concept-based, and mechanistic interpretability), highlights underexplored research directions, and informs the design of core programming interfaces. Beyond being a research method, the deductive nature of the SIM offers pedagogical grounding for interpretability curricula and may shift the scientific community's perspective of a discipline that has long been fragmented.

• The paper introduces the SIM framework using Lagrangian mechanics to formulate and enforce interpretability constraints in machine learning.
• It deduces optimal interpretable models by treating user-defined invariances as symmetry constraints, balancing accuracy and interpretability.
• Empirical results validate that constraint-based and architectural methods better preserve concept semantics and prediction dependencies.

The Standard Interpretable Model: A Lagrangian-Theoretic Framework for Deductive Interpretability in Machine Learning

Introduction and Theoretical Foundation

The paper "The Standard Interpretable Model: A general theory of interpretable machine learning to deductively design interpretable methods using Lagrangian mechanics" (2606.12289) introduces a unified theory for deducing and operationalizing interpretability in ML. The work addresses the lack of systematic, deductive methods for designing interpretable ML models by introducing the Standard Interpretable Model (SIM), a framework inspired by the role of invariances in mathematical physics and geometric deep learning. Interpretability, under this theory, is formalized not as an ad-hoc property or post-hoc artifact, but via a Lagrangian whose minima correspond to models optimally balancing accuracy and interpretability relative to a specified user.

The SIM formalism treats interpretability constraints as invariances (symmetries) under user-motivated transformations, translating these invariances into operational constraints on the ML pipeline. This approach yields a Lagrangian $L=T-V$, where $T$ encodes parameter dynamics and $V$ defines an interpretability landscape as a function of the model parameters. Deduction of the optimal model is achieved via variational principles, mirroring the principle of least action in physics: minimizers of $\iint L \, dz\,dt$ correspond to interpretable and accurate models.

Figure 1: The Standard Interpretable Model characterizes interpretable ML models via a Lagrangian; the interpretability landscape determines model selection, and parameter dynamics govern optimization trajectories.

Formal Structure of the SIM Framework

The SIM's workflow is a six-step method for translating interpretability desiderata into quantifiable and actionable constraints:

1. Interpretability Premises: Interpretability is grounded relative to a user $h$, who is fully parameterized by their vocabulary, conceptual semantics, and bounded reasoning capabilities. This explicit user model allows the SIM to encompass subjectivity and variability across users.
2. Symmetry Induction: Interpretability properties are formalized as invariances under a set of user-specified transformations (symmetries), reflecting what properties should be preserved for interpretability to hold.
3. Constraint Construction: Each symmetry yields an explicit constraint $q(z)\le 0$ that can be measured or imposed in models.
4. Lagrangian Formulation: The landscape $V$—involving task accuracy and interpretability constraints—is cast as part of a model Lagrangian.
5. Optimization (Trajectory Approach): The model can be trained with explicit optimization to minimize $V$, using the principle of least action for parameter updates.
6. Architecture Compilation: Alternatively, constraints can be embodied directly in model structure, guaranteeing interpretability characteristics by construction.

This formalism provides a deductive recipe to construct, analyze, and compare interpretable ML models.

Instantiation: Interpretability for Bounded and Formal Entities

The SIM is instantiated with three core interpretability premises:

• Shared Concept Semantics: The semantics of symbols used by the model must agree with those understood by the user. Formally, if $c_w$ is a model's concept map for symbol $w$, and $T$0 is the human's, then the strict preorder induced over objects (via ranking) must be preserved up to monotonic transformation.
• Prediction-Concept Dependency: The model's output must depend exclusively on the shared concepts (no spurious or hidden variables). This is operationalized by requiring the prediction Jacobian to be contained in the span of the concept map Jacobians.
• Bounded Reasoning: The ways in which concepts are composed to yield predictions must remain within the user’s tractable reasoning class, often formalized by a bounded hypothesis space or operator annihilation (e.g., linear, piecewise, or otherwise bounded complexity).

Each premise is formalized as a symmetry, then as a constraint (functional), then (optionally) as a specialized architecture.

Empirical Validation and Numerical Findings

SIM is empirically validated with experiments examining both constraint-based optimization and architectural compilation.

Concept Semantics: Premise I (Symmetry~1)

Models trained solely to fit concept values (e.g., via $T$1 loss) may fit ground truth scores but violate the semantic ordering constraint, i.e., they do not preserve human concept semantics. A constraint-based loss or architectural approach that enforces semantic monotonicity yields models that precisely preserve the concept preorder with respect to human-annotated concepts, even when MAE is not minimized.

Figure 2: Learned concept maps visualized by predicted values (top); sorted by ground-truth semantic ranking (bottom), revealing that only constraint-based models preserve the preorder, whereas MAE-based models may fit scores but violate semantics.

Prediction-Concept Dependency: Premise II (Symmetry~2)

Optimizing for local alignment of prediction and concept gradients yields improvements only near the optimally constrained points, but fails to enforce global dependence. Only architectures with the dependency structurally imposed (e.g., bottlenecking predictions through concept maps) guarantee that the output is determined by shared concepts everywhere.

Figure 3: Demonstration that optimizing for local gradient alignment enforces dependency only locally, while architectural compilation yields global alignment.

Figure 4: Visualization of the functional relationship between task predictions and concepts; only architectures with explicit bottlenecks guarantee unambiguous dependency.

Bounded Reasoning: Premise III (Symmetry~3)

Increasing the constraint penalty for formula complexity restricts the learned reasoning formula to the desired function class, with architectural compilation ensuring the composition remains within the bounded hypothesis space.

Figure 5: Visualization of learned concept compositions as the regularization parameter increases, showing convergence to bounded complexity as enforced by the differential operator.

Large-Scale Concept-Based Model Analysis

In large-scale pre-trained vision-LLMs (VLMs), pre-extracted concept annotations often fail to preserve monotonic semantic orderings, leading to inconsistent or even contradictory results in pairwise rankings.

Figure 6: Pairwise judgment matrices for "red" concept, showing that concept maps in common VLMs violate basic semantic orderings, and are not reliable for labeling without additional structure.

However, imposing a simple nearest-prototype ranking in latent space, using a set of sorted prototypes, allows for perfect recovery of the semantic ordering without any retraining or fine-tuning of the VLM itself, demonstrating that the semantic structure is present but not properly decoded in the standard pipeline.

Figure 7: Imposing a prototype-based ordering fixes concept semantics in pre-trained models without training—prototype assignment recovers perfect ordering.

Further, in models such as Steerling-8B, next-token predictions depend on only a small number of concept variables, and aggressive test-time sparsification can restrict prediction dependency to as few as 16 concepts with negligible impact on interpretability constraints.

Figure 8: The dependency of prediction Jacobians on concept Jacobians in Steerling-8B, demonstrating that few concepts drive predictions and sparsification is possible without retraining.

Contradictory and Notable Claims

• The paper provides strong evidence that traditional MAE-based metrics do not correlate with interpretability, highlighting the importance of symmetry/constraint-based evaluation.
• It demonstrates that most popular label-free concept annotation procedures in VLMs do not in fact yield consistent or interpretable concept maps, contradicting the implicit assumption of many large-scale explainability pipelines.
• The analysis of chain-of-thought rationales in LLMs provides quantitative evidence that such explanations are not used in the computation of the prediction and thus are not faithful explanations, challenging the prevailing view in some explainability communities.

Practical and Theoretical Implications

The SIM establishes a general framework for developing, comparing, and implementing interpretable ML models. On the practical side:

• It enables deductive design of interpretable architectures, avoiding ad-hoc or post-hoc rationalizations.
• Highlights the gaps and overconstraints of popular concept-based/interpretable methods (e.g., CBMs, sparse autoencoders, prototype models), facilitating more principled model selection and innovation.
• Provides a path for diagnosing and correcting the failure of VLM-concept alignment without expensive retraining, via prototype ranking.
• Offers a foundation for software frameworks supporting interpretable ML, as implemented in PyTorch Concepts.

Theoretical implications include:

• Establishing interpretability as a variational principle, in direct analogy to physics, introduces new avenues for theoretical analysis and constraint-based ML.
• Enabling precise falsification and analytical validation of interpretability claims, rather than relying on qualitative user studies alone.
• Providing a framework within which interpretability guarantees can be verified, optimized, and hardened, offering robustness not achievable with post-hoc analyses.

Directions for Future Research

• The SIM formalism implies that interpretability can, and should, be systematically compiled into ML pipelines, and opens questions regarding the automation of constraint extraction and translation into architectures.
• A key challenge is mapping user desiderata to precise symmetry constraints, and investigating the completeness of the currently identified invariances.
• The hierarchy of optimization vs. architectural approaches to enforcing interpretability constraints suggests further research into trade-offs for efficiency, flexibility, and robustness.
• Extending the framework toward unsupervised or dynamically-evolving user models is a fertile ground for future investigation.
• The SIM's synthesis of geometric, physical, and logical principles positions it as an organizing center for a broader, deductive theory of interpretable computation.

Conclusion

The Standard Interpretable Model establishes a formal, unified, and operational foundation for interpretable ML by leveraging Lagrangian mechanics and symmetry considerations. It enables systematic deduction, analysis, and certification of interpretability in both model design and evaluation, with direct applications to architecture design, optimization routines, and empirical assessment. The framework reveals non-trivial shortcomings in current practice and charts a principled path forward for robust, user-aware, and theoretically grounded interpretable machine learning.