Do Two AI Scientists Agree?

Abstract: When two AI models are trained on the same scientific task, do they learn the same theory or two different theories? Throughout history of science, we have witnessed the rise and fall of theories driven by experimental validation or falsification: many theories may co-exist when experimental data is lacking, but the space of survived theories become more constrained with more experimental data becoming available. We show the same story is true for AI scientists. With increasingly more systems provided in training data, AI scientists tend to converge in the theories they learned, although sometimes they form distinct groups corresponding to different theories. To mechanistically interpret what theories AI scientists learn and quantify their agreement, we propose MASS, Hamiltonian-Lagrangian neural networks as AI Scientists, trained on standard problems in physics, aggregating training results across many seeds simulating the different configurations of AI scientists. Our findings suggests for AI scientists switch from learning a Hamiltonian theory in simple setups to a Lagrangian formulation when more complex systems are introduced. We also observe strong seed dependence of the training dynamics and final learned weights, controlling the rise and fall of relevant theories. We finally demonstrate that not only can our neural networks aid interpretability, it can also be applied to higher dimensional problems.

• The paper introduces the MASS framework with HLNNs to compare AI-learned Hamiltonian and Lagrangian representations from identical data.
• The paper finds that richer, more varied training data promotes convergence toward a common physical theory, mirroring scientific consensus.
• The paper shows that random initialization significantly influences the learned models, often yielding distinct clusters of theoretical interpretations.

The paper "Do Two AI Scientists Agree?" (2504.02822) investigates whether distinct AI models, when trained on identical scientific datasets, converge to the same underlying physical theory or develop divergent representations. It draws an analogy to the historical evolution of scientific theories, where consensus emerges or fractures based on accumulating experimental evidence. The authors propose that a similar dynamic governs AI models trained for scientific discovery tasks.

Methodological Framework: MASS and Hamiltonian-Lagrangian Neural Networks

To probe the internal representations learned by AI models and assess their concordance, the authors introduce the Mechanistic Agreement of Scientific Solutions (MASS) framework. This framework utilizes Hamiltonian-Lagrangian Neural Networks (HLNNs) as surrogate "AI scientists." HLNNs are a class of physics-informed neural networks architecturally constrained to represent either a Hamiltonian ($H$) or a Lagrangian ($L$) function. Given state variables (e.g., position $q$, momentum $p$, or position $q$, velocity $\dot{q}$), these networks learn a scalar function ($H(q, p)$ or $L(q, \dot{q})$) whose derivatives, according to Hamilton's or Euler-Lagrange equations, predict the system's time evolution.

The choice of HLNNs is motivated by their inherent connection to fundamental physics principles. By training these networks to predict system dynamics from observed data (e.g., trajectories), the learned internal function ($H_{learned}$ or $L_{learned}$) can be interpreted as the physical theory discovered by the AI. The core idea of MASS is to train multiple instances of HLNNs, typically differing only by their random weight initializations (seeds), on the same dataset and then compare the resulting learned Hamiltonians or Lagrangians to quantify agreement.

Training an HLNN typically involves minimizing a loss function based on the difference between the predicted time derivatives (obtained by automatic differentiation of the learned $H$ or $L$) and the true time derivatives derived from the training data. For a Hamiltonian network learning $H(q, p)$, the loss might involve terms like:

$$\mathcal{L} = \sum_{i} \left\| \frac{\partial H_{learned}}{\partial p_i} - \dot{q}_i^{true} \right\|^2 + \left\| -\frac{\partial H_{learned}}{\partial q_i} - \dot{p}_i^{true} \right\|^2$$

Similarly, for a Lagrangian network learning $L(q, \dot{q})$:

$$\mathcal{L} = \sum_{i} \left\| \frac{d}{dt}\left(\frac{\partial L_{learned}}{\partial \dot{q}_i}\right) - \frac{\partial L_{learned}}{\partial q_i} \right\|^2$$

where the total time derivative requires evaluating second derivatives of $L_{learned}$.

Experimental Design and Simulation of AI Scientists

The study employs standard physics problems as testbeds. Training data consists of trajectories or states sampled from simulations of these systems. The concept of "increasingly more systems provided in training data" likely refers to augmenting the dataset with trajectories from a wider variety of initial conditions, system parameters, or even qualitatively different types of physical systems governed by related principles. This simulates the process of scientists gathering more diverse experimental evidence.

To simulate a population of AI scientists approaching the same problem, the authors train numerous HLNN instances on the same dataset(s), varying only the random seed used for parameter initialization. This allows for exploring the diversity of solutions accessible through standard stochastic optimization procedures. The results across these multiple seeds are then aggregated and analyzed using the MASS framework.

Convergence, Divergence, and Theory Formation

A key finding is that the degree of agreement between the learned theories (represented by the learned $H$ or $L$ functions) depends on the richness of the training data. When trained on data from simpler setups or fewer system variations, the AI scientists (different HLNN instances) may learn disparate theories. However, as the training data incorporates more complex systems or a wider range of phenomena, the learned theories tend to converge. This mirrors the scientific process where more comprehensive data constrains the space of viable theories.

Despite the general trend towards convergence with richer data, the authors observe that the AI scientists sometimes form distinct clusters. Each cluster represents a different, internally consistent theory learned by a subset of the models. This suggests that even with substantial data, multiple incompatible (or perhaps subtly different) theoretical interpretations can emerge, potentially corresponding to different local minima in the optimization landscape that are nonetheless consistent with the provided data. Quantifying the agreement likely involves comparing the functional form or predicted dynamics of the learned $H$ or $L$ functions across different seeds.

Transition from Hamiltonian to Lagrangian Learning

The study reports an interesting transition in the type of physical formulation predominantly learned by the HLNNs. In simpler scenarios, the models often converge to representations best described as Hamiltonians. However, when trained on more complex systems, the AI scientists show a tendency to favor learning Lagrangian formulations.

This shift might be attributed to several factors:

1. Expressive Power: Lagrangian mechanics can sometimes handle constraints or certain types of potentials more naturally than Hamiltonian mechanics, which might be advantageous for more complex systems.
2. Coordinate Systems: The choice between $(q, p)$ and $(q, \dot{q})$ coordinates, inherent in H vs. L formulations, might make one representation easier to learn or more compact for certain dynamics.
3. Optimization Landscape: The loss landscape associated with learning $L$ might possess more favorable properties (e.g., wider minima) for complex dynamics compared to the landscape for learning $H$.
4. Inductive Bias: The specific architecture and training procedure might implicitly favor one formulation under certain data regimes.

This finding suggests that the complexity of the scientific problem itself can influence the preferred theoretical framework adopted by AI discovery systems.

Influence of Initialization and Training Dynamics

The research highlights a strong dependence of the final learned theory on the random seed used for initialization. Different initial weights can lead the optimization process into different basins of attraction, resulting in convergence to distinct $H$ or $L$ functions, even when trained on the exact same data. This underscores the stochastic nature of deep learning optimization and its impact on scientific reproducibility and interpretation in the context of AI-driven discovery.

The authors frame this seed dependence as governing the "rise and fall" of specific theories within the population of AI scientists. A particular theory (a specific learned function structure) might be frequently discovered with certain initializations but rarely with others, suggesting that the accessibility of a given theoretical solution is sensitive to the starting point of the learning process.

Interpretability and Scalability

A significant advantage claimed for the HLNN-based approach is enhanced interpretability. Because the network is explicitly designed to learn a physical function ($H$ or $L$), the learned parameters directly map onto a representation of a physical theory. This allows for mechanistic interpretation – analyzing the learned function to understand the physical principles the AI has identified.

Furthermore, the paper demonstrates that this methodology is not limited to simple, low-dimensional toy problems but can be applied to higher-dimensional systems, suggesting potential for tackling more complex and realistic scientific challenges.

Conclusion

The paper (2504.02822) provides evidence that AI models trained on scientific tasks exhibit convergence behaviors analogous to human scientific communities, with richer data generally leading to greater consensus on the underlying theory. The proposed MASS framework, using HLNNs, offers a way to mechanistically interpret the learned theories and quantify agreement among AI models. Key findings include the data-dependent convergence of theories, the potential formation of distinct theory clusters, an observed shift from Hamiltonian to Lagrangian learning preferences with increasing system complexity, and a strong influence of initialization on the final learned representation. These results have implications for understanding the nature of knowledge representation in AI models used for science and for developing more robust and interpretable AI discovery systems.