Absorbing Many-Body Correlations into Core-Optimized Orbitals

Abstract: The cost of simulating quantum many-body systems - on classical or quantum hardware - scales with the number of variational parameters, so progress at fixed computational budget hinges on more parameter-efficient ansätze. Configuration Interaction (CI) is widely dismissed as parameter-heavy; we show this verdict is an artifact of the orbital basis. Co-optimizing the orbital basis with a sparse CI wavefunction - a method we call Core-Optimized Orbitals (COO) - absorbs a large fraction of the dynamical correlation directly into the single-particle basis, cutting the determinant count by several orders of magnitude beyond the already compact TrimCI ansatz on which it builds. On Fe$_4$S$_4$, a billion-determinant TrimCI+COO wavefunction reaches accuracy that would require $3!\times!10^{14}$ determinants in a localized basis. At matched accuracy, it is $8\times$ more compact than the largest unrestricted-DMRG benchmark ($25\times$ with PT2). Across the iron-sulfur series - from Fe$_2$S$_2$ to the P-cluster (114e,73o) - TrimCI+COO is $10$-$100\times$ more compact than SU(2)-adapted DMRG with entanglement-minimized orbitals at matched accuracy. A tunable Hubbard-on-graph model factorizes the advantage into an orbital-basis gain and an ansatz gain, the latter capturing multi-center entanglement that resists MPS localization. COO therefore changes the picture of CI efficiency: sparse CI with optimized orbitals can outperform state-of-the-art tensor networks on strongly correlated multi-center systems.

• The paper introduces a COO protocol that alternates between TrimCI core discovery and orbital rotation optimization to absorb many-body correlations.
• It demonstrates orders of magnitude compression in wavefunction determinants compared to traditional localized orbitals and tensor network methods.
• The approach provides a scalable, open-source framework for efficiently simulating strongly correlated multi-center molecular systems.

Core-Optimized Orbitals: Many-Body Compactness via Joint Orbital and Sparse CI Optimization

Motivation and Context

Quantum many-body simulations are limited by the exponential scaling of variational parameter space. Historically, ansätze such as Configuration Interaction (CI), tensor networks (DMRG/MPS), and neural network quantum states have traded off expressive power, parameter efficiency, and tractability. CI approaches are conventionally viewed as parameter-inefficient, especially when working in a product-space (localized basis) due to the rapid growth in determinant expansion. The paper "Absorbing Many-Body Correlations into Core-Optimized Orbitals" (2605.22977) challenges this paradigm, arguing that the perceived inefficiency of CI is an artifact of the basis choice. It introduces the Core-Optimized Orbitals (COO) protocol, showing that optimizing the single-particle orbital basis for compactness vastly improves parameter efficiency—enabling sparse CI to outperform state-of-the-art tensor networks on strongly correlated, multi-center molecular systems.

COO Framework and Algorithmic Details

COO alternates between two coupled optimizations: (i) constructing a highly compact CI wavefunction core using the TrimCI protocol, and (ii) variationally optimizing orbital rotations to absorb dynamical correlations directly into the single-particle basis. This feedback loop proceeds as follows:

• TrimCI discovers a compact, high-quality determinant core (typically $10^2$–$10^4$ determinants) that captures dominant ground-state structure.
• Orbital gradients are computed from reduced density matrices (RDMs) of this core, enabling efficient closed-form optimization of the rotation parameters $\kappa$ via BFGS.
• Each trial orbital set is used to rediagonalize the projected Hamiltonian within the core, yielding an exact variational energy at each step. This sharply resolves the parameter landscape and avoids local minima endemic to large-wavefunction orbital optimization approaches (CASSCF/DMRG-SCF).
• The optimized orbitals feed back into TrimCI, producing a fresh determinant core, and the process iterates until convergence (typically 5–10 cycles).

This design leverages the tractability of small cores for rapid orbital optimization—each BFGS iteration costs milliseconds on a single node, making the full procedure scalable. Notably, the orbital rotation acts globally on the CI expansion: a compact COO wavefunction expressed in the rotated basis translates to a long-tail determinant expansion in the original basis, thus "absorbing" many-body correlations into the orbitals.

Figure 1: COO orbital rotation compresses a wavefunction from thousands of determinants (LMO) to a hundred (COO); orbital optimization trajectory and energy transfer upon expansion.

Numerical Results and Benchmarks

Compression on Iron-Sulfur Clusters

On multi-center iron-sulfur clusters (Fe$_2$S$_2$, Fe$_4$S$_4$, and the P-cluster), COO achieves orders of magnitude compression over both localized molecular orbitals (LMO) and advanced tensor network approaches.

• For Fe$_4$S$_4$, a billion-determinant TrimCI+COO wavefunction matches the accuracy achieved by $3\times10^{14}$ determinants in the LMO basis and is at least $10^4$0 more compact than the largest unrestricted-DMRG benchmark ($10^4$1 with PT2 corrections).
• Across the iron-sulfur series, COO is $10^4$2–$10^4$3 more parameter-efficient than SU(2)-DMRG with entanglement-minimized orbitals (EMO) at matched variational accuracy.

  Figure 2: Variational energies and error scaling of TrimCI+COO vs UDMRG and LMO baselines for [Fe$10^4$4S$10^4$5]; COO curve decays faster (slope $10^4$6 vs $10^4$7).

  

  Figure 3: COO workflow and expansion across the iron-sulfur series, showing phase 0 orbital optimization and subsequent determinant expansion, achieving systematic compression.

Beyond Entanglement Minimization

Quantum algorithms favor trial states with high overlap and low parameter count. COO produces wavefunctions with dominant-determinant weights ($10^4$8) $10^4$9 higher than EMO and up to $\kappa$0 higher than LMO on the P-cluster at fixed parameter count ($\kappa$1 for DMRG). At matched energy, COO uses $\kappa$2–$\kappa$3 fewer parameters than DMRG+EMO, and up to five orders of magnitude less than DMRG+LMO.

Figure 4: Dominant-determinant weight and variational energy normalized to parameter count for COO, EMO, and LMO bases; COO is consistently far more compact.

Strong Correlation and Multi-Center Effects

Orbital mutual information analysis reveals that COO-extracted orbitals expose multi-center entanglement structures that cannot be compressed into narrow bands for MPS efficiency. In [Fe$\kappa$4S$\kappa$5], all strongly entangled pairs bridge distinct atomic centers, with $\kappa$6 mutual information distributed across a wide (half-width $\kappa$7) band—a stark contrast to 1D chains where $\kappa$8 suffices.

Figure 5: Comparison of parameter efficiency for TrimCI+COO and DMRG on a tunable Hubbard-on-graph model; ratio increases with graph connectivity.

Physical Interpretation and Theoretical Implications

The COO protocol physically leverages the global action of orbital rotations: a rotation learned from a compact core coherently transforms the entire CI determinant space, transferring compression gains across orders-of-magnitude expansion. This approach shifts the compactness hierarchy: for strongly correlated multi-center systems, a sparse CI in a COO basis is strictly more compact than structured MPS representations—contradicting standard tensor network-centric compressibility assumptions.

Excitation analysis confirms that nearly all high-weight determinants in iron-sulfur clusters are multi-center (not localized), with $\kappa$9–$_2$0 of the excitation weight spanning multiple atomic centers. 1D orbital orderings are therefore fundamentally incompatible with efficient MPS representations on these systems.

Figure 6: Fiedler-ordered mutual information; [Fe$_2$1S$_2$2] shows wide entanglement bandwidth vs H$_2$3 chain.

Figure 7: Multi-center excitation patterns among high-weight determinants reveal cluster-wide correlation; dominant determinant fractions and excitation weights reported.

Practical Impacts and Future Directions

COO introduces a scalable, parameter-efficient regime for sparse CI expansion—enabling variational calculations up to $_2$4 determinants (requiring distributed GPU architectures and mini-task parallelism) with routine jobs on academic clusters. The approach is particularly impactful for quantum chemistry and materials science applications, where strongly correlated electron systems (biological metalloenzymes, transition metal clusters, catalysis sites, etc.) demand efficient many-body representations. The COO workflow is broadly available as an open-source package for integration with classical and hybrid quantum simulation pipelines.

Figure 8: Full TrimCI+COO pipeline, illustrating global optimization, local refinement, and fast expansion phases.

Outlook and Speculation

The demonstrated COO gains suggest a reassessment of CI efficiency vis-à-vis tensor networks. COO is extensible to larger, more complex targets (FeMo cofactor, biological N$_2$5 fixation, and beyond), and can be integrated with alternative many-body ansätze (coupled-cluster, neural quantum states). Its compression mechanism—absorption of dynamical correlations via basis optimization—may generalize to other strongly correlated regimes, potentially shifting ground in classical and quantum electronic structure methods.

Conclusion

Core-Optimized Orbitals reframe the efficiency landscape of quantum many-body simulation. By coupling orbital rotations with sparse CI core discovery, the method absorbs many-body correlations into optimized single-particle bases, yielding wavefunctions that are drastically more compact than conventional representations. On multi-center, strongly correlated systems, COO enables sparse CI to outperform advanced tensor-network approaches, supporting both classical scalability and quantum circuit-friendly state preparation. The method's modularity, practical deployment, and strong numerical performance open further avenues in strongly correlated electronic structure, quantum chemistry, and scalable quantum computing research.