Investigation of the $J_1$-$J_2$ Heisenberg model on the triangular lattice: A study with projected entangled-pair states

Abstract: The nature of the quantum spin liquid (QSL) phase in the frustrated $J_1$-$J_2$ Heisenberg model on the triangular lattice remains an open and actively debated problem. In this work, we employ the infinite projected entangled-pair state (PEPS) to systematically investigate the model under different symmetry constraints. Our simulations reveal a direct transition from the $120<sup>\circ$ Néel state to a putative QSL at $J_2/J_1\approx 0.08$, signaled by the collapse of magnetic order. We further show that, through either an appropriate unitary rotation or spontaneous spin long-range order, the stripe antiferromagnetic phase can also be accurately captured within the infinite PEPS framework. A central focus of our study is the role played by the PEPS symmetry in approximating the QSL ground-state sandwiched between the two magnetic phases. We first found that a fully-symmetric topological $\mathbb{Z}_2$ Resonating Valence Bond state, which can be written as a simple PEPS with bond dimension $D=3$, exhibits a reasonably good variational energy. Motivated by this finding, we have further constructed generic $\mathbb{Z}_2$-symmetric PEPS of larger bond dimension (up to $D=7$). We found that, under wavefunction optimization, spinons condense and, simultaneously, topological vison excitations get confined, hence precluding $\mathbb{Z}_2$ topological order. This strongly indicates the gapless (or critical) nature of the QSL phase, which is most naturally consistent with a U(1) Dirac spin liquid scenario.

• The paper introduces iPEPS with both unrestricted and symmetry-restricted ansatz to accurately capture phase boundaries in the frustrated J1-J2 Heisenberg model.
• It demonstrates that a transition from 120° Néel order to a gapless U(1) spin liquid occurs near J2/J1 ≈ 0.08, ruling out a gapped Z2 phase.
• Extrapolated energies and correlation measurements benchmark favorably against DMRG, establishing a robust framework for studying frustrated quantum magnets.

Investigation of the $J_1$-$J_2$ Heisenberg Model on the Triangular Lattice via Infinite Projected Entangled-Pair States

Introduction and Context

The $J_1$-$J_2$ spin-$1/2$ Heisenberg model on the triangular lattice is a canonical platform for exploring nontrivial quantum phases induced by geometric frustration. Antiferromagnetic nearest-neighbor ($J_1$) and next-nearest-neighbor ($J_2$) couplings compete, generating a rich phase diagram encompassing classical long-range magnetic orders and highly entangled quantum-disordered states. Theoretical and numerical studies have established three major regimes: a $120^\circ$ N\'eel ordered phase, a quantum spin liquid (QSL), and a stripe antiferromagnetic phase, with the detailed nature of the intermediate QSL still unresolved, especially regarding its potential topological order.

Recent progress in tensor network methods, particularly infinite projected entangled-pair states (iPEPS), allows controlled variational studies in two dimensions without finite-size artifacts. This work systematically analyzes the $J_1$-$J_2$ Heisenberg model using iPEPS with both unrestricted and symmetry-respecting ans\"atze to chart the phase boundaries and elucidate the structure of the intermediate QSL (2606.31021).

Model and Tensor Network Methodology

The studied Hamiltonian is:

$J_2$0

with $J_2$1. The intrinsic coordination number $J_2$2 makes direct PEPS optimization numerically challenging due to high tensor ranks. The authors utilize a coarse-graining mapping from the triangular to an effective square lattice, grouping three sites per block, allowing iPEPS optimization with reduced tensor rank while keeping the full quantum statistics (see Figure 1).

Figure 1: PEPS construction on the coarse-grained lattice, facilitating efficient tensor network contraction on the triangular system.

Order parameters and correlation functions—spin, dimer, and topological excitations—are extracted via CTMRG-environment tensors (Figure 2).

Figure 2: Schematic of correlation function measurement in PEPS: (a) spin, (b) dimer, (c) spinon, (d) vison string operators.

Phase Diagram and Benchmarking

The phase diagram comprises three regimes:

• For small $J_2$3, a $J_2$4 N\'eel state is stabilized.
• For $J_2$5, magnetic order collapses, and a QSL regime emerges.
• For large $J_2$6 ($J_2$7), the system transitions to a stripe AF phase.

  Figure 3: Phase diagram of the $J_2$8-$J_2$9 model, displaying $J_1$0 N\'eel, QSL, and stripe AF phases.

Extensive finite bond-dimension scaling demonstrates that the iPEPS method quantitatively matches or slightly outperforms contemporary DMRG and VMC results in both energy and order parameters (see Figure 4 for the Heisenberg benchmark at $J_1$1).

Figure 4: Finite bond dimension extrapolation of ground-state energy per site and magnetic order for $J_1$2, benchmarked against DMRG and other TNS approaches.

Bond-energy distributions confirm that increasing the bond-dimension restores lattice symmetries (Figure 5).

Figure 5: Evolution of bond energy anisotropy with bond dimension; high $J_1$3 restores expected rotational symmetry.

Quantum Spin Liquid Regime

Numerical Detection of QSL Phase

For $J_1$4, extrapolated magnetization vanishes (Figure 6), indicating breakdown of long-range magnetic order and the stabilization of a nonmagnetic QSL. This boundary is robust across various finite-$J_1$5 scaling analyses.

Figure 6: Finite-$J_1$6 scaling for ground state observables at increasing $J_1$7, showing collapse of magnetic order at $J_1$8.

Symmetry-Restricted PEPS: Probing Topological Order

1. SU(2) Symmetric and $J_1$9 Point Group PEPS: The implementation of full $J_2$0 symmetry, extended on both the original and coarse-grained lattices, yields competitive but not optimal energies at accessible bond-dimensions. The nearest-neighbor RVB state (bond dimension $J_2$1) is captured exactly and improved by allowing long-range RVB components (Figure on $J_2$2 ansatz).
2. $J_2$3-Symmetric PEPS: By imposing only virtual $J_2$4 gauge symmetry (originating from the center of $J_2$5), the authors access ans\"atze up to $J_2$6. Extrapolated energies systematically improve with $J_2$7 and nearly saturate to those of the unrestricted PEPS, outperforming basic VMC and RVB-type states (Figure 7).

   Figure 7: $J_2$8 scaling of ground-state energies for $J_2$9-symmetric PEPS compared to unrestricted PEPS and various RVB ans\"atze.

Breakdown of $1/2$0 Topological Order

A detailed analysis of the transfer matrix spectrum in the $1/2$1-symmetric sector reveals a two-fold degeneracy associated with spinon condensation (Figure 3c, Figure on Z2 ansatz). While spin and dimer correlations are short-ranged, the spinon correlator exhibits perfect long-range order, and the vison string correlator decays exponentially—signaling vison confinement and absence of deconfined topological order (Figure 3d).

Thus, the QSL in this parameter regime is not a gapped $1/2$2 spin liquid, in contrast to certain DMRG predictions. Instead, the data is consistent with a gapless or critical regime, compatible with a $1/2$3 Dirac spin liquid.

Stripe Antiferromagnetic Phase

At large $1/2$4, the stripe AF phase is accessed by either spontaneous symmetry breaking or explicit unitary rotations of the iPEPS ansatz (Figure 8). The ground state energy and order parameter are sharply differentiated depending on whether translation symmetry breaking is allowed (rotated ansatz) or artificially imposed (unrotated, Schrödinger-cat-like state), but both approaches yield the correct stripe pattern in long-range spin correlations.

Figure 8: Energetics and order parameter scaling for stripe AF phase at $1/2$5 in rotated vs unrotated PEPS. (c) Transfer matrix spectra; (d) Spin and dimer correlations (long-range and exponentially decaying, respectively).

Discussion and Implications

This work definitively confirms a robust nonmagnetic QSL phase at intermediate $1/2$6, with the boundary sharply diagnosed by the vanishing of the extrapolated magnetic order parameter (see Figure 6). The unrestricted iPEPS approach not only resolves both classical magnetic regimes but, critically, through controlled symmetry-restricted optimization, robustly rules out a gapped $1/2$7 topological phase in favor of a scenario consistent with gapless $1/2$8 Dirac spin liquid behavior.

The methodology—combining iPEPS with controlled symmetry constraints and high-efficiency coarse-graining—sets a quantitative benchmark for future 2D frustrated magnetism studies and offers a template for addressing more complex models (e.g., $1/2$9-$J_1$0 square and kagome systems). Technically, the systematic analysis of transfer-matrix degeneracies deepens the understanding of how tensor environments encode long-range order and emergent symmetry breaking, especially in exotic quantum phases.

Conclusion

The $J_1$1-$J_1$2 Heisenberg model on the triangular lattice, analyzed via advanced iPEPS implementations, exhibits sharply defined $J_1$3 N\'eel, QSL, and stripe AF phases. The QSL regime is robustly established but does not support $J_1$4 topological order—instead, wavefunction optimization leads to spinon condensation and vison confinement, precluding a gapped topological phase. The variational energetics and correlation diagnostics are most compatible with a gapless $J_1$5 Dirac spin liquid. These results provide both a framework and inspiration for future tensor network studies of highly frustrated 2D quantum magnets and their (non-)topological quantum spin liquid states.