EuS/Bi2-xSbxSe3: TI-FMI Heterostructures

• The paper reveals that Sb tuning in EuS/Bi2-xSbxSe3 heterostructures controls interfacial charge transfer, strain, and exchange phenomena, influencing Dirac surface transport.
• The study employs MBE growth, Raman spectroscopy, and magnetotransport to evaluate interfacial morphology and anomalous planar Hall effects in distinct TI thickness regimes.
• The findings underscore that precise Sb composition and growth conditions dictate whether topological surface states exhibit proximity-induced exchange gaps or maintain weak antilocalization.

EuS/Bi$_{2-x}$Sb$_x$Se$_3$ heterostructures are exchange-coupled ferromagnetic-insulator/topological-insulator bilayers in which EuS is interfaced with Sb-substituted Bi$_2$Se$_3$ thin films to combine insulating ferromagnetism with Dirac surface-state transport. Within this materials class, experiments have focused on epitaxial growth, interfacial strain, charge transfer, planar Hall transport, weak antilocalization, and the extent to which magnetic proximity opens an exchange gap in the topological surface states. Reported results span both positive and negative manifestations of coupling: unconventional planar Hall and planar magnetoresistance effects were observed in EuS/Bi$_{0.22}$Sb$_{0.78}$Se$_3$ ultrathin films, whereas other EuS/Bi$_{2-x}$Sb$_x$Se$_x$0 samples with $_x$1 and $_x$2 exhibited persistent weak antilocalization and no evidence for long-range magnetic order in the TI surface states under the reported growth conditions (Rakhmilevich et al., 2018, Wang et al., 10 Sep 2025). Analysis of the parent EuS/Bi$_x$3Se$_x$4 system further identified large interfacial charge transfer as a central mechanism and explicitly proposed Sb composition as a control parameter for barrier height, strain, and exchange-coupled functionality in EuS/Bi$_x$5Sb$_x$6Se$_x$7 (Osterhoudt et al., 2017).

1. Materials platform and heterostructure realizations

The experimentally reported EuS/Bi$_x$8Sb$_x$9Se$_3$0 structures are based on MBE-grown TI films on SrTiO$_3$1(111), followed by in situ EuS deposition and Al$_3$2O$_3$3 capping. In the transport study of unconventional planar Hall response, the TI composition was Bi$_3$4Sb$_3$5Se$_3$6 ($_3$7), the TI thicknesses were 4 QL ($_3$8 nm) and 3 QL ($_3$9 nm), the EuS overlayer was 5 nm, and the cap was 4 nm Al$_2$0O$_2$1 on 0.25 mm thick SrTiO$_2$2(111) substrates (Rakhmilevich et al., 2018). In the later transport study emphasizing proximity failure, the TI thickness was 6 nm, the EuS thickness was 3 nm, the cap was 2 nm Al$_2$3O$_2$4, and the nominal Sb concentrations were $_2$5 and $_2$6 on 0.5 mm thick SrTiO$_2$7(111) substrates (Wang et al., 10 Sep 2025). The parent EuS/Bi$_2$8Se$_2$9 structures used for Raman and interfacial analysis employed 7 QL Bi$_3$0Se$_3$1 ($_3$2 nm), EuS layers of 2, 5, or 10 nm, and a 5 nm Al$_3$3O$_3$4 cap on either Al$_3$5O$_3$6 or SrTiO$_3$7 substrates (Osterhoudt et al., 2017).

Structure	Representative stack	Reported characterization
EuS/Bi$_3$8Se$_3$9	7 QL Bi$_{0.22}$0Se$_{0.22}$1 / 2, 5, or 10 nm EuS / 5 nm Al$_{0.22}$2O$_{0.22}$3	Raman; earlier XRD and cross-sectional TEM; reflectivity fringes in XRD
EuS/Bi$_{0.22}$4Sb$_{0.22}$5Se$_{0.22}$6	4 QL or 3 QL TI / 5 nm EuS / 4 nm Al$_{0.22}$7O$_{0.22}$8	RHEED, XRD, AFM, cross-section TEM, magnetotransport
EuS/Bi$_{0.22}$9Sb$_{0.78}$0Se$_{0.78}$1 with $_{0.78}$2	6 nm TI / 3 nm EuS / 2 nm Al$_{0.78}$3O$_{0.78}$4	RHEED; Hall transport; weak antilocalization analysis

The common architectural logic is an ultrathin TI layer thick enough to sustain surface transport but thin enough for strong interface sensitivity, combined with an insulating EuS overlayer to avoid metallic shunting. Structurally, the $_{0.78}$5 devices were supported by X-ray diffraction showing distinct $_{0.78}$6 peaks from Bi$_{0.78}$7Sb$_{0.78}$8Se$_{0.78}$9 and $_3$0 EuS with no extra phases or interdiffusion, and by AFM/cross-section TEM indicating a continuous EuS film and sharp TI/EuS interface (Rakhmilevich et al., 2018). By contrast, the $_3$1 and $_3$2 study reported RHEED as the sole in situ structural feedback and explicitly noted that no ex situ X-ray diffraction rocking curves or cross-sectional transmission electron microscopy were presented (Wang et al., 10 Sep 2025). This methodological difference is significant because several reported physical conclusions hinge on atomic-scale interface abruptness.

2. Interfacial strain and charge-transfer energetics

In the parent EuS/Bi$_3$3Se$_3$4 system, Raman spectroscopy resolved the Bi$_3$5Se$_3$6 phonons at $_3$7 cm$_3$8 ($_3$9), $_{2-x}$0 cm$_{2-x}$1 ($_{2-x}$2), $_{2-x}$3 cm$_{2-x}$4 ($_{2-x}$5), and $_{2-x}$6 cm$_{2-x}$7 ($_{2-x}$8), enabling direct assessment of interfacial strain through phonon shifts (Osterhoudt et al., 2017). The frequency shift was expressed as

$_{2-x}$9

or equivalently

$_x$0

with typical $_x$1–$_x$2 for the $_x$3 modes. Experimentally, Bi$_x$4Se$_x$5 on sapphire showed tensile in-plane strain $_x$6 with $_x$7 cm$_x$8, whereas Bi$_x$9Se$_x$00 on SrTiO$_x$01 showed compressive in-plane strain $_x$02 with $_x$03 cm$_x$04; adding EuS from 2 to 10 nm increased net tensile strain and shifted $_x$05 downward by up to $_x$06 cm$_x$07 relative to bare TI (Osterhoudt et al., 2017).

The same study treated the EuS/TI interface as a metal-semiconductor junction with $_x$08 eV, $_x$09 eV, $_x$10 eV, and $_x$11 eV for EuS, yielding

$_x$12

and

$_x$13

Within that model, the Fermi level in EuS is driven $_x$14 eV down into its valence band, the entire 2–10 nm EuS layer is depleted of conduction electrons, and optical transitions across the gap require $_x$15 eV so that a 2.33 eV laser can only drive virtual processes; the EuS Raman resonance therefore collapses (Osterhoudt et al., 2017). This is the basis for the reported absence of the EuS magnetic Raman feature in EuS/Bi$_x$16Se$_x$17 heterostructures. A common alternative explanation—Fabry–Pérot suppression—was numerically excluded: the multilayer interference calculation predicted an overall $_x$18–$_x$19 enhancement relative to bare EuS/Al$_x$20O$_x$21, not a suppression.

For EuS/Bi$_x$22Sb$_x$23Se$_x$24, the same analysis proposed explicit compositional trends. Sb doping was stated to shrink the in-plane lattice constant of Bi$_x$25Se$_x$26 by $_x$27 per 10% Sb, to blue-shift the Bi$_x$28Se$_x$29 phonons by $_x$30 cm$_x$31 per 0.1% compression, and to feed back a small tensile strain into EuS of $_x$32 (Osterhoudt et al., 2017). It also proposed that the TI work function rises by $_x$33 eV per 10% Sb, so that the EuS–TI Schottky barrier

$_x$34

can be tuned from 3.05 eV up to $_x$35 eV. A larger $_x$36 was therefore linked to stronger hole doping in EuS, potentially higher $_x$37, and deeper depletion of Raman-active levels. At the interface, the model Hamiltonian was written as

$_x$38

Because these statements were presented as implications for Sb-alloyed structures rather than direct measurements on them, they should be read as an explicit modeling framework rather than a completed experimental verification.

3. Unconventional planar Hall transport in the $_x$39 regime

In EuS/Bi$_x$40Sb$_x$41Se$_x$42 heterostructures, magnetotransport measurements revealed an unconventional planar Hall effect and a gate-tunable hysteretic planar magnetoresistance under in-plane field (Rakhmilevich et al., 2018). The devices used standard Hall-bar geometry with current $_x$43 along the $_x$44-axis and transverse voltage probes along $_x$45, while an in-plane magnetic field $_x$46 up to $_x$47 Oe was rotated by angle $_x$48 relative to $_x$49. The transverse signal $_x$50 showed clear hysteresis loops with EuS coercivity $_x$51–$_x$52 Oe. The angular dependence was anomalous: $_x$53 was largest for $_x$54 ($_x$55) and vanished for $_x$56 ($_x$57), unlike the usual $_x$58 pattern of conventional planar Hall effects.

The data were compared to the phenomenological form

$_x$59

but $_x$60 was treated as an effective fitting amplitude because the measured $_x$61 scaled $_x$62 (Rakhmilevich et al., 2018). Gate dependence sharpened the distinction between surface-state and non-surface contributions: in 4 QL samples, $_x$63 peaked sharply when the chemical potential crossed the Dirac point at $_x$64 V, reaching several $_x$65 with Hall angle $_x$66. Longitudinal resistance under in-plane field exhibited a butterfly-shaped hysteresis, with PMR up to $_x$67 in 5 nm EuS/4 QL samples, defined by

$_x$68

Its maximum occurred at $_x$69 V, slightly off the Dirac point, implying some bulk contribution.

Thickness dependence was central to the interpretation. Below $_x$70 QL for (Bi,Sb)$_x$71Se$_x$72, top and bottom surface states hybridize and open a gap at $_x$73. In the 3 QL heterostructure, both PMR and PHE collapsed around the charge-neutral point, the $_x$74-$_x$75 curve showed a larger on/off swing, and the $_x$76–$_x$77 curves became non-linear near the CNP but remained linear away from it (Rakhmilevich et al., 2018). These results were taken as evidence that gapless Dirac surface states are essential to the observed PHE.

Theoretical analysis in the same work excluded a linear Hall mechanism for $_x$78 by mirror symmetry. For $_x$79, the system is symmetric under $_x$80; under $_x$81, $_x$82 and $_x$83, so a linear $_x$84 must vanish. Two non-linear mechanisms were then outlined. One was a spin-orbit-torque-induced anomalous Hall response, in which a current-generated effective field $_x$85 tilts the in-plane magnetization out of plane. The other was an intrinsic non-linear Hall response of Dirac surface states based on

$_x$86

for which Berry curvature is zero everywhere except at a single gapless point, so no linear Hall term is allowed, but a second-order Hall current $_x$87 is symmetry-permitted. In the clean limit and as $_x$88, the scaling

$_x$89

was reported to explain the enhancement near the Dirac point and the disappearance when a hybridization gap opens (Rakhmilevich et al., 2018).

4. Carrier density, mobility, and weak antilocalization in the $_x$90 and $_x$91 regime

A later EuS/Bi$_x$92Sb$_x$93Se$_x$94 study concentrated on low-temperature electronic transport and found no magnetotransport signature of proximity-induced time-reversal symmetry breaking under its growth conditions (Wang et al., 10 Sep 2025). Hall resistance $_x$95 was linear from $_x$96–$_x$97 K and gave $_x$98 cm$_x$99 for $_3$00 and $_3$01 cm$_3$02 for $_3$03, both $_3$04-type. Increasing Sb content therefore moved the Fermi level closer to the Dirac point. The low-field mobility,

$_3$05

decreased monotonically with temperature from $_3$06 cm$_3$07/V s at 2 K to $_3$08 cm$_3$09/V s at 60 K in both samples, and a fit to $_3$10 was taken to show that phonon scattering dominates over ionized-impurity scattering from Sb dopants across the entire measured temperature range. Zero-field $_3$11 showed metallic behavior with $_3$12, which the authors interpreted as indicating that surface states dominate conduction and bulk carriers are suppressed.

Weak antilocalization was analyzed through the 2D Hikami–Larkin–Nagaoka expression

$_3$13

with $_3$14 and fits performed within $_3$15 T, even though magnetoconductance was measured up to $_3$16 T (Wang et al., 10 Sep 2025). The prefactor $_3$17 remained between $_3$18 and $_3$19 for all samples and temperatures, consistent with transport dominated by topological surface channels plus small bulk contributions. The coherence length in the $_3$20 film was $_3$21 nm at 2 K and dropped to $_3$22–$_3$23 nm at 60 K; in the $_3$24 film, $_3$25 nm at 2 K and converged to similar values by 30–60 K.

Two dephasing regimes were distinguished. Below $_3$26 K, dephasing was enhanced in the higher-doped $_3$27 film and was attributed to stronger electron–electron interactions induced by disorder; above 30 K, phonon scattering dominated and $_3$28 became nearly independent of $_3$29 (Wang et al., 10 Sep 2025). Although the use of SrTiO$_3$30(111) would permit back-gating in principle, no gate-dependent data were reported in this study; control of $_3$31 was achieved exclusively by Sb stoichiometry. This contrasts with the earlier $_3$32 work, where gate tuning was integral to isolating the PHE peak at the Dirac point.

5. Magnetic proximity, exchange coupling, and conflicting experimental outcomes

The magnetic interpretation of EuS/Bi$_3$33Sb$_3$34Se$_3$35 heterostructures remains non-uniform across the reported literature. EuS itself is an insulating ferromagnet with bulk Curie temperature $_3$36–$_3$37 K and an in-plane easy axis associated with shape anisotropy; earlier EuS/TI studies cited in the transport literature reported that the EuS magnetization lies within the film plane with negligible out-of-plane component, and the exchange interaction at the interface was written as

$_3$38

with $_3$39 on the order of a few–tens of meV, though not directly measured in that work (Rakhmilevich et al., 2018). In the parent EuS/Bi$_3$40Se$_3$41 context, prior magnetometry summarized in the Raman study found that EuS $_3$42 increased from bulk 16 K to $_3$43–$_3$44 K, with even paramagnetic signatures up to 300 K, and the same summary noted that charge transfer in EuS can strengthen superexchange and RKKY interactions, raising $_3$45 by $_3$46 (Osterhoudt et al., 2017).

Against that backdrop, the $_3$47 EuS/Bi$_3$48Sb$_3$49Se$_3$50 results were presented as evidence for an exchange-coupled FMI/TI heterostructure in which Dirac surface states mediate an unconventional non-linear Hall response (Rakhmilevich et al., 2018). By contrast, the later $_3$51 and $_3$52 study stated that long-range magnetic order is not formed in the TI surface states under its growth conditions. Its central argument was the persistence of weak antilocalization from 2 K to 60 K with no crossover to weak localization even below $_3$53; in an ideal EuS/TI heterostructure with strong interfacial exchange, the expectation was that the WAL dip in $_3$54 would be replaced by WL because of time-reversal symmetry breaking and an exchange gap $_3$55 (Wang et al., 10 Sep 2025). No SQUID or vibrating-sample magnetometry of those heterostructures was reported, and there was no quantitative estimate of $_3$56 or $_3$57.

A separate ambiguity arises from spectroscopy. In EuS/Bi$_3$58Se$_3$59, the disappearance of the EuS Raman magnetic mode could, in principle, be read as consistent with either full ferromagnetic order, which quenches spin disorder, or suppression of the resonant intermediate states; the detailed optical analysis favored the latter mechanism, namely large charge transfer into the EuS valence sector, rather than magnetic ordering alone (Osterhoudt et al., 2017). Taken together, the literature does not support a single universal magnetic outcome for EuS/Bi$_3$60Sb$_3$61Se$_3$62. A plausible implication is that exchange phenomena in this platform are exceptionally sensitive to interface morphology, stoichiometry, and thickness, so that transport signatures interpreted as proximity effects in one growth window may be absent in another.

6. Design constraints, optimization strategies, and open directions

Several practical design rules recur across the reported studies. To maximize the planar Hall response, the TI thickness should be kept at or above 4 QL so that the surface states remain gapless, the chemical potential should be tuned to the Dirac point, and the in-plane easy axis of EuS can be used without relying on out-of-plane remanence; in the $_3$63 devices on SrTiO$_3$64, the PHE maximum occurred near $_3$65 V (Rakhmilevich et al., 2018). At the same time, the later transport study argued that moderate Sb doping is preferable from the standpoint of coherence: heavier doping brings $_3$66 closer to the Dirac point but increases disorder-enhanced electron–electron dephasing below 30 K, so that $_3$67 was described as optimal for maintaining large $_3$68 while suppressing bulk carriers (Wang et al., 10 Sep 2025).

The 2017 interfacial model proposed a broader EuS/Bi$_3$69Sb$_3$70Se$_3$71 optimization principle: the optimal $_3$72 balances three requirements—maximized $_3$73 via hole transfer into EuS, sufficient band bending to maintain surface-state exchange coupling, and tolerable interfacial strain to avoid dislocation formation (Osterhoudt et al., 2017). This framing links composition, interfacial electrostatics, and lattice mismatch within a single control problem rather than treating Sb substitution solely as a Fermi-level tuning parameter.

The negative proximity result in the $_3$74 and $_3$75 samples translated directly into growth recommendations. The reported suggestions were a pre-EuS-deposition anneal of the Bi$_3$76Sb$_3$77Se$_3$78 surface to desorb excess Se and heal vacancies, lower-temperature and more gentle EuS growth to minimize interdiffusion, and in situ characterization such as cross-sectional TEM or X-ray reflectivity to verify sub-nm interface roughness (Wang et al., 10 Sep 2025). Additional possibilities proposed there were insertion of an ultrathin seed layer, for example a few QLs of clean Bi$_3$79Se$_3$80, to improve wetting and prevent chemical intermixing, or reversing the interface sequence by growing EuS at elevated substrate temperature followed by a low-temperature Bi$_3$81Sb$_3$82Se$_3$83 cap.

The resulting picture is technically specific. EuS/Bi$_3$84Sb$_3$85Se$_3$86 is not merely a magnetic-proximity variant of Bi$_3$87Se$_3$88; it is a heterostructure family in which Sb composition affects carrier density, coherence, lattice mismatch, and the EuS/TI barrier height, while interface sharpness determines whether those ingredients produce detectable exchange coupling. The principal open issue is therefore not whether EuS and Sb-doped Bi$_3$89Se$_3$90 can coexist epitaxially—they can—but under what interfacial conditions the coexistence yields a reproducible exchange gap rather than only charge-transfer-modified transport and surface-state weak antilocalization.