Magnetic order in a frustrated two-dimensional atom lattice at a semiconductor surface

Abstract

Two-dimensional electron systems, as exploited for device applications, can lose their conducting properties because of local Coulomb repulsion, leading to a Mott-insulating state. In triangular geometries, any concomitant antiferromagnetic spin ordering can be prevented by geometric frustration, spurring speculations about 'melted' phases, known as spin liquid. Here we show that for a realization of a triangular electron system by epitaxial atom adsorption on a semiconductor, such spin disorder, however, does not appear. Our study compares the electron excitation spectra obtained from theoretical simulations of the correlated electron lattice with data from high-resolution photoemission. We find that an unusual row-wise antiferromagnetic spin alignment occurs that is reflected in the photoemission spectra as characteristic 'shadow bands' induced by the spin pattern. The magnetic order in a frustrated lattice of otherwise non-magnetic components emerges from longer-range electron hopping between the atoms. This finding can offer new ways of controlling magnetism on surfaces.

ARTICLE

Received 26 Oct 2012 | Accepted 20 Feb 2013 | Published 27 Mar 2013

Gang Li1, Philipp Hpfner2, Jrg Schafer2, Christian Blumenstein2, Sebastian Meyer2, Aaron Bostwick3, Eli Rotenberg3, Ralph Claessen2 & Werner Hanke1

Two-dimensional electron systems, as exploited for device applications, can lose their conducting properties because of local Coulomb repulsion, leading to a Mott-insulating state. In triangular geometries, any concomitant antiferromagnetic spin ordering can be prevented by geometric frustration, spurring speculations about melted phases, known as spin liquid. Here we show that for a realization of a triangular electron system by epitaxial atom adsorption on a semiconductor, such spin disorder, however, does not appear. Our study compares the electron excitation spectra obtained from theoretical simulations of the correlated electron lattice with data from high-resolution photoemission. We nd that an unusual row-wise antiferromagnetic spin alignment occurs that is reected in the photoemission spectra as characteristic shadow bands induced by the spin pattern. The magnetic order in a frustrated lattice of otherwise non-magnetic components emerges from longer-range electron hopping between the atoms. This nding can offer new ways of controlling magnetism on surfaces.

DOI: 10.1038/ncomms2617

Magnetic order in a frustrated two-dimensional atom lattice at a semiconductor surface

1 Institut fr Theoretische Physik und Astrophysik, Universitat Wrzburg, Am Hubland, Wrzburg 97074, Germany. 2 Physikalisches Institut, Universitat Wrzburg, Am Hubland, Wrzburg 97074, Germany. 3 Advanced Light Source, Lawrence Berkeley National Laboratory, 1 Cyclotron Road, Berkeley, California 94720, USA. Correspondence and requests for materials should be addressed to J.S. (email: mailto:[email protected]

Web End [email protected] ).

NATURE COMMUNICATIONS | 4:1620 | DOI: 10.1038/ncomms2617 | http://www.nature.com/naturecommunications

Web End =www.nature.com/naturecommunications 1

& 2013 Macmillan Publishers Limited. All rights reserved.

ARTICLE NATURE COMMUNICATIONS | DOI: 10.1038/ncomms2617

a

b

Geometric frustration and its interplay with strong electronic correlations has long been a subject of very active research, since Anderson1 suggested that a spin-1/2

Heisenberg antiferromagnet (AF) on the triangular lattice would have a gapless quantum spin liquid ground state, that is, a resonant valence bond state. Quite generally, this interplay is believed to be the source of a variety of exotic phases in triangular systems: examples span from the possible quantum spin liquid state in k-(ET)2Cu2(CN)3 (refs 2,3) over to a magnetically ordered Mott insulator, which provides the parent phase for the superconductivity found in k-(BEDT-TTF)2X organics4,5 and

NaxCoO2?yH2O (ref. 6), with the latter having possibly a topological superconductivity phase, that is, a gap structure breaking time-reversal symmetry7.

One key issue, which has recently been intensively studied, is the possibility of a magnetically ordered state appearing despite the frustration and, in particular, how to pin down its appearance in a convincing theoretical and experimental manner. The possibility of an ordered magnetic state in a spatially anisotropic triangular AF has received special interest for the spin-1/2 Heisenberg AF: for nearly isotropic exchange (J0 J, where J

and J0 are the anisotropic nearest-neighbour couplings) this model displays long-range spiral order, in accordance with strong-correlation studies of the corresponding Hubbard models with isotropic hopping8,9. Several studies have proposed that, as J0 is enhanced, that is, anisotropy introduced, quantum uctuations destroy this order and induce an 2D spin liquid phase9. Therefore, it came as a substantial surprise that order, that is, a collinear AF (CAF) state, was stabilized in the spatially anisotropic AF at small J0=J, in particular, as it is unusual for quantum uctuations to stabilize an ordered state of higher classical energy10.

In this study, we verify that the CAF state can be achieved even in an isotropic triangular lattice from a different perspective, including additional hopping processes beyond nearest neigh-bours. The latter situation is obviously very relevant for realistic frustrated materials, such as the actual adatom system on a semiconductor surface, that is, Sn/Si(111)-3

p 3

p ,

3 3

t

t

Si

Sn

c

3

K

M

E F

E(k)

E(k)

d

E F

Q

M3

3

M3 K3

3

Figure 1 | Electron correlations in a triangular lattice. (a,b): Schematic presentation of the3

p 3

p

-Sn/Si(111) surface and the isotropic triangular antiferromagnet in the case of a spiral AF order (t; t0: electron-hopping parameters). (c) The single QP band E(k) in the Fermi liquid regime corresponding to our LDA surface band. (d) QP (heavy line) and shadow band (light line) induced by the Mott metal-insulator transition.

which we

consider here. In a previous work11, we found that, by including the next-nearest-neighbour hopping as dictated by density functional theory (DFT) calculations, the magnetic spin susceptibility indicates an instability not towards the 120 AF, that is, spiral order, but towards the CAF state. The earlier conjecture of a CAF pattern is made here rather concrete by combining a dynamical cluster approximation (DCA) plus DFT theoretical analysis of the single-particle spectral function of the triangular lattice Sn/Si(111) system with high-precision angle-resolved photoelectron spectroscopy (ARPES). In particular, the ordered collinear magnetic structure can be identied in the ARPES spectra in terms of shadow bands (that is, spectral weight redistributions) induced by the collinear spin pattern. This establishes the single-particle spectral function and the combination of up-to-date many-body calculations with high-precision ARPES as a powerful tool for extracting spin patterns in frustrated magnets.

The adatom system Sn/Si(111)-

3

p 3

p ,

with its triangular lattice arrangement (Fig. 1a) of Sn atoms on top of Si(111), provides an ideal playground for studying geometric frustration in the presence of strong electronic correlations: a dip in the conductance at the Fermi level, which was observed in scanning tunnelling spectroscopy at low temperature12 together with the theoretical calculations11,1315 reveal that indeed correlations are strong and drive the system into a Mott metal-insulator transition.

In such a geometrically frustrated lattice arrangement, any magnetic ordering may be prevented. In this case, one may expect

a spin liquid, which is a magnetic system that has melted because of quantum uctuations, even at zero temperature. However, in our combined theoretical and experimental work, the realization of a frustrated triangular arrangement of Sn adatoms on a Si(111) surface yields instead an ordered CAF state. The key mechanism that drives the formation of magnetic order in an otherwise non-magnetic surface are electron-hopping processes beyond nearest-neighbour atoms in the presence of strong electronic correlations.

ResultsEffect of magnetic order on the electronic states. Let us start out with the possibility of a magnetically ordered 120 AF (that is, spiral) state, as indicated in Fig. 1, and its inuence on the quasi-particle (QP) spectral function A(k, E). This spiral magnetic reconstruction pattern corresponds to a (3 3) reconstruction

(Fig. 1b), while the CAF order, after the three rotations (see Fig. 3b) preserving the lattice symmetry (domain averaging), results in a 23

p 2 3

p

reconstruction.

In the Fermi liquid regime, which may be here taken as our surface band from the local density approximation (LDA) (see below) crossing the Fermi energy EF, the QP peak of A(k, E)

denes a single energy E(k) for each value of k in the rst surface Brillouin zone (SBZ), as shown in Fig. 1c. When a spin-density wave (SDW) is present, as in Fig. 1d, k is mixed with k Q by

exchange Bragg scattering, where Q is the magnetic ordering vector. This creates energy eigenstates |CkS, which no longer have a sharp momentum, that is, |CkS uk|kS vk|k QS,

with u2k v2k 1, where |kS, |k QS are the (for example, LDA)

eigenstates of Fig. 1c, uk, vk denote the standard coherence factors

2 NATURE COMMUNICATIONS | 4:1620 | DOI: 10.1038/ncomms2617 | http://www.nature.com/naturecommunications

Web End =www.nature.com/naturecommunications

& 2013 Macmillan Publishers Limited. All rights reserved.

NATURE COMMUNICATIONS | DOI: 10.1038/ncomms2617 ARTICLE

of BCS theory, which describe here the SDW gap features introduced by AF order16. The energy spectrum of the |CkS states is schematically drawn in Fig. 1d, where the states denoted by a heavy line carry a large weight, that is, u2k 4 1=2, and the states denoted by a light line carry the small weight v2k o 1=2. Thus, if one creates a hole of energy E(k) in the lower band by ARPES, one will observe the emitted electron having momenta k with probability u2k and (k Q) with probability v2k. Hence, the SDW

Bragg scattering leads to a shadow band, displaced by Q in Fig. 1d, a celebrated phenomenon observed for example, for the high-Tc cuprates in the square-lattice AF, that is, undoped situation16.

Many-body modelling of strong electron correlations. We now turn to the Sn/Si(111) surface system. Before we discuss our results in Figs 24, a few remarks are in order about our theoretical and experimental set-ups, both of which are more detailed in the Methods section and in the Supplementary Information, that is, Supplementary Figs S1S4.

In our theoretical description, we start from a DFT (LDA) electronic structure calculation of a slab conguration of (six) Si-layers, with the Sn atoms placed on the top-layer in equivalent (T4) lattice positions. This is in agreement with geometrical structure determinations using X-ray diffraction17. In particular, it excludes any additional lattice superstructure, for example, a (3 3) lattice periodicity. For technical reasons (see Methods) the

Si bottom-layer is saturated by H-atoms. In the next step, the Sn-related surface band (shown in Fig. 1c) is projected onto a maximally localized Wannier function (MLWF) basis from which the dominant electronic hopping parameters t and t0 as indicated in Fig. 1b are derived and a single-band Hubbard model is constructed11. This model is then solved employing the DCA18, with the continuous-time quantum Monte Carlo method19 as a cluster solver. Details of the (LDA DCA) methodology can

again be found in the Methods section. Finally, to determine the strength of the electronic correlations, the corresponding Coulomb interaction UB0.66 eV was chosen so as to achieve best agreement for the local density of states (DOS) between theory and experiment (Supplementary Fig. S4). Using this U-value, our approach starts from the gapped correlated state. The spin-susceptibility calculations11 suggest a magnetic, that is, collinear order. Here, we perform a detailed study of the single-particle excitation, which, in terms of the shadow bands and their spectral weight transfer (Figs 2,3), provides rm support for the collinear state.

Experimentally, the Sn/Si(111) system, with a

3

p 3

p

to a (3 3)

symmetry, which could, in principle, be consistent with a spiral SDW (Fig. 1b) or a (3 3) surface reconstruction20,21.

(ii) However, closer inspection of Fig. 2 reveals a more complex and exciting underlying reconstruction pattern: rst of all, if a (3 3) symmetry would be realized, then a smoking-

gun test would be that the energy value of the shadow band at the G-point should be degenerate, that is, at the same energy as

ARPES

a b

K

M

LDA+DCA

LDA + DCA

0.3 0 0.3

0.3 0 0.3

ARPES

EF

EF

K

M3

K

M3

EF

E(eV)

0.8 0.6 0.4 0.2 0 0.8 0.6 0.4 0.2 0

0.8 0.6 0.4 0.2 0

c d

EF

M

M

K3

surface reconstruction of high quality as veried by low-energy electron diffraction (LEED) and scanning tunnelling microscopy (STM), was exposed to ARPES carried out at the Advanced Light Source in Berkeley. The photoelectrons were detected with an energy resolution of 25 meV throughout all measurements (see Methods section).

Spectral function with shadow bands. Figure 2 summarizes our main ndings for both the experimental (ARPES) and theoretical (LDA DCA) k-resolved photoemission spectra along two high-

symmetry directions (that is, GK and GM). A rather good overall agreement emerges from this comparison, on the basis of which we can condently extract the existence of an unexpected new magnetic order, that is, the CAF order:(i) The upper row (Fig. 2a,b) for GK seems rst to convey the existence of a (3 3) reconstruction pattern in accordance with

the 120 AF order of Fig. 1b and the SDW-derived bands displaying both large and small spectral weights, as schematically depicted in Fig. 1d. In particular, we nd the shadow bands between the G and M3-symmetry points of the (3 3) SBZ

(depicted in Fig. 3a) carrying the characteristically small spectral weights. These shadow band regions are observed in both ARPES and in the (LDA DCA) calculations. They are high

lighted close to the G point by the four yellow circles in Fig. 2. This is contrasted by the LDA bands, which are in this k-space regime above the Fermi energy, not contributing to the photo-emission spectral weight. Closer inspection reveals another structure just below 0.6 eV close to the G-point, which is a

bulk band of the Si(111) substrate (see Li et al.11) and not a surface band and, therefore, not relevant to the surface magnetic pattern.

So, at a rst glance, the appearance of the shadow band seems to modify the spectral dispersion from3

p 3

p

K3

M23

M23

EF

E(eV)

0.8 0.6 0.4 0.2 0

E(eV) E(eV)

EF

Figure 2 | Spectral function from photoemission and many-body theory. Comparison of the ARPES data (TE60 K, hv 130 eV) in (a,c) with the

spectral functions from LDA DCA calculations in (b,d), along GK and GM

directions, respectively. White arrows indicate band maxima, which are not present in the metallic LDA-derived bands (see LDA plots, left-hand side), but are created by the Mott-insulating state, characterized by shadow bands. Yellow circles around G highlight these shadow bands observed in both ARPES and in the LDA DCA calculations.

NATURE COMMUNICATIONS | 4:1620 | DOI: 10.1038/ncomms2617 | http://www.nature.com/naturecommunications

Web End =www.nature.com/naturecommunications 3

& 2013 Macmillan Publishers Limited. All rights reserved.

ARTICLE NATURE COMMUNICATIONS | DOI: 10.1038/ncomms2617

a

b

3 3

3 3

AF spin

spiral

K

33

M

1 dEAk; Ef E; T. As

before, A(k, E) denotes the spectral function in the sector around k and f(E, T) the Fermi function, thus, n(k) is a measure of the spectral weight below EF in a momentum sector dened by the vector k.n(k) is plotted in Fig. 3d as a function of the correlation strength U for three different momentum-space areas termed K, G and M, as indicated in Fig. 3b for the examples of the small hexagons around the G- and K-points. The three momentum sectors arise naturally from the underlying lattice symmetry in our 9-site DCA scheme (see Supplementary Fig. S3). This scheme is based on coarse graining the k-space (here the3

p 3

p

K3

M23

M3

23 23

CAF zone

3 3 DCA

K23

M

K

c d

10.8

Spectral Weight Transfer

0.60.40.2 0

K

M

n(k)

SBZ), into 9 sections. The technical details of this procedure are not essential and, thus, are relegated to the Methods section. Very importantly, the inclusion of next nearest-neighbour hoppings in the presence of electron correlations inherently leads to AF ordering, where strong interactions are found to favour CAF order11.

DiscussionWhat is of relevance here is that we observe an increase of nG (around the G point) with increasing interaction U, while nK (around the K point) behaves exactly the opposite. In contrast, nM stays almost constant. This is exactly what one expects in a Mott

Hubbard type of metal-insulator transition, driven by strong correlations. The momentum region around the K-point transfers spectral weight to the G-point region (see Fig. 1c,d), thereby creating the lower Hubbard band now visible as a shadow band in photoemission below EF. The constant nM demonstrates even further that this spectral weight transfer occurs between G- and K-regions. This part of our test, thus, proves the existence of correlation (SDW)-induced shadow bands and concomitant spectral weight transfer. To discriminate between the (3 3)-

type spiral AF order and CAF order with effective 2

3

p 2 3

p

0.3

0.4 0.5 0.6 0.7

U(eV)

Figure 3 | Collinear antiferromagnetic ordering. (a,b) Magnetic-order induced zone boundaries for the3

p 3

p

-Sn/Si(111) SBZ, (a) with 120

AF, (b) with CAF order in three-fold rotational superposition, leading to an effective 23

p 2 3

p

SBZ. (c) Row-wise spin alignment of the CAF order. (d) Spectral weight transfer (see arrow) consistent with shadow bands induced by the CAF superstructure.

the QP band at the K-point. This is clearly not the case in Fig. 2a,b, both in experiment and in theory (see black vertical dashed lines). Thus, within the theoretical calculation and, independently, within the ARPES data, we observe about a tenth of an eV energy offset between G- and K-points. On the scale of the bandwidth of B0.3 eV, this is a relevant energy difference and, therefore, allows us to identify the magnetic structure from the electron band symmetries.(iii) The white arrows in Fig. 2 indicate the new SDW-induced band maxima not present in the LDA-derived bands. The maxima appear at the transition point from a large to a small spectral weight (for example, around the M3-symmetry point in

Fig. 2a,b). Further support that the (3 3) symmetry is not

realized comes from the observation (more pronounced in the theoretical results in Fig. 2d) that, along the symmetry direction GM, the transition point (white arrow) is not located in the close vicinity of the (3 3) symmetry point K3, but at a new

symmetry point, denoted by M2

3

symmetry, we refer to the rst part of our analysis based on Fig. 2, which clearly argues for the latter case.

To make this point even more transparent, we overlay in Fig. 4 the ARPES intensity (false colour plot) with our theoretical (LDA DCA) QP peak dispersion (red colour). Again, we can

read off the good overall agreement and we nd that the shadow band at the G point is not degenerate, that is, at the same energy, as the QP band at the K point. This degeneracy would only happen if the band would be back-folded by the (3 3) magnetic order of Fig. 1b, that is, the spiral (120) AF

order. Again, this part of the analysis clearly rules out (3 3)

order. On the other hand, back-folding, according to the collinear 23

p 2 3

p

SBZ in Fig. 3b, results in agreement between theory and experiment. Taken together with our earlier conjecture for a CAF (that is, the row-wise AF), based on spin-susceptibility calculations11, this indeed establishes the collinear magnetic order in an isotropic triangular AF. We note, that a CAF order has recently been detected also in the triangular lattice AF CuFeO2 (ref. 22), in which a collinear 4-sublattice AF order is realized in 2D-planes, however, embedded in a 3D-environment.

In summary, this work demonstrates that a comparison of high-precision ARPES with state-of-the-art theoretical calculations, which combine rst-principle electronic structure (LDA) with cluster many-body (DCA) calculations, allows for detailed

p in Fig. 2c,d.

In summary, both ARPES experiment and theory rule out a (3 3) reconstruction, which could be of the spiral 120 AF

magnetic origin as illustrated in Fig. 1b. Similarly, a (3 3)

lattice reconstruction, which has also been discussed for the Sn/ Si(111) adatom system20,21, is excluded by both our LDA DCA calculation where all Sn atoms are located at

equivalent lattice sites, and the non-degeneracy (in Fig. 2) of the bands at G and K positions.

Redistribution of spectral weight and magnetic order. Next, we provide a second part of our smoking-gun test, which concerns our MottHubbard physical picture, that is, that a magnetic order appears in the strongly correlated Sn/Si(111) system despite the obvious lattice frustration. This is illustrated in Fig. 3 where the effective surface SBZs for 120 AF (excluded) and CAF orders11 are depicted in panels (a) and (b), respectively. Figure 3c illustrates the row-wise spin alignment in real space. A

characteristic ngerprint is the formation of shadow bands and the related spectral shifts, for example, the spectral weight shift from the momentum region around the K-point to the region around the G-point (just as in Fig. 1c,d). Such an analysis can straightforwardly be performed in our LDA DCA scheme by

extracting the average occupation number n(k) at each momentum sector, that is, nk R

1

4 NATURE COMMUNICATIONS | 4:1620 | DOI: 10.1038/ncomms2617 | http://www.nature.com/naturecommunications

Web End =www.nature.com/naturecommunications

& 2013 Macmillan Publishers Limited. All rights reserved.

NATURE COMMUNICATIONS | DOI: 10.1038/ncomms2617 ARTICLE

3 / 23 M3 K3

M3/ K23 M23 3 / 23

EF

3

kx

2

0.4

2t1 coskx 2 cos 3

p

2 ky cos

0.2

2t2

cos 3

p ky 2 cos

32 kx cos 3

p

2 ky

2t3cos2kx 2 coskx cos 3

p ky

0

E (eV)

52 kx cos 3

p

2 ky cos2kx cos 3

p ky

2

where t1 52.7 meV. Using t1 as the energy unit, the longer-range hoppings,

which is t2/t1, t3/t1, t4/t1, t5/t1, are derived as 0.3881, 0.1444, 0.0228, 0.0318,

respectively.

As shown in Supplementary Fig. S2(b), the non-interacting DOS of the tight-binding Hamiltonian equation (2) is asymmetric with respect to the Fermi level, in particular exhibiting a large intensity at the upper band edge. This is quite different from the DOS of the triangular model with only nearest-neighbour hopping. As discussed in our main text, the inclusion of the realistic higher-order hopping processes is decisive for the appearance of the CAF11.

Equation (1) was solved by using the DCA26, which recovers the innite lattice by proliferating Nc-site clusters with periodic boundary condition. In DCA, to restore translational invariance within the cluster, the intercluster hopping amplitudes are modulated by the superlattice momentum ~k as well as the intracluster hopping integrals. Supplementary Fig. S3 illustrates the rst SBZ of a triangular lattice, which is dispatched into Nc 9 equal-area sectors. Each red dot

represents the origin of a momentum sector. An arbitrary momentum k in the original rst SBZ can be expressed as a sum of the superlattice momentum ~k and cluster momentum K, that is, k K

0.2

4t4

cos

cos

kx

2

0.4

p

3 2 ky 2t5cos2 3

p ky

cos

3

0.6

M

K

0.4

2 cos3kx cos 3

p ky

Figure 4 | Band back-folding in magnetically ordered state. Comparison of the (LDA DCA) QP dispersion (red curve) with the ARPES intensity

(false colour plot) in the original

3

p 3

p

SBZ (lower labels) and the

(3 3) and 2 3

p 2 3

p

SBZs (upper labels). Inset: replica of the ARPES shadow band within dashed box, displayed with enhanced contrast, which highlights the non-degeneracy between G and K.

insight and determination of complex spin patterns in frustrated magnetic systems. This has been demonstrated here for the model system Sn/Si(111), which can accurately be tuned by external parameters (for example, temperature T) into a Mott metal-insulator transition with unexpected CAF order. But the key role established here for the single-particle spectral function of Sn/ Si(111) goes way beyond this specic adatom surface system, and should offer a rm basis for identifying other recently much-discussed complex spin patterns in frustrated systems, and also resolve whether or not a spin liquid is stabilized by geometrical frustration.

Methods

Sample preparation and ARPES experiment. The Sn/Si(111)-

3

p 3

p

~k. In the DCA, the Greens function is coarse-grained18 over the superlattice momentum ~k, that is,

GsK; ion

Nc

N

X~k

1ion m eK ~k sK; ion: 3

The DCA self-consistency loop is closed by the Dyson equation,G 1sK; ion GsK; ion 1 sK; ion. To calculate the cluster self-energy

sK; ion from Gs(K, ion) an Nc-site cluster problem has to be solved. In this

study, we employ the continuous-time quantum Monte Carlo method with the paramagnetic condition, that is, G"K; ion G#K; ion. It is important to note

that although we consider only a 9-site cluster in our calculation, all different hopping processes in equation (2) are included in equation (3), after proliferating the 9-site cluster to innite size in DCA. Thus, our single-band model with the longer-ranged hoppings dictated by a priori LDA calculations is different from any study of the triangular model with only the nearest-neighbour hopping.

The value of the Coulomb interaction U was xed so as to achieve the best agreement of the DOS in both the LDA DCA and the experiment. As shown in

Supplementary Fig. S4, at UB0.66 eV (which is slightly above UcE0.60 eV required for the metal-insulator transition in this system11,14), we nd a rather good agreement at all temperatures studied. According to our calculations the spectral weight at the Fermi level EF is gradually enhanced with the increase of temperature. The respective on-set temperature TcB120 K is higher than what has been experimentally found by other groups12.

References

1. Anderson, P. W. The resonating valence bond state in La2CuO4 and superconductivity. Science 235, 11961198 (1987).

2. Moessner, R. & Sondhi, S. L. Resonating valence bond phase in the triangular lattice quantum dimer model. Phys. Rev. Lett 86, 18811884 (2001).

3. Shimizu, Y., Miyagawa, K., Kanoda, K., Maesato, M. & Saito, G. Spin liquid state in an organic Mott insulator with a triangular lattice. Phys. Rev. Lett 91, 107001 (2003).

4. Kanoda, K. Electron correlation, metal-insulator transition and superconductivity in quasi-2D organic systems, (ET)2X. Phys. C

Superconductivity 282-287, 299 (1997).5. McKenzie, R. H. Similarities between organic and cuprate superconductors. Science 278, 820821 (1997).

6. Takada, K. et al. Superconductivity in two-dimensional CoO2 layers. Nature 422, 5355 (2003).

7. Kiesel, M., Platt, C., Hanke, W. & Thomale, R. Anisotropic chiral d id

superconductivity in NaxCoO2?yH2O. Preprint at http://arxiv.org/abs/1301.5662v1

Web End =http://arxiv.org/abs/ http://arxiv.org/abs/1301.5662v1

Web End =1301.5662v1 , (2013).8. Heidarian, D., Sorella, S. & Becca, F. Spin-1/2 Heisenberg model on the anisotropic triangular lattice: from magnetism to a one-dimensional spin liquid. Phys. Rev. B 80, 012404 (2009).

triangular surface system, examined by ARPES, has been prepared as follows: N-type Si substrates (ro0.01 Ocm) were ashed for several times at E1,250 C until a sharp (7 7) pattern was observed in LEED. Subsequently, 1/3 of a

monolayer of tin was deposited on the substrates by electron beam evaporation from a pyrolytic boron nitride crucible. After an additional anneal at E700 C the

Sn/Si(111)-

3

p 3

p

surface reconstruction was veried in high quality by LEED, see Supplementary Fig. S1(a). The STM image of occupied states at room temperature (Supplementary Fig. S1(b)) serves as a proof of the low-defect density at the atomic scale. Sn atoms appear as bright protrusions in a hexagonal arrangement of3

p 3

p

periodicity. Importantly, we have not found any signature of a (3 3) or any other kind of superstructure within a large temperature

range, that is, T 20300 K, by means of LEED. This is consistent with earlier

ndings by STM, where the absence of a structural change down to T 6 K was

stated23.

ARPES experiments were carried out at the electronic structure factory endstation of beamline 7.01 at the Advanced Light Source, which provides helium cooling down to 10 K and is equipped with a 6-axis goniometer. Photoelectrons, excited by linearly polarized radiation, were detected with a Scienta R 4000 spherical analyser with energy resolution set as 25 meV throughout all measurements.

Dynamical cluster approximation (DCA). To study the competing low-T phases in the Sn/Si(111) system, we combine an LDA Hamiltonian represented by HLDA,

with an on-site Coulomb interaction, represented by a Hubbard-U term. The essential idea then is to include long-ranged hoppings (as shown in Fig. 1b of main text) and to demonstrate that they are responsible for the unexpected magnetic order. We thus start from

H HLDA U X

i

ni"ni#; 1

where HLDA P

k;s ekcy

kscks is the Hamiltonian of the low-energy effective band for

the Sn/Si(111) surface calculated from the LDA to the DFT. ek is well separated from the other bands and mainly due to Sn 5pz-orbitals. This band is projected onto MLWF following the standard recipes24,25. The single-particle hopping

parameters ti are then determined from the MLWF matrix elements of the Kohn Sham (LDA) Hamiltonian, giving rise to the following dispersion relation:

ek

NATURE COMMUNICATIONS | 4:1620 | DOI: 10.1038/ncomms2617 | http://www.nature.com/naturecommunications

Web End =www.nature.com/naturecommunications 5

& 2013 Macmillan Publishers Limited. All rights reserved.

ARTICLE NATURE COMMUNICATIONS | DOI: 10.1038/ncomms2617

9. Balents, L. Spin liquids in frustrated magnets. Nature 464, 199208 (2010).10. Starykh, O. A. & Balents, L. Ordering in spatially anisotropic triangular antiferromagnets. Phys. Rev. Lett. 98, 077205 (2007).

11. Li, G., Laubach, M., Fleszar, A. & Hanke, W. Geometrical frustration and the competing phases of the Sn/Si(111)

3

p 3

p R30 surface systems. Phys. Rev. B

83, 041104 (R) (2011).

12. Modesti, S. et al. Insulating ground state of Sn/Si(111)-

23. Morikawa, H., Matsuda, I. & Hasegawa, S. STM observation of Si(111)-a-3

p 3

p -Sn at low temperature. Phys. Rev. B 65, 201308 (2002).24. Mosto, A. A. et al. Wannier90: a tool for obtaining maximally-localised Wannier functions. Comput. Phys. Commun. 178, 685699 (2008).

25. Giannozzi, P. et al. Quantum espresso: a modular and open-source software project for quantum simulations of materials. J. Phys. Condens. Matter 21, 395502 (2009).

26. Hettler, M. H., Tahvildar-Zadeh, A. N., Jarrell, M., Pruschke, T. & Krishnamurthy, H. R. Nonlocal dynamical correlations of strongly interacting electron systems. Phys. Rev. B 58, R7475R7479 (1998).

Acknowledgements

W.H. acknowledges discussions with R. Thomale (Stanford) and A. Fleszar (Wrzburg). This work is nancially supported by the Deutsche Forschungsgemeinschaft under grant FOR 1162.

Author contributions

G.L. and W.H. performed the theoretical calculations., P.H., J.S., C.B., S.M., A.B. and E.R. were involved in the experiments, G.L., P.H. and J.S. analysed the data, G.L., P.H., J.S., R.C. and W.H. discussed the scientic content with all authors and wrote the manuscript.

Additional information

Supplementary Information accompanies this paper at http://www.nature.com/naturecommunications

Web End =http://www.nature.com/ http://www.nature.com/naturecommunications

Web End =naturecommunications

Competing nancial interests: The authors declare no competing nancial interests.

Reprints and permission information is available online at http://npg.nature.com/reprintsandpermissions/

Web End =http://npg.nature.com/ http://npg.nature.com/reprintsandpermissions/

Web End =reprintsandpermissions/

How to cite this article: Li, G. et al. Magnetic order in a frustrated two-dimensional atom lattice at a semiconductor surface. Nat. Commun. 4:1620 doi: 10.1038/ncomms2617 (2013).

p R30.

Phys.

3

p 3

Rev. Lett 98, 126401 (2007).13. Profeta, G. & Tosatti, E. Triangular Mott-Hubbard insulator phases of Sn/ Si(111) and Sn/Ge(111) surfaces. Phys. Rev. Lett 98, 086401 (2007).14. Schuwalow, S., Grieger, D. & Lechermann, F. Realistic modeling of the electronic structure and the effect of correlations for Sn/Si(111) and Sn/Ge(111) surfaces. Phys. Rev. B 82, 035116 (2010).

15. Profeta, G., Continenza, A., Ottaviano, L., Mannstadt, W. & Freeman, A. J. Structural and electronic properties of the Sn/Si(111)

3

p 3

p R30 surface.

Phys. Rev. B 62, 15561559 (2000).

16. Schrieffer, J. R., Wen, X. G. & Zhang, S. C. Dynamic spin uctuations and the bag mechanism of high-Tc superconductivity. Phys. Rev. B 39, 1166311679 (1989).

17. Conway, K. M., Macdonald, J. E., Norris, C., Vlieg, E. & van der Veen, J. F. The structure of the Si(111)

3

p 3

p R30-Sn

surface determined using x-ray diffraction. Surface Sci. 215, 555565 (1989).18. Maier, T., Jarrell, M., Pruschke, T. & Hettler, M. H. Quantum cluster theories. Rev. Mod. Phys. 77, 10271080 (2005).

19. Rubtsov, A. N., Savkin, V. V. & Lichtenstein, A. I. Continuous-time quantum monte carlo method for fermions. Phys. Rev. B 72, 035122 (2005).

20. Uhrberg, R. I. G. et al. Electronic structure of Sn/Si(111)

3

p 3

p : indications of a low-temperature phase. Phys. Rev. B 62, 80828086 (2000).21. Lobo, J., Tejeda, A., Mugarza, A. & Michel, E. G. Electronic structure of Sn/ Si(111)-3

p 3

p R30

as a function of Sn coverage. Phys. Rev. B 68, 235332

(2003).

22. Nakajima, T. et al. Spin-wave spectrum in single-domain magnetic ground state of triangular lattice antiferromagnet CuFeO2. J. Phys. Soc. Jpn 80, 014714 (2011).

6 NATURE COMMUNICATIONS | 4:1620 | DOI: 10.1038/ncomms2617 | http://www.nature.com/naturecommunications

Web End =www.nature.com/naturecommunications

& 2013 Macmillan Publishers Limited. All rights reserved.