LAUSR

How a certain ground state of complex physical systems emerges, especially in two-dimensional materials, is a fundamental question in condensed-matter physics. A particularly interesting case is systems belonging to the class of XY Hamiltonian where the magnetic order parameter of conventional nature is unstable in two-dimensional materials leading to a Berezinskii−Kosterlitz−Thouless transition. Here, we report how the XXZ-type antiferromagnetic order of a magnetic van der Waals material, NiPS3, behaves upon reducing the thickness and ultimately becomes unstable in the monolayer limit. Our experimental data are consistent with the findings based on renormalization-group theory that at low temperatures a two-dimensional XXZ system behaves like a two-dimensional XY one, which cannot have a long-range order at finite temperatures. This work provides the experimental examination of the XY magnetism in the atomically thin limit and opens opportunities of exploiting these fundamental theorems of magnetism using magnetic van der Waals materials.Exploring the ground state of two-dimensional (2D) magnetic systems enriches the fundamental understanding of magnetism and boosts the applications. The authors here show the suppression of magnetic order and 2D XY behavior in XXZ-type antiferromagnet NiPS3 when approaching the monolayer limit by Raman spectroscopy.