LAUSR

The series of intermetallic compounds $R$NiSi$_3$ ($R$ = rare earth) shows interesting magnetic properties evolving with $R$ and metamagnetic transitions under applied magnetic field for some of the compounds. The microscopic magnetic structures must be determined to rationalize such rich behavior. Here, resonant x-ray magnetic diffraction experiments are performed on single crystals of GdNiSi$_{3}$ and TbNiSi$_{3}$ at zero field. The primitive magnetic unit cell matches the chemical cell below the Neel temperatures $T_{N}$ = 22.2 and 33.2 K, respectively. The magnetic structure is determined to be the same for both compounds (magnetic space group $Cmmm'$). It features ferromagnetic {\it ac} planes that are stacked in an antiferromagnetic $+-+-$ pattern, with the rare-earth magnetic moments pointing along the $\vec{a}$ direction, which contrasts with the $+--+$ stacking and moment direction along the $\vec{b}$ axis previously reported for YbNiSi$_3$. This indicates a sign reversal of the coupling constant between second-neighbor $R$ planes as $R$ is varied from Gd and Tb to Yb. The long {\it b} lattice parameter of GdNiSi$_{3}$ and TbNiSi$_{3}$ shows a magnetoelastic expansion upon cooling below $T_N$, pointing to the conclusion that the $+-+-$ stacking is stabilized under lattice expansion. A competition between distinct magnetic stacking patterns with similar exchange energies tuned by the size of $R$ sets the stage for the magnetic ground state instability observed along this series.