LAUSR

Based on the first-principles calculations and theoretical analysis, we investigate the electronic structures, topological phase transition (TPT), and topological properties of the layered magnetic compound ${\mathrm{MnSb}}_{2}{\mathrm{Te}}_{4}$. We have synthesized a ${\mathrm{MnSb}}_{2}{\mathrm{Te}}_{4}$ sample and determined its crystal structure. It has a crystal similar to that of the magnetic topological insulator ${\mathrm{MnBi}}_{2}{\mathrm{Te}}_{4}$ but has Mn and Sb site mixing. For the ideal case without such site mixing, our calculation indicates ${\mathrm{MnSb}}_{2}{\mathrm{Te}}_{4}$ is antiferromagnetic (AFM), and there is no band inversion at $\mathrm{\ensuremath{\Gamma}}$. The band inversion can be realized by increasing the spin-orbit coupling (SOC) of Sb by more than $30%$, and this results in a TPT from a trivial AFM insulator to an AFM topological insulator or an axion insulator. The compressive uniaxial strain can also drive a similar TPT if the interlayer distance is shortened by more than $5%$. For the ferromagnetic (FM) case without Mn and Sb site mixing, it is a normal FM insulator. The band inversion can happen when SOC is enhanced by $10%$ or the interlayer distance is decreased by more than $1%$. Thus, FM ${\mathrm{MnSb}}_{2}{\mathrm{Te}}_{4}$ can be tuned to be the simplest type-I Weyl semimetal with only one pair of Weyl nodes on the threefold rotational axis, which is different from the proposal that Mn and Sb site mixing can result in a ferrimagnetic state and a type-II Weyl semimetal state.