Phase diagram of a frustrated two-dimensional $J_1$-$J_2$-$J_3$ quantum antiferromagnet: An approach based on spherically symmetric green's functions

The phase diagram of the frustrated$J_1$-$J_2$-$J_3$ $S= 1/2$ Heisenberg model on a square lattice is studied in the range of parameters corresponding to the spin-liquid state of the system. The study is performed using the self-consistent two-time retarded spin–spin Green's functions, which do not break both translational and $SU(2)$ symmetries. The inclusion of the damping of spin fluctuations allows us to obtain a good agreement with the cluster calculations. Using a consistent analytical approach, we have found continuous transitions via the spin–liquid state between the phases with three types of the long-range order: checkerboard, stripe, and ($k, k$) helical. In addition, there are indications confirming the existence of the ($k, \pi$) helical phase.