LAUSR

To enhance the coexistence performance of multiple-input multiple-output (MIMO) radar system and multiuser MIMO communication system with hybrid beamforming architecture, this work proposes a codesign problem regarding the digital and analog beamformers of communication base station, the receive beamformers of downlink users, and the transmit waveform of radar. An information-theoretic criterion is adopted to improve the radar detection performance, while the signal-to-interference-plus-noise ratio requirement is imposed on each downlink user to ensure communication quality. In addition, similarity restriction is also involved to manage some significant radar waveform characteristics. An iterative algorithm is introduced to tackle the resulted nonconvex and NP-hard problem. In each iteration, several subproblems are developed to decompose the original problem and are solved by the lower bound maximization and the alternating direction method of multiplier algorithm. Numerical simulations show that the proposed algorithm allows the objective function to increase monotonically and finally converge to a finite value. It also shows that the codesign scheme can effectively optimize the radar detection performance while ensuring the achievable sum rate and the bit error rate of the communication system.