Comparison Analysis of the Hexagonal Multilevel QAM and Rectangular Multilevel QAM

The performances of two multilevel modulation systems are compared in terms of capacity and cutoff rate, under the condition that the average energy per channel use is the same for the two systems. The systems are versions of the scheme of Imai and Hirakawa, and we consider the multistage suboptimal receiver for the Gaussian channel. The first system is an eight-level modulation system using rectangular QAM-signaling and a binary alphabet, and the second is a five-level modulation system using hexagonal QAM-signaling and a ternary alphabet. It is shown that, under these conditions, the capacity for the hexagonal system is close to 1 dB better than the capacity for the rectangular system for a range of signal-to-noise ratios. The Chernoff bounding parameter is also calculated for hexagonal QAM with an infinite number of signal points.