Modulation instability of two TE modes in a thin left-handed film on a nonlinear right-handed substrate

The intramode wave beams in a thin left-handed film on a Kerr substrate are considered at a frequency near zero mode group velocity. Four coupled (1 + 1)-dimensional nonlinear Schrödinger equations, describing the interaction of forward and backward propagating beams with positive and negative group velocities, are derived. It is shown that self- and cross-phase modulation for four simultaneously propagating modes is possible only at strictly matched perturbations of their propagation constants, which is due to the contribution of spatial parametric mixing. The modulation instability of only two waveguide modes is analysed for different versions of their propagation. The specific features of modulation instability, related to the propagation of modes with negative group velocities, are investigated.