Boundary conditions and surface state spectra in topological insulators

We study surface states spectra in 2D topological insulators (TIs) based on HgTe/(Hg,Cd)Te quantum wells and 3D Bi$_2$Se$_3$-type compounds by constructing a class of feasible time-reversal invariant boundary conditions (BCs) for an effective ${\mathbf k}{\mathbf p}$-Hamiltonian and a tight-binding model of the topological insulators. The BCs contain some phenomenological parameters which implicitly depend on both bulk Hamiltonian parameters and crystal potential behavior near the crystal surface. Space symmetry reduces the number of the boundary parameters to four real parameters in the 2D case and three in the 3D case. We found that the boundary parameters may strongly affect not only an energy spectrum but even the very existence of these states inside the bulk gap near the Brillouin zone center. Nevertheless, we reveal in frames of the tight-binding model that when surface states do not exist in the bulk gap in the Brillouin zone center they cross the gap in other points of the Brillouin zone in agreement with the bulk-boundary correspondence.