Boundary values of holomorphic functions, singular unitary representations of $O(p,q)$, and their limits as $q\to\infty$

Let $\Omega$ be a bounded circular domain in $\mathbb C^N$, let $M$ be a submanifold in the boundary of $\Omega$, and let $H$ be a Hilbert space of holomorphic functions in $\Omega$. We show that, under certain conditions stated in terms of the reproducing kernel of the space $H$, the restriction operator to the submanifold $M$ is well defined for all functions from $H$. We apply this result to constructing a family of “singular” unitary representations of the groups $SO(p,q)$. The singular representations arise as discrete components of the spectrum in the decomposition of irreducible unitary highest weight representations of the groups $U(p,q)$ restricted to the subgroups $SO(p,q)$. Another property of the singular representations is that they persist in the limit as $q\to\infty$. Bibliography: 68 titles.