Sbornik: Mathematics
RUS  ENG    JOURNALS   PEOPLE   ORGANISATIONS   CONFERENCES   SEMINARS   VIDEO LIBRARY   PACKAGE AMSBIB  
General information
Latest issue
Forthcoming papers
Archive
Impact factor
Guidelines for authors
License agreement
Submit a manuscript

Search papers
Search references

RSS
Latest issue
Current issues
Archive issues
What is RSS



Mat. Sb.:
Year:
Volume:
Issue:
Page:
Find






Personal entry:
Login:
Password:
Save password
Enter
Forgotten password?
Register


Sbornik: Mathematics, 2022, Volume 213, Issue 8, Pages 1041–1057
DOI: https://doi.org/10.4213/sm9673e
(Mi sm9673)
 

Inner functions of matrix argument and conjugacy classes in unitary groups

Yu. A. Neretinabc

a Faculty of Mathematics, University of Vienna, Vienna, Austria
b Institute for Information Transmission Problems of the Russian Academy of Sciences (Kharkevich Institute), Moscow, Russia
c Faculty of Mechanics and Mathematics, Lomonosov Moscow State University, Moscow, Russia
References:
Abstract: Let $\mathrm{B}_n$ denote the set of complex square matrices of order $n$ whose Euclidean operator norms are less than one. Its Shilov boundary is the set $\operatorname{U}(n)$ of all unitary matrices. A holomorphic map $\mathrm{B}_m\to\mathrm{B}_n$ is inner if it sends $\operatorname{U}(m)$ to $\operatorname{U}(n)$. On the other hand we consider the group $\operatorname{U}(n+mj)$ and its subgroup $\operatorname{U}(j)$ that is embedded in $\operatorname{U}(n+mj)$ in the block-diagonal way ($m$ blocks $\operatorname{U}(j)$ and a unit block of size $n$). To any conjugacy class of $\operatorname{U}(n+mj)$ with respect to $\operatorname{U}(j)$ we assign a ‘characteristic function’, which is a rational inner map $\mathrm{B}_m\to\mathrm{B}_n$. We show that the class of inner functions that can be obtained as ‘characteristic functions’ is closed with respect to such natural operations as pointwise direct sums, pointwise products, compositions, substitutions into finite-dimensional representations of general linear groups and so on. We also describe explicitly the corresponding operations on conjugacy classes.
Bibliography: 24 titles.
Keywords: inner functions, operator colligations, classical complex domains, characteristic operator functions, transfer functions.
Funding agency Grant number
Austrian Science Fund P31591
This research was carried out with the support of Fonds zur Förderung der wissenschaftlichen Forschung (FWF) – Austrian Science Fund (grant no. P31591).
Received: 19.09.2021 and 16.02.2022
Russian version:
Matematicheskii Sbornik, 2022, Volume 213, Number 8, Pages 26–43
DOI: https://doi.org/10.4213/sm9673
Bibliographic databases:
Document Type: Article
MSC: Primary 32H02, 32M05, 32M10, 32Q02; Secondary 20G05, 47A48
Language: English
Original paper language: Russian

§ 1. Formulation of results

1.1. Some notation

Below:

Let $V$ and $W$ be linear spaces with bases $e_1,\dots,e_p$ and $f_1,\dots,f_q$, respectively. We order the elements of the basis of the tensor product $V\otimes W$ as

$$ \begin{equation*} e_1\otimes f_1, \dots, e_p\otimes f_1, \ e_1\otimes f_2, \dots, e_p\otimes f_2, \ \dots, \ e_1\otimes f_q, \dots, e_p\otimes f_q. \end{equation*} \notag $$
We write tensor products of matrices in accordance with this ordering.

1.2. Matrix balls and inner functions

Denote by $\|\cdot\|$ the operator norm in Euclidean space, that is, $\|z\|^2$ is the maximum eigenvalue of the matrix $z^*z$. Denote by $\mathrm{B}_n$ the set of complex matrices $z$ of order $n$ such that $\|z\|<1$ (matrix ball) and by $\overline{\mathrm{B}}_n$ its closure, that is, the set of matrices satisfying $\|z\|\leqslant 1$. We denote the boundary of $\mathrm{B}_n$, that is, the set of matrices with norm 1, by $\partial \mathrm{B}_n$. The unitary group $\operatorname{U}(n)$ is contained in $\partial\mathrm{B}_n$ and is the Shilov boundary of $\mathrm{B}_n$.

Recall that the pseudounitary group $\operatorname{U}(n,n)$ acts on $\mathrm{B}_n$ by biholomorphic transformations, and $\mathrm{B}_n$ is the symmetric space

$$ \begin{equation*} \mathrm B_n\simeq \operatorname{U}(n,n)\big/(\operatorname{U}(n)\times \operatorname{U}(n)) \end{equation*} \notag $$
(see, for example, [21], § 6, [15], § 2.3, and also § 2.1 below).

We say that a holomorphic map $F\colon \mathrm{B}_m\to\mathrm{B}_\alpha$ is inner if its limit values are defined almost everywhere on $\operatorname{U}(m)$ and $F$ sends $\operatorname{U}(m)$ to $\operatorname{U}(\alpha)$. Below we discuss only rational maps, so the meaning of the term ‘limit values’ is clear here.

Remark 1.1. Recall that inner functions $\mathrm{B}_1\to \mathrm{B}_1$ (that is, holomorphic maps of the unit disc $|z|<1$ to itself that also send the circle $|z|=1$ to itself) are a classical topic of the theory of functions of a complex variable (see [8], for example). Inner functions $\mathrm{B}_1\to \mathrm{B}_\alpha$ arose in the context of 1946–1954 works by Livshits on the spectral theory of operators close to unitary operators (see [11], [12], and also [23]). Potapov [22] obtained a multiplicative representation for such functions; see also [3]. Inner functions $\mathrm{B}_m\to \mathrm{B}_\alpha$ arose in [16] and [17] in representation theory of infinite-dimensional classical groups.

1.3. Colligations and characteristic functions

Fix $\alpha$ and $m\in \mathbb{N}$. Choose $j=0,1,2,\dots$ . Consider the unitary group $\operatorname{U}(\alpha+mj)$ and its subgroup $\operatorname{U}(j)$ embedded as

$$ \begin{equation*} T\mapsto \begin{pmatrix} 1_\alpha&0\\ 0& T\otimes 1_m \end{pmatrix} := \begin{pmatrix} 1_\alpha&0&0&\dots&0\\ 0&T&0&\dots&0\\ 0&0&T&\dots&0\\ \vdots&\vdots&\vdots&\ddots&\vdots\\ 0&0&0&\dots&T \end{pmatrix}\in \operatorname{U}(\alpha+mj). \end{equation*} \notag $$
Consider conjugacy classes of the group $\operatorname{U}(\alpha+mj)$ with respect to the subgroup $\operatorname{U}(j)$, that is, matrices defined up to the equivalence
$$ \begin{equation*} g\sim hgh^{-1}, \quad \text{where } g\in \operatorname{U}(\alpha+mj), \quad h\in \operatorname{U}(j). \end{equation*} \notag $$
We call such conjugacy classes colligations. Let $S\in \mathrm{B}_m$ and let $s_{\mu\nu}$ be its matrix elements. We write $g$ as a block matrix of size $\alpha+\underbrace{j+\dots+j}_{m\text{ times}}$:
$$ \begin{equation} g=\begin{pmatrix} a&b_1&\dots\\ c_1&d_{11}&\dots\\ \vdots&\vdots&\ddots \end{pmatrix} \in \operatorname{U}(\alpha+mj), \end{equation} \tag{1.1} $$
and consider the following relation:
$$ \begin{equation} \begin{pmatrix} p\\ x_1\\ \vdots\\ x_m \end{pmatrix} = \begin{pmatrix} a&b_1&\dots&b_m\\ c_1&d_{11}&\dots&d_{1m}\\ \vdots&\vdots&\ddots&\vdots\\ c_k&d_{m1}&\dots&d_{mm} \end{pmatrix} \begin{pmatrix} q\\ s_{11}x_1+\dots +s_{1m}x_m \\ \vdots \\ s_{m1}x_1+\dots +s_{mm}x_m \end{pmatrix}, \end{equation} \tag{1.2} $$
where the columns $p$ and $q$ belong to $ \mathbb{C}^\alpha$, and $x_1,\dots,x_m\in \mathbb{C}^j$. We eliminate the variables $x_1,\dots,x_m$ and obtain a dependence
$$ \begin{equation*} p =\Theta[g;S] q, \end{equation*} \notag $$
where $\Theta[g;S]$ is a rational matrix-valued function of the variable $S$ depending on the parameter $g$. By [16], Theorem 4.1, this function depends only on the conjugacy class containing $g$ and is an inner function of the matrix variable $S$. We call $\Theta[g;S]$ by the characteristic function of the colligation.

Let us repeat this definition in other terms. Set

$$ \begin{equation*} 1_j\otimes S:= \begin{pmatrix} s_{11}\cdot 1_j&\dots&s_{1m}\cdot 1_j\\ \vdots&\ddots&\vdots\\ s_{m1}\cdot 1_j&\dots&s_{mm}\cdot 1_j \end{pmatrix}. \end{equation*} \notag $$
Then
$$ \begin{equation} \Theta[g;S]=a+b(1_j\otimes S)(1_{mj}-d(1_j\otimes S) )^{-1} c. \end{equation} \tag{1.3} $$

Remark 1.2. Let $H=H_1\oplus H_2$ be a Hilbert space and $\begin{pmatrix} a&b\\c&d \end{pmatrix}$ be a unitary operator in $H$. The Livshits characteristic function, which arose in the spectral theory of non-normal operators (see [11], [12], and also [23], [7] and [6]), is given by

$$ \begin{equation} \Theta(\lambda)=a+b\lambda(1-\lambda d)^{-1}c. \end{equation} \tag{1.4} $$
In our construction this corresponds to $m=1$. We prefer to use the original term ‘characteristic function’, which emphasizes an analogy with characteristic numbers and characteristic polynomials. Our equation (1.2) can be regarded as an extension of the equation $Ax=s x$. However, the term ‘characteristic function’ is overburdened (it is commonly used in two other meanings, of an indicator function and of the Fourier transforms of a measure in probability). There is also another term for (1.4), a transfer function (see, for example, [5]), which has come from system theory.

Remark 1.3. Colligation-type structures arise in the representation theory of infinite dimensional classical groups as ‘semigroups of double cosets’. This was first observed by Olshanski [20]; see also [14], Ch. 9, §§ 3 and 4. Such semigroups act in the spaces of unitary representations of he relevant classical groups via certain operators with Gaussian kernels. A Gaussian kernel is determined by a matrix, and Olshanski showed that such matrices are given by expressions similar to matrix-valued characteristic functions of one variable. In fact, inner functions of matrix variables in [16] and [17] had a similar origin, but the starting point was a more general class of unitary representations from Nessonov’s paper [19]. From this point of view basic objects are semigroups of ‘colligations’ and inner functions are a tool for understanding colligations. In our considerations here we do not refer to representation theory and regard inner function of matrix variables and colligations as abstract topics.

1.4. A conjecture

Denote by

$$ \begin{equation*} \operatorname{Inn}(m,\alpha)=\operatorname{Inn}[\mathrm B_m,\mathrm B_\alpha] \end{equation*} \notag $$
the space of all rational inner maps $F\colon \overline{\mathrm{B}}_m\to\overline{\mathrm{B}}_\alpha$ and be $\operatorname{Inn}_\circ(m,\alpha)$ its subset consisting of the maps $F$ such that $F(\mathrm{B}_m)\subset \mathrm{B}_\alpha$. By the maximum modulus principle the last condition is equivalent to the following: for some $z_0\in\mathrm{B}_m$ we have $F(z_0)\in \mathrm{B}_\alpha$.

We also define the space

$$ \begin{equation*} \operatorname{Char}(m,\alpha)=\operatorname{Char}[\mathrm B_m,\mathrm B_\alpha] \end{equation*} \notag $$
of characteristic functions defined by all possible elements of $\operatorname{U}(\alpha+mj)$ for $j=0,1,2,\dots$ . Let $\operatorname{Char}_\circ(m,\alpha)$ denote its subset consisting of functions $\Theta[g;\cdot\,]$ sending the open ball $\mathrm{B}_m$ to the open ball $\mathrm{B}_\alpha$. In the notation (1.1), a map $\Theta[g;\cdot\,]$ belongs to $\operatorname{Char}_\circ(m,\alpha)$ if and only if $\|a\|<1$.

Conjecture 1.4. Any rational inner function is the characteristic function of some colligation, that is,

$$ \begin{equation*} \operatorname{Inn}(m,\alpha)=\operatorname{Char}(m,\alpha). \end{equation*} \notag $$

Remark 1.5. This conjecture was formulated in [17]. It is not doubtless since a similar statement for inner functions in polydiscs is false (or is only valid under some additional conditions on rational inner functions; cf. the case of a polydisc in [9] and [4]).

1.5. Operations in $\operatorname{Char}(m,\alpha)$

In this paper we show that the class $\coprod_{m,\alpha}\operatorname{Char}(m,\alpha) \subset\coprod_{m,\alpha}\operatorname{Inn}(m,\alpha)$ is closed with respect to several natural operations.1 In all cases we describe explicitly operations over colligations corresponding to operations over inner functions; these formulas are presented in proofs.

Theorem 1.6. a) Let $F_1\in \operatorname{Char}(m,\alpha)$ and $F_2\in \operatorname{Char}(m,\beta)$. Then $F_1\oplus F_2\in \operatorname{Char}(m,\alpha+\beta)$.

b) Let $F\in \operatorname{Char}(m,\alpha+\beta)$ admit a decomposition into a direct sum $F=F_1\oplus F_2$, where $F_1\in\operatorname{Inn}(m,\alpha)$ and $F_2\in\operatorname{Inn}(m,\beta)$. Then $F_1\in\operatorname{Char}(m,\alpha)$ and ${F_2\in \operatorname{Char}(m,\beta)}$.

The first (trivial) part of the statement is proved in § 3.1 and the second part in § 3.5.

The following result was obtained in [16], Theorem 4.1.

Theorem 1.7. Let $F_1,F_2\in \operatorname{Char}(m,\alpha)$. Then the pointwise product $F_1 F_2$ of matrix-valued functions $F_1$ and $F_2$ belongs to $\operatorname{Char}(m,\alpha)$.

Theorem 1.8. Let $F_1\in \operatorname{Char}(m,\alpha)$ and $F_2\in \operatorname{Char}(m,\beta)$. Then the pointwise tensor product $F_1\otimes F_2$ of matrix-valued functions $F_1$ and $F_2$ belongs to $\operatorname{Char}(m,\alpha\beta)$.

This is proved in § 3.3.

Theorem 1.9. Let $G\in \operatorname{Char}( \beta,\gamma)$ be defined by a matrix $\begin{pmatrix} a&b\\c&d \end{pmatrix}\in \operatorname{U}(\gamma+\beta j)$, and $F\in \operatorname{Char}(\alpha,\beta)$ be defined by a matrix $\begin{pmatrix} p&q\\r&t \end{pmatrix}\in \operatorname{U}(\beta+\alpha i)$. Let

$$ \begin{equation} \det(1_{\beta j}-d(1_j\otimes p))\ne 0. \end{equation} \tag{1.5} $$
Then $G\circ F\in \operatorname{Char}(\alpha,\gamma)$. More generally, the same conclusion holds if
$$ \begin{equation} \det(1_{\beta j}-d(1_j\otimes F(S_0)) )\ne 0 \quad\textit{for some } S_0\in \mathrm B_\alpha. \end{equation} \tag{1.6} $$

Remark 1.10. a) In particular, the condition (1.5) holds if $\|d\|<1$ or if $\|p\|<1$. Recall that $\|d\|\leqslant 1$, $\|p\|\leqslant 1$.

b) It can occur that the image of $F$ is contained in the discontinuity set of $G$. Condition (1.6) is sufficient (but not necessary) for avoiding this situation.

This theorem is proved in § 3.4.

Next, consider a unitary finite-dimensional representation $\rho$ of the unitary group $\operatorname{U}(n)$. Then (see, for example, [24], § 42) it admits a unique holomorphic continuation to a representation of $\operatorname{GL}(n,\mathbb{C})$. The representation $\rho$ is called polynomial if all the matrix elements of $\rho(g)$ are polynomials in matrix elements of $g\in\operatorname{GL}(n,\mathbb{C})$. Consider the semigroup $\operatorname{Mat}^\times(n)$ of all matrices of order $n$ with respect to multiplication. The group $\operatorname{GL}(n,\mathbb{C})$ is dense in $\operatorname{Mat}^\times(n)$, and all polynomial representations of $\operatorname{GL}(n,\mathbb{C})$ have a continuous extensions to the semigroup $\operatorname{Mat}^\times(n)$ (the matrices $\rho(\,\cdot\,)$ are given y the same polynomials).

Theorem 1.11. Let $\rho$ be a polynomial unitary representation of $\operatorname{U}(\alpha)$. Let $F\in\operatorname{Inn}(m,\alpha)$. Then $\rho\circ F$ is contained in $\operatorname{Inn}(m,\dim\rho)$.

This statement is proved in § 3.6.

Corollary 1.12. Let $F\in\operatorname{Inn}(m,\alpha)$. Then $\det(F)\in \operatorname{Inn}(m,1)$.

1.6. Some remarks on the behaviour of inner functions on strata of boundaries

Recall (see [21], § 6, and also § 2.3 below) that the boundary of the domain $\mathrm{B}_m\subset \operatorname{Mat}(m)$ is a disjoint union of a continual family of (open) complex manifolds (boundary components); these components are maximal complex manifolds contained in the boundary. Each component $C$ is biholomorphically equivalent to some matrix ball $\mathrm{B}_\nu$, where $\nu=0,1,\dots,m-1$, and a biholomorphic map $\mathrm{B}_\mu\to C$ extends continuously to a homeomorphism of the closures $\overline{\mathrm{B}}_\mu\to \overline C$.

The following statements are obvious.

Proposition 1.13. a) Let $F\in \operatorname{Inn}[\mathrm{B}_m,\mathrm{B}_\alpha]\setminus \operatorname{Inn}_\circ[\mathrm{B}_m,\mathrm{B}_\alpha]$. Then $F(\mathrm{B}_m)$ is contained in a unique boundary component $C\subset\mathrm{B}_\alpha$. Moreover, $F$ belongs to $\operatorname{Inn}_\circ[B_m,C]$.

b) Let $F\in \operatorname{Inn}[\mathrm{B}_m,\mathrm{B}_\alpha]$ be continuous at some point of a boundary component $C\subset \mathrm{B}_m$. Then $F\in \operatorname{Inn}[C,\mathrm{B}_\alpha]$.

The next statement is proved in § 3.7.

Theorem 1.14. a) Let $F\in \operatorname{Char}[\mathrm{B}_m,\mathrm{B}_\alpha]$ send $\mathrm{B}_m$ to a boundary component ${C\subset \overline{\mathrm{B}}_\alpha}$. Then $F\in \operatorname{Char}_\circ[\mathrm{B}_m,C]$.

b) Let $F\in \operatorname{Char}[\mathrm{B}_m,\mathrm{B}_\alpha]$ be defines by a matrix $\begin{pmatrix}a&b\\ c&d\end{pmatrix}\in \operatorname{U}(\alpha+mj)$. Let $C\subset \overline{\mathrm{B}}_m$ be a boundary component. Let $\det(1_{mj}- d(1_j\otimes S_0))\ne 0$ for some $S_0\in C$. Then the restriction of $F$ to $C$ belongs to $\operatorname{Char}[C,\mathrm{B}_\alpha]$.

1.7. On some extensions of the construction

As we mentioned above, $\mathrm{B}_n$ is a symmetric space. Our construction of inner functions can automatically be extended to the Hermitian symmetric spaces in the series

$$ \begin{equation*} \operatorname{U}(p,q)/(\operatorname{U}(p)\times \operatorname{U}(q)), \qquad \operatorname{Sp}(2n,\mathbb R)/\operatorname{U}(n)\quad\text{and} \quad \operatorname{SO}^*(2n,\mathbb R) \end{equation*} \notag $$
(see, for example, [21]; these are called classical complex domains of types I, II and III) and to direct products of such spaces (see [16] and [18]).

However, the source and target spaces in such constructions are not independent. For instance, this approach does not produce inner functions from the usual unit ball $\operatorname{U}(n,1)/(\operatorname{U}(n)\times \operatorname{U}(1))$ to the unit disc (they exist according to Aleksandrov [1] and Løw [13]), or, more generally, from $\operatorname{U}(p,q)/(\operatorname{U}(p)\times \operatorname{U}(q))$ to the unit disc for $p\ne q$ (these functions also exist according to Aleksandrov [2]).

However, it is possible to take $m=\infty$. Then the following question arises.

Question 1.15. Let $g$ be a unitary operator

$$ \begin{equation*} \mathbb C\oplus \ell^2\oplus \ell^2\,\to \mathbb C\oplus \ell^2, \end{equation*} \notag $$
that is, $gg^*=1$ and $g^*g=1$. Represent $g$ in the block form
$$ \begin{equation*} g=\left(\begin{array}{c|cc} a&b_1&b_2 \\ \hline c&d_1&d_2 \end{array}\right). \end{equation*} \notag $$
Let $(s_1,s_2)$ be a point of the open unit ball in $\mathbb{C}^2$. Set
$$ \begin{equation} \begin{aligned} \, \notag \Theta[g;(s_1,s_2)] &:=a+\begin{pmatrix} b_1& b_2\end{pmatrix} \begin{pmatrix}s_1\cdot 1_\infty\\ s_2\cdot 1_\infty\end{pmatrix} \left(1_\infty- \begin{pmatrix} d_1& d_2\end{pmatrix} \begin{pmatrix}s_1\cdot 1_\infty\\ s_2\cdot 1_\infty\end{pmatrix} \right)^{-1}c \\ &=a+\begin{pmatrix} s_1 b_1+s_2 b_2\end{pmatrix} [1- s_1d_1-s_2d_2]^{-1}c. \end{aligned} \end{equation} \tag{1.7} $$
Is it possible to find an inner function of such a type? Is it possible to find conditions on $g$ under which $\Theta[g;\cdot\,]$ is an inner function in the unit ball in $\mathbb{C}^2$?

An argument for this conjecture is very simple. We write the following relation:

$$ \begin{equation*} \begin{pmatrix} p\\ x\end{pmatrix} =\left(\begin{array}{c|cc}a&b_1&b_2\\ \hline c&d_1&d_2 \end{array}\right) \begin{pmatrix} q\\ s_1 x\\ s_2 x \end{pmatrix}. \end{equation*} \notag $$
Eliminating the variables $x$ we arrive at the relation $p=\Theta[g,(s_1,s_2)]\,q$. On the other hand the matrix $g$ is unitary, and therefore
$$ \begin{equation*} |p|^2+\|x\|^2=|q|^2+|s_1|^2 \|x\|^2+ |s_2|^2 \|x\|^2, \end{equation*} \notag $$
that is,
$$ \begin{equation*} |p|^2=|q|^2-(1-|s_1|^2-|s_2|^2)\|x\|^2. \end{equation*} \notag $$
If $|s_1|^2+|s_2|^2<1$, then $|p|\leqslant |q|$ and $|\Theta[g,(s_1,s_2)]|\leqslant1$. At first glance it seems that $|s_1|^2+|s_2|^2=1$ immediately implies $|p|=|q|$. But it is not that simple, since the matrix in square brackets in (1.7) can be noninvertible in this case.

It is easy to present examples when the function $\Theta[g;\cdot\,]$ is not inner. However a Livshits characteristic function (1.4) is not always inner either (see [23], Ch. VI, § 1).

§ 2. Preliminaries

2.1. Linear fractional maps

We realize the pseudo-unitary group $\operatorname{U}(n,n)$ as the group of all complex block matrices $g=\begin{pmatrix} A&B\\ C&D \end{pmatrix}$ of size $(n+n)$ satisfying the condition

$$ \begin{equation*} g \begin{pmatrix} -1_n&0\\ 0&1_n \end{pmatrix}g^* =\begin{pmatrix} -1_n&0\\ 0&1_n \end{pmatrix}. \end{equation*} \notag $$
For each $g\in\operatorname{U}(n,n)$ we consider the following linear fractional transformation of the space $\operatorname{Mat}(n)$:
$$ \begin{equation} \gamma[g;z]=(A+zC)^{-1}(B+zD). \end{equation} \tag{2.1} $$
Such transformations send the matrix ball $\mathrm{B}_n$ to itself (see, for example, [21], § 6, and [15], § 2.3). This action of $\operatorname{U}(n,n)$ is transitive, and the stabilizer of the point $z=0$ consists of the matrices $\begin{pmatrix}a&0\\ 0&d\end{pmatrix}$, where $a\in\operatorname{U}(n)$ and $d\in \operatorname{U}(n)$. So $\mathrm{B}_n$ is a homogeneous space:
$$ \begin{equation*} \mathrm B_n=\operatorname{U}(n,n)/(\operatorname{U}(n)\times\operatorname{U}(n)). \end{equation*} \notag $$

2.2. Realizing $\mathrm{B}_n$ as a domain in a Grassmannian

Consider the pseudo-Euclidean space

$$ \begin{equation*} V^{2n}=V_-^n\oplus V_+^n:=\mathbb C^n\oplus \mathbb C^n \end{equation*} \notag $$
equipped with the Hermitian form $\mathscr{M}=\mathscr{M}_n$ specified by the matrix $\begin{pmatrix}-1_n&0\\ 0&1_n\end{pmatrix}$. Our group $\operatorname{U}(n,n)$ preserves this form.

Denote by $\operatorname{Gr}(n)$ the Grassmannian of all $n$-dimensional subspaces in $V^{2n}$. We say that a subspace $L\in\operatorname{Gr}(n)$ is negative (semi-negative) if the form $\mathscr{M}$ is negative definite (semi-negative definite, respectively) on $L$. A subspace $L\in\operatorname{Gr}(n)$ is isotropic if the form $\mathscr{M}$ is zero on $L$. We denote the corresponding subsets of the Grassmannian by

$$ \begin{equation*} \operatorname{Gr}^{<0}(n), \qquad \operatorname{Gr}^{\leqslant 0}(n) \quad\text{and}\quad \operatorname{Gr}^{0}(n), \end{equation*} \notag $$
respectively.

For any linear map $z\colon V_-^n\to V_+^n$ we consider the $n$-dimensional space $L[z]$ of vectors of the form $(v_-,v_-z)$. If $z\in \mathrm{B}_n$, then the form $\mathscr{M}$ is negative on $L(z)$. Vice versa, any $n$-dimensional negative subspace in $V$ has the form $L(z)$ for some $z\in \mathrm{B}_n$. Formula (2.1) corresponds to the natural action of $\operatorname{U}(n,n)$ on the set of negative subspaces. Also,

$$ \begin{equation*} \overline{\mathrm B}_n\simeq \operatorname{Gr}^{\leqslant 0}(n) \quad\text{and}\quad \operatorname{U}(n)\simeq \operatorname{Gr}^0(n). \end{equation*} \notag $$

2.3. The structure of the boundary of the matrix ball $\overline{\mathrm{B}}_n$

The boundary of $\mathrm{B}_n$ consists of $n$ orbits $\mathscr{O}_j$ of the group $\operatorname{U}(n,n)$, where $j=1,2,\dots,n-1$. Orbits have representatives of the form $\begin{pmatrix} 1_j&0\\ 0& 0_{n-j} \end{pmatrix}$. The Shilov boundary $\operatorname{U}(n)$ corresponds to ${j=n}$.

In the language of the Grassmannian $\operatorname{Gr}(n)$ the orbit $\mathscr{O}_j$ corresponds to the semi-negative subspaces $L$ such that the form $\mathscr{M}$ on $L$ has rank ${n-j}$.

Using a linear fractional transformation, any component of the boundary $\partial \mathrm{B}_n$ can be reduced to the form

$$ \begin{equation*} \begin{pmatrix}u&0\\ 0& 1_j\end{pmatrix}, \quad\text{where $u$ ranges in $\mathrm B_{n-j}$}. \end{equation*} \notag $$

In the language of Grassmannians, the boundary components $C$ are enumerated by integers $j$ and $j$-dimensional isotropic subspaces $W\subset \mathbb{C}^n\oplus \mathbb{C}^n$. The corresponding component consists of all $n$-dimensional $\mathscr{M}$-semi-negative subspaces $L\supset W$ such that the kernel of $\mathscr{M}$ on $L$ coincides with $W$.

2.4. Linear relations

Let $V$ and $W$ be linear spaces. A linear relation $Y\colon {V\!\rightrightarrows\! W}$ is a linear subspace of $V\oplus W$. For linear relations $Y\colon V\rightrightarrows W$ and $Z\colon W\rightrightarrows U$ we define their product $ZY\colon V\rightrightarrows U$ to be the set of all $v\oplus u\in V\oplus U$ for which there exists $w\in W$ satisfying $v\oplus w\in Y$ and $w\oplus u\in Q$.

Let $H\subset V$ be a linear subspace. Then the subspace $YH\subset W$ consists of $w\in W$ for which there exists $v\in H$ satisfying $v\oplus w\in Y$. We can consider $H$ as the linear relation $0\rightrightarrows V$. Hence we can treat $YH$ as a product of linear relations.

For a linear relation $Y\colon V\rightrightarrows W$ we define

The product of linear relations $Y\colon V\rightrightarrows W$ and $Z\colon W\rightrightarrows Y$ is a continuous operation $(Y,Z)\to ZY$ outside the sets

$$ \begin{equation*} \ker Z\cap \operatorname{indef} Y\ne 0 \quad\text{and}\quad \operatorname{im} Y+\operatorname{dom} Z\ne W. \end{equation*} \notag $$

2.5. The isotropic category

Objects of the isotropic category (see [15], § 2.10) are spaces

$$ \begin{equation*} V^{2n}=V_+^n\oplus V_-^n\simeq \mathbb C^n\oplus\mathbb C^n, \end{equation*} \notag $$
the $n=0,1,2,\dots$ . A morphism $V^{2n}\to V^{2m}$ is a linear relation $Y\colon V^{2n}\rightrightarrows V^{2m}$ satisfying conditions

A product of morphism is the product of the corresponding linear relations. The automorphisms group of $V^{2n}$ is $\operatorname{U}(n,n)$.

We emphasize that the product has points of discontinuity.

We equip $V^{2n}\oplus V^{2m}$ with the difference of Hermitian forms in this space:

$$ \begin{equation*} \mathscr M_{m,n}(v\oplus w, v'\oplus w'):=\mathscr M_n(v,v')-\mathscr M_m(w,w'). \end{equation*} \notag $$
Then the subspace $V_-^n\oplus V_+^m\subset V^{2n}\oplus V^{2m}$ is negative with respect to the form $\mathscr{M}_{m,n}$, and the subspace $V_+^n\oplus V_-^m\subset V^{2n}\oplus V^{2m}$ is positive. So we can apply the above reasoning and obtain the following.

A relation $P\colon V^{2n}\rightrightarrows V^{2m}$ is isotropic if and only if $P$ is the graph of a unitary operator $V_-^n\oplus V_+^m\to V_+^n\oplus V_-^m$.

Thus the set of morphisms from $V^{2n}$ to $V^{2m}$ is in a one-to-one correspondence with the unitary group $\operatorname{U}(n+m)$, and the product of morphisms $Y\colon V^{2n}\rightrightarrows V^{2m}$ and $Z\colon V^{2m}\rightrightarrows V^{2k}$ induces an operation

$$ \begin{equation*} \operatorname{U}(n+m)\times \operatorname{U}(m+k)\to \operatorname{U}(n+k). \end{equation*} \notag $$

For the following statement, see, for example, [15], Theorem 2.8.4.

Proposition 2.1. Let $Y\colon V^{2k}\rightrightarrows V^{2m}$ correspond to a unitary matrix $\upsilon=\begin{pmatrix} p&q\\ r&t \end{pmatrix}\in \operatorname{U}(k+m)$ and $Z\colon V^{2m}\rightrightarrows V^{2n}$ correspond to a unitary matrix ${\zeta=\begin{pmatrix}a&b\\ c&d\end{pmatrix}\in \operatorname{U}(n+m)}$. Let

$$ \begin{equation*} \det(1-pd)^{-1}\ne 0. \end{equation*} \notag $$
Then $Z Y $ corresponds to the matrix
$$ \begin{equation} \zeta\circledast\upsilon =:\begin{pmatrix} a+b(1-pd)^{-1}pc& b(1-pd)^{-1} q \\ r(1-dp)^{-1}c&t+rd(1-pd)^{-1}q \end{pmatrix}. \end{equation} \tag{2.2} $$

2.6. Krein-Shmul’yan maps

Let $L\in\operatorname{Gr}^{\leqslant 0}(m)$. Applying a morphism $Z$: ${V^{2m}\rightrightarrows V^{2n}}$ of the isotropic category to $L$ we obtain an element of $\operatorname{Gr}^{\leqslant 0}(n)$, so that we obtain a map $\overline{\mathrm{B}}_m\to \overline{\mathrm{B}}_m$ (see [15], Theorem 2.9.1). Let $\zeta=\begin{pmatrix} a&b\\ c&d \end{pmatrix}$ be the unitary matrix corresponding to $Z$. Then the corresponding map $\sigma[\zeta]$ is given by the formula

$$ \begin{equation} \sigma[\zeta;u]\colon u\mapsto a+bu(1_m-ud)^{-1}c, \quad\text{where } u\in \overline{\mathrm B}_m. \end{equation} \tag{2.3} $$
This holds if $u$ satisfies the condition $\det(1-ud)\ne 0$.

Notice that

Remark 2.2. A map $\sigma[\zeta;z]$ is a special case of Krein-Shmul’yan maps (see [10], and also [15], § 2.9).

Remark 2.3. A map (2.3) $\mathrm{B}_m\to \mathrm{B}_m$ is inner and, moreover, its characteristic function is determined by an element of $\operatorname{U}(n+m\cdot 1)$.

Lemma 2.4. Let $\zeta$ and $\upsilon$ be the same as in Proposition 2.1. Then for any $u\in \mathrm{B}_k$

$$ \begin{equation} \sigma[\zeta;\,\sigma[\upsilon;\,u] ]=\sigma[\zeta\circledast \upsilon;\, u ]. \end{equation} \tag{2.4} $$

Remark 2.5. Cf. [15], Theorem 2.9.4, but the assumptions of this theorem are not satisfied here. Basically, Lemma 2.4 claims the associativity of the product of linear relations $0\rightrightarrows V^{2k}\rightrightarrows V^{2m}\rightrightarrows V^{2n}$. However, formula (2.2) is not valid on the surface $\det(1-pd)=0$. To avoid giving references to proofs or repeating proofs, we present a formal calculation.

Proof of Lemma 2.4. We must transform the following expression to the Krein-Shmul’yan form:
$$ \begin{equation} a+b\, \bigl(u(1-tu)^{-1} |_{u=p+qz(1-tz)^{-1}r} \bigr)\, c. \end{equation} \tag{2.5} $$

Step 1. It is sufficient to examine the expression in large parentheses; it is a sum $I+J$ of two terms,

$$ \begin{equation*} I:=p [1-dq-dq z(1-tz)^{-1}r]^{-1} \end{equation*} \notag $$
and
$$ \begin{equation*} J:=qz(1-tz)^{-1} [1-dq-dq z(1-tz)^{-1}r]^{-1}. \end{equation*} \notag $$

First we must show that the inverse matrix $[\,{\dots}\,]^{-1}$ exists. Since $(1-dp)^{-1}$ is invertible, we can transform $[\,{\dots}\,]^{-1}$ as follows:

$$ \begin{equation} [\,{\dots}\,]^{-1}=(1-dq)^{-1}(1- dq z(1-tz)^{-1}\cdot r(1-dq)^{-1})^{-1}. \end{equation} \tag{2.6} $$
Next we notice that two matrices $(1-AB)$ and $(1-BA)$ are invertible or not simultaneously. Therefore, it is sufficient to verify the existence of the matrix
$$ \begin{equation*} (1- r(1-dq)^{-1}\cdot dq z(1-tz)^{-1})^{-1} =(1-tz) (1- \{t+r(1-dq)^{-1}\cdot dq\}\cdot z)^{-1}. \end{equation*} \notag $$
Since $\|t\|\leqslant 1$ and $\|z\|<1$, the matrix $(1-tz)$ is invertible. Next, the expression in curly brackets coincides with the lower right-hand block of the matrix (2.2). Therefore, $\|\{\,{\dots}\,\}\|\leqslant 1$ and the second factor is well defined.

Step 2. Transforming the term $J$ using (2.6) we obtain

$$ \begin{equation*} J=q z(1-tz)^{-1}\cdot r(1-dq)^{-1}(1- dq z(1-tz)^{-1}\cdot r(1-dq)^{-1})^{-1}. \end{equation*} \notag $$
Bearing in mind the matrix identity
$$ \begin{equation*} A(1-BA)^{-1}=(1-AB)^{-1}A, \end{equation*} \notag $$
we arrive at
$$ \begin{equation*} \begin{aligned} \, J&=q z(1-tz)^{-1}\cdot (1- r(1-dq)^{-1}\cdot dq z(1-tz)^{-1})^{-1}r(1-dq)^{-1} \\ &=qz (1- \{t+r(1-dq)^{-1}\cdot dq\}\cdot z)^{-1}r(1-dq)^{-1}. \end{aligned} \end{equation*} \notag $$
Next we transform the term $I$ using (2.6) and apply the identity
$$ \begin{equation} (1-C)^{-1}=1+C(1-C)^{-1} \end{equation} \tag{2.7} $$
to the second factor in (2.6). We arrive at
$$ \begin{equation*} \begin{aligned} \, I&=p(1-dp)^{-1} \\ &\qquad + p(1-dp)^{-1} d\cdot q z(1-tz)^{-1} r(1-dq)^{-1} (1- dq z(1-tz)^{-1} r(1-dq)^{-1})^{-1} \\ &=p(1-dp)^{-1}+ p(1-dp)^{-1} d\cdot J. \end{aligned} \end{equation*} \notag $$
Thus,
$$ \begin{equation*} \begin{aligned} \, &I+J =p(1-dp)^{-1}+\{p(1-pd)^{-1}d+1\}\cdot J \\ &\ =(1-pd)^{-1}p+\{(1-pd)^{-1}\}\cdot qz (1- \{t+r(1-dq)^{-1}\cdot dq\}\cdot z)^{-1}r(1-dq)^{-1} \end{aligned} \end{equation*} \notag $$
(we have applied (2.7) to the expression in curly brackets). We substitute the result into (2.5) for the expression in large parentheses and obtain the required formula. The lemma is proved.

2.7. Characteristic functions and Krein-Shmul’yan maps

Formula (1.3) can be written as

$$ \begin{equation} \Theta\biggl[\begin{pmatrix} a&b\\ c&d \end{pmatrix};S \biggr]= \sigma\biggl[\begin{pmatrix} a&b\\ c&d \end{pmatrix};1_j\otimes S\biggr]. \end{equation} \tag{2.8} $$

2.8. A polynomial representations for $\operatorname{GL}(n, {\mathbb{C}})$

Irreducible holomorphic representations $\rho_\mathbf{m}(g)$ of $\operatorname{GL}(n,\mathbb{C})$ are enumerated by ‘signatures’

$$ \begin{equation*} \mathbf m:=(m_1, \dots, m_n), \quad \text{where } m_j\in \mathbb Z \text{ and } m_1\geqslant m_2\geqslant\dots\geqslant m_n \end{equation*} \notag $$
(see, for example, [24], §§ 49 and 50). Recall a semi-explicit construction of $\rho_\mathbf{m}$.

Consider the space $\mathbb{C}^n$ with the standard basis $e_1,\dots,e_n$. The representation $\lambda_k(g)$ with signature $(\,\underbrace{1,\dots,1}_{k\text{ times}},0,\dots,0)$ is the representation in the $k$th exterior power $\bigwedge^k \mathbb{C}^n$; its highest weight vector is $v_k=e_1\wedge \cdots \wedge e_k$. The matrix element $\langle \rho(g)v_k,v_k\rangle$ is the $k$th principal minor $\Delta_k(g)$ of $g$. The representation $\lambda_n$ is simply $\det(g)$ (in particular, we can consider its negative tensor powers).

The representation $\rho_\mathbf{m}$ is a subrepresentation of

$$ \begin{equation} \bigotimes_{k=1}^{n-1} \lambda_k^{\otimes (m_k-m_{k+1})}(g)\otimes\lambda_k(g)^{m_n}. \end{equation} \tag{2.9} $$
More precisely, $\rho_\mathbf{m}$ is the cyclic span of the highest weight vector
$$ \begin{equation*} \xi_\mathbf m:=\bigotimes_{k=1}^{n-1} (e_1\wedge \dots \wedge e_k)^{\otimes( m_k-m_{k+1})}\otimes (e_1\wedge\dots\wedge e_n)^{\otimes m_n}. \end{equation*} \notag $$
The matrix element
$$ \begin{equation*} \langle \rho_\mathbf m(g)\xi_\mathbf m,\,\xi_\mathbf m\rangle=\prod_{k=1}^{n-1} \Delta_k^{m_k-m_{k+1}}\cdot \det(g)^{m_n} \end{equation*} \notag $$
is polynomial if and only if $m_n\geqslant 0$. On the other hand, for $m_n\geqslant 0$ the representation $\rho_\mathbf{m}$ is a polynomial representation by construction.

§ 3. Proofs

3.1. Direct sums

Proof of Theorem 1.6, a). Consider an element $g\in \operatorname{U}(\alpha+mi)$ written as
$$ \begin{equation} g=\left(\begin{array}{c|ccc} a&b_1&b_2&\dots \\ \hline c_1&d_{11}&d_{12}&\dots \\ c_2&d_{21}&d_{22}&\dots \\ \vdots&\vdots&\vdots&\ddots \end{array}\right), \end{equation} \tag{3.1} $$
and an element $\widetilde g\in \operatorname{U}(\beta+mj)$ written as
$$ \begin{equation} \widetilde g=\left(\begin{array}{c|ccc} \widetilde a&\widetilde b_1&\widetilde b_2&\dots \\ \hline \widetilde c_1&\widetilde d_{11}&\widetilde d_{12}\vphantom{\widetilde {A^{A^A}}}&\dots \\ \widetilde c_2&\widetilde d_{21}&\widetilde d_{22}&\dots \\ \vdots&\vdots&\vdots&\ddots \end{array} \right). \end{equation} \tag{3.2} $$
Consider the block matrix of order
$$ \begin{equation*} \alpha+\beta+i+j+\dots+i+j=(\alpha+\beta)+\underbrace{(i+j)+\dots+(i+j)}_{m\text{ times}} \end{equation*} \notag $$
given by
$$ \begin{equation*} g(\oplus)\widetilde g:= \left( \begin{array}{cc|ccccc} a&0&b_1&0&b_2&0&\dots \\ 0&\widetilde a&0&\widetilde b_1&0&\widetilde b_2&\dots \\ \hline c_1&0&d_{11}&0&d_{12}&0&\dots \\ 0&\widetilde c_1&0&\widetilde d_{11}&0&\widetilde d_{12}&\dots \\ c_2&0&d_{21}&0&d_{22}&0&\dots \\ 0&\widetilde c_2&0&\widetilde d_{21}&0&\widetilde d_{22}&\dots \\ \vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\ddots \end{array}\right). \end{equation*} \notag $$
By formula (2.8),
$$ \begin{equation*} \Theta[g(\oplus)\widetilde g;S]=\Theta[g;S]\oplus\Theta[\widetilde g;S] \end{equation*} \notag $$
(we have applied the Krein-Shmul’yan map determined by the matrix $g(\oplus)\widetilde g$ to the matrix $S\otimes 1_{i+j}$).

3.2. Pointwise products

Theorem 1.7 was obtained in [16]. We present here the corresponding operation on colligations for completeness. Let $g\in \operatorname{U}(\alpha+mi)$ be represented as in (3.1) and $\widetilde g\in \operatorname{U}(\alpha+mj)$ be represented as in (3.2). We define the matrix $g\odot\widetilde g$ by

$$ \begin{equation} g\odot\widetilde g\,{:=}\, \left(\begin{array}{c|ccccc} a&b_1&0&b_2&0&\dots \\ \hline c_1&d_{11}&0&d_{12}&0&\dots \\ 0&0&1_j&0&0&\dots \\ c_2&d_{21}&0&d_{22}&0&\dots \\ 0&0&0&0&1_j&\dots \\ \vdots&\vdots&\vdots&\vdots&\vdots&\ddots \end{array}\right) \left(\begin{array}{c|ccccc} \widetilde a&0&\widetilde b_1&0&\widetilde b_2&\dots \\ \hline 0&1_i&0&0&0&\dots \\ \widetilde c_1&0&\widetilde d_{11}&0&\widetilde d_{12}&\dots \\ 0&0&0&1_i&0&\dots \\ \widetilde c_2&0&\widetilde d_{21}&0&\widetilde d_{22}&\dots \\ \vdots&\vdots&\vdots&\vdots&\vdots&\ddots \end{array} \right). \end{equation} \tag{3.3} $$
Then
$$ \begin{equation*} \Theta[g\odot \widetilde g;S]=\Theta[g;S]\,\Theta[\widetilde g;S]. \end{equation*} \notag $$

3.3. Pointwise tensor products

Proof of Theorem 1.8. This is a corollary of the previous statement. Since
$$ \begin{equation*} A\otimes B=(1\otimes B)\cdot(A\otimes 1), \end{equation*} \notag $$
it is sufficient to verify the claim for $F_1\otimes 1$ and $1\otimes F_2$.

The function $F_1\otimes 1$ is a direct sum of several copies of the function $F_1$. By Theorem 1.6 it is a characteristic function. More precisely, if $F_1$ is generated by a matrix $\begin{pmatrix}p&q\\ r&t\end{pmatrix}$, then

$$ \begin{equation} \begin{aligned} \, \notag &\Theta\left[\begin{pmatrix}p&q_1&\dots&q_m\\ r_1&t_{11}&\dots&t_{1m}\\ \vdots&\vdots&\ddots&\vdots\\ r_m&t_{m1}&\dots&t_{mm} \end{pmatrix};S \right]\otimes 1_\beta \\ &\qquad =\Theta\left[\begin{pmatrix} p\otimes 1_\beta&q_1\otimes 1_\beta&\dots&q_m\otimes 1_\beta\\ r_1\otimes 1_\beta&t_{11}\otimes 1_\beta&\dots&t_{1m}\otimes 1_\beta\\ \vdots&\vdots&\ddots&\vdots\\ r_m\otimes 1_\beta&t_{m1}\otimes 1_\beta&\dots&t_{mm}\otimes 1_\beta \end{pmatrix};S \right]. \end{aligned} \end{equation} \tag{3.4} $$

The identity

$$ \begin{equation} 1_\alpha\otimes\Theta\left[\begin{pmatrix}a&b\\ c&d\end{pmatrix};S \right] = \Theta\left[\begin{pmatrix}1_\alpha\otimes a&1_\alpha\otimes b \\ 1_\alpha\otimes c&1_\alpha\otimes d\end{pmatrix};S \right] \end{equation} \tag{3.5} $$
immediately follows from (2.8). Therefore,
$$ \begin{equation*} \Theta\left[\begin{pmatrix} p&q\\ r&t \end{pmatrix};S \right]\otimes \Theta\left[\begin{pmatrix} a&b\\ c&d \end{pmatrix};S \right] \end{equation*} \notag $$
is the characteristic function of the colligation
$$ \begin{equation} \left( \begin{array}{c|ccccc} p\otimes a&q_1\otimes 1_\beta&p\otimes b_1& q_2\otimes 1_\beta&p\otimes b_2&\dots\\ \hline r_1\otimes a& t_{11}\otimes 1_\beta& r_1\otimes b_1& t_{12}\otimes 1_\beta& r_1\otimes b_2&\dots\\ 1_\alpha\otimes c_1&0&1_\alpha\otimes d_{11}& 0&1_\alpha\otimes d_{12}&\dots\\ r_2\otimes a& t_{21}\otimes 1_\beta& r_2\otimes b_1& t_{22}\otimes 1_\beta& r_2\otimes b_2&\dots\\ 1_\alpha\otimes c_2&0&1_\alpha\otimes d_{21}& 0&1_\alpha\otimes d_{22}&\dots\\ \vdots& \vdots&\vdots&\vdots&\vdots&\ddots \end{array} \right). \end{equation} \tag{3.6} $$
The theorem is proved.

3.4. Compositions

Proof of Theorem 1.9. By (2.8) and (3.5) we have
$$ \begin{equation*} \begin{aligned} \, G\circ F(S) &=\sigma\left[\begin{pmatrix}a&b\\ c&d\end{pmatrix}; 1_j\otimes \sigma\left[\begin{pmatrix} p&q\\ r&t \end{pmatrix};1_i\otimes S \right]\right] \\ &= G\circ F(S)=\sigma\left[\begin{pmatrix}a&b\\ c&d\end{pmatrix}; \sigma\left[\begin{pmatrix} 1_j\otimes p&1_j\otimes q\\ 1_j\otimes r&1_j\otimes t \end{pmatrix};1_i\otimes S \right]\right]. \end{aligned} \end{equation*} \notag $$
If $\det(1_{\beta j}-d(1_j\otimes p))\ne 0$ (condition (1.5)), then we can apply formula (2.2) and Lemma 2.4. In this case we arrive at
$$ \begin{equation*} \sigma\left[\begin{pmatrix}a&b\\ c&d\end{pmatrix} \circledast \begin{pmatrix} 1_j\otimes p&1_j\otimes q\\ 1_j\otimes r&1_j\otimes t \end{pmatrix}; 1_{ij}\otimes S \right], \end{equation*} \notag $$
where the $\circledast$-product of matrices is defined by (2.2). Since both ‘factors’ are unitary matrices, we obtain an inner map $\mathrm{B}_\alpha\to \mathrm{B}_\gamma$.

Corollary 3.1. Let $F\in \operatorname{Char}[m,\alpha]$. Then the following hold.

a) For $h\in \operatorname{U}(\alpha, \alpha)$ we have $\gamma[h]\circ F\in \operatorname{Char}[m,\alpha]$.

b) For $h'\in \operatorname{U}(m,m)$ we have $F\circ \gamma[h']\in \operatorname{Char}[m,\alpha]$.

Indeed, in these cases our condition holds.

Let us finish the proof of Theorem 1.9. Let condition (1.6) hold, that is, let $\det(1_{\beta j}-d(1_j\otimes F(S_0)) )\ne 0$. We take $h\in \operatorname{U}(\alpha,\alpha)$ that sends 0 to $S_0$; then $F(\gamma[h;0])=F(S_0)$. We apply Corollary 3.1 to $G\circ(F\circ \gamma[h])$ and refer to the already proved part of Theorem 1.9. Thus, $G\circ (F\circ \gamma[h])\in\operatorname{Char}(\alpha,\gamma)$. Next we apply Corollary 3.1 to $(G\circ F\circ \gamma[h])\circ \gamma[h^{-1}]$ and obtain the desired statement. Theorem 1.9 is proved.

3.5. Splitting off summands

Proof of Theorem 1.6, b). Let the characteristic function $F\in\operatorname{Char}[m,\alpha+\beta]$ have the block form
$$ \begin{equation*} F(z):=\begin{pmatrix} F_1(z)&0\\ 0&F_2(z) \end{pmatrix}. \end{equation*} \notag $$
We show that $F_1\in \operatorname{Char}[m,\alpha]$.

First assume that $F_2\in \operatorname{Inn}_\circ[m,\beta]$. We consider the Krein-Shmul’yan map $G\colon \mathrm{B}_{\alpha+\beta}\to \mathrm{B}_\alpha$ determined by the matrix

$$ \begin{equation} \left(\begin{array}{c|cc} 0&1_\alpha&0\\ \hline 1_\alpha &0&0\\ 0&0&1_\beta \end{array}\right), \end{equation} \tag{3.7} $$
and take the composition $G\circ F$:
$$ \begin{equation*} \begin{aligned} \, G\circ F(z) &=\begin{pmatrix}1_\alpha&0 \end{pmatrix} \begin{pmatrix}F_1(z)&0\\ 0&F_2(z)\end{pmatrix} \\ &\qquad\times \left\{\begin{pmatrix}1_\alpha&0\\ 0&1_\beta\end{pmatrix} - \begin{pmatrix}0&0\\ 0&1_\beta \end{pmatrix} \begin{pmatrix}F_1(z)&0\\ 0&F_2(z)\end{pmatrix}\right\}^{-1} \begin{pmatrix}1_\alpha\\ 0 \end{pmatrix}. \end{aligned} \end{equation*} \notag $$
The matrix in curly brackets is
$$ \begin{equation*} \begin{pmatrix}1_\alpha&0\\ 0&1_\beta-F_2(z) \end{pmatrix}. \end{equation*} \notag $$
It is invertible, and by Theorem 1.9 the map $G\circ F$ belongs to $\operatorname{Char}[m,\alpha]$. But
$$ \begin{equation*} G\circ F=F_1(z), \end{equation*} \notag $$
and this implies our statement.

Second, let $F_2\notin \operatorname{Inn}_\circ(m,\alpha)$. Then $F_2(\mathrm{B}_m)$ is contained in some component $C$ of the boundary of $\mathrm{B}_\beta$. By Corollary 3.1 we can assume that $C$ is in a canonical form, that is, $C$ consists of the matrices

$$ \begin{equation*} \begin{pmatrix} u&0\\ 0&1_{k} \end{pmatrix}, \quad\text{where } u\in \mathrm B_{\beta-k}, \end{equation*} \notag $$
and therefore $F_2(z)$ has the form
$$ \begin{equation} F_2(z)=\begin{pmatrix} R(z)&0\\ 0&1_{k} \end{pmatrix}, \quad\text{where } \|R(z)\|<1 \quad\text{for } z\in \mathrm B_m. \end{equation} \tag{3.8} $$
Now we choose $\lambda\in\mathbb{C}$ such that $|\lambda|=1$ and $\lambda\ne1$. Instead of (3.7) we take the matrix
$$ \begin{equation*} \left(\begin{array}{c|cc} 0&1_\alpha&0\\ \hline 1_\alpha &0&0\\ 0&0&\lambda\cdot 1_\beta \end{array} \right) \end{equation*} \notag $$
and the corresponding Krein-Shmul’yan map $G$. By (3.8) the matrix
$$ \begin{equation*} \left\{\begin{pmatrix}1_\alpha&0\\ 0&1_\beta\end{pmatrix} - \begin{pmatrix}0&0\\ 0&\lambda 1_\beta \end{pmatrix} \begin{pmatrix}F_1(z)&0\\ 0&F_2(z)\end{pmatrix}\right\} = \begin{pmatrix}1_\alpha&0\\ 0&1_\beta- \lambda F_2(z)\end{pmatrix} \end{equation*} \notag $$
is invertible, and by Theorem 1.9 we have $G\circ F=F_1(z)\in \operatorname{Char}(m,\alpha)$. The theorem is proved.

3.6. Compositions with polynomial representations

Proof of Theorem 1.11. By Theorem 1.6, a), it is sufficient to consider irreducible representations. By the construction given in § 2.8 any irreducible polynomial representation $\rho_\mathbf{m}=\rho_{m_1,\dots,m_n}$ is contained in the tensors
$$ \begin{equation*} \bigotimes_{k=1}^n \biggl(\bigwedge^k \mathbb C^n \biggr)^{\otimes(m_k-m_{k-1})}\,\otimes\, \biggl(\bigwedge^n\mathbb C^n \biggr)^{m_n}\subset (\mathbb C^n)^{\otimes( \sum_{k=1}^n m_k)}. \end{equation*} \notag $$
By Theorem 1.8, for any characteristic function $F(z)$ and any $L>0$ the function $F(z)^{\otimes L}$ is a characteristic function. By Theorem 1.6, b), we can split off a direct summand. The theorem is proved.

3.7. Boundary components

Proof of Theorem 1.14. a) Without loss of generality we can assume that $C$ has the canonical form $\begin{pmatrix} u&0\\ 0&1_k \end{pmatrix}$. Therefore, our function $F$ decomposes into a direct sum. By Theorem 1.6, b), we can split off a summand.

b) Again, we can assume that $C\subset \mathrm{B}_m$ consists of the matrices $\begin{pmatrix} u&0\\ 0&1_l \end{pmatrix} $. As in § 3.4, we can assume that $S_0:=\begin{pmatrix} 0&0\\ 0&1_l \end{pmatrix}$. The identical embedding $u\mapsto \begin{pmatrix} u&0\\ 0&1_l \end{pmatrix}$ is the Krein-Shmul’yan map determined by the matrix

$$ \begin{equation*} \left(\begin{array}{cc|c} 0&0& 1_{m-l}\\ 0&1_l&0\\ \hline 1_{m-l}&0&0 \end{array}\right). \end{equation*} \notag $$
Now we can apply Theorem 1.9. Theorem 1.14 is proved.


Bibliography

1. A. B. Aleksandrov, “The existence of inner functions in the ball”, Mat. Sb. (N.S.), 118(160):2(6) (1982), 147–163  mathnet  mathscinet  zmath; English transl. in Sb. Math., 46:2 (1983), 143–159  crossref
2. A. B. Aleksandrov, “Inner functions on compact spaces”, Funktsional. Anal. i Prilozhen., 18:2 (1984), 1–13  mathnet  mathscinet  zmath; English transl. in Funct. Anal. Appl., 18:2 (1984), 87–98  crossref
3. D. Alpay, An advanced complex analysis problem book. Topological vector spaces, functional analysis, and Hilbert spaces of analytic functions, Birkhäuser/Springer, Cham, 2015, ix+520 pp.  crossref  mathscinet  zmath
4. J. A. Ball and V. Bolotnikov, “Canonical transfer-function realization for Schur-Agler-class functions of the polydisk”, A panorama of modern operator theory and related topics, Birkhäuser/Springer Basel AG, Basel, 2012, 75–122  crossref  mathscinet  zmath
5. H. Bart, “Transfer functions and operator theory”, Linear Algebra Appl., 84 (1986), 33–61  crossref  mathscinet  zmath
6. M. S. Brodskii, “Unitary operator colligations and their characteristic functions”, Uspekhi Mat. Nauk, 33:4(202) (1978), 141–168  mathnet  mathscinet  zmath; English transl. in Russian Math. Surveys, 33:4 (1978), 159–191  crossref
7. V. M. Brodskiĭ, “On operator nodes and their characteristic functions”, Dokl. Akad. Nauk SSSR, 198:1 (1971), 16–19  mathnet  mathscinet  zmath; English transl. in Soviet Math. Dokl., 12 (1971), 696–700
8. J. B. Garnett, Bounded analytic functions, Pure Appl. Math., 96, Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], New York–London, 1981, xvi+467 pp.  mathscinet  zmath
9. G. Knese, “Rational inner functions in the Schur-Agler class of the polydisk”, Publ. Mat., 55:2 (2011), 343–357  crossref  mathscinet  zmath
10. M. G. Kreĭn and Ju. L. Šmul'jan (Shmul'yan), “On linear-fractional transformations with operator coefficients”, Mat. Issled. (Kishinev), 2:3 (1967), 64–96  mathscinet  zmath; English transl. in Amer. Math. Soc. Transl. Ser. 2, 103, Amer. Math. Soc., Providence, RI, 1974, 125–152  crossref
11. M. S. Livšic (Livshits), “On a class of linear operators in Hilbert space”, Mat. Sb., 19(61):2 (1946), 239–262  mathnet  mathscinet  zmath; English transl. in Amer. Math. Soc. Transl. Ser. 2, 13, Amer. Math. Soc., Providence, RI, 1960, 61–83  crossref  mathscinet  zmath
12. M. S. Livšic (Livshits), “On the spectral decomposition of linear non-selfadjoint operators”, Mat. Sb., 34(76):1 (1954), 145–199  mathnet  mathscinet  zmath; English transl. in Amer. Math. Soc. Transl. Ser. 2, 5, Amer. Math. Soc., Providence, RI, 1957, 67–114  crossref  mathscinet  zmath
13. E. Løw, “A construction of inner functions on the unit ball in $C^p$”, Invent. Math., 67:2 (1982), 223–229  crossref  mathscinet  zmath  adsnasa
14. Yu. A. Neretin, Categories of symmetries and infinite-dimensional groups, Èditorial URSS, Moscow, 1998, 431 pp.; English transl. London Math. Soc. Monogr. (N.S.), 16, The Clarendon Press, Oxford Univ. Press, New York, 1996, xiv+417 pp.  mathscinet  zmath
15. Yu. A. Neretin, Lectures on Gaussian integral operators and classical groups, EMS Ser. Lect. Math., Eur. Math. Soc. (EMS), Zürich, 2011, xii+559 pp.  crossref  mathscinet  zmath
16. Yu. A. Neretin, “Multi-operator colligations and multivariate characteristic functions”, Anal. Math. Phys., 1:2–3 (2011), 121–138  crossref  mathscinet  zmath
17. Yu. A. Neretin, “Sphericity and multiplication of double cosets for infinite-dimensional classical groups”, Funktsional. Anal. i Prilozhen., 45:3 (2011), 79–96  mathnet  crossref  mathscinet  zmath; English transl. in Funct. Anal. Appl., 45:3 (2011), 225–239  crossref
18. Yu. A. Neretin, “Multiplication of conjugacy classes, colligations, and characteristic functions of matrix argument”, Funktsional. Anal. i Prilozhen., 51:2 (2017), 25–41  mathnet  crossref  mathscinet  zmath; English transl. in Funct. Anal. Appl., 51:2 (2017), 98–111  crossref
19. N. I. Nessonov, “Factor-representation of the group $GL(\infty)$ and admissible representations $GL(\infty)^X$”, Mat. Fiz. Anal. Geom., 10:2 (2003), 167–187 (Russian)  mathnet  mathscinet  zmath
20. G. I. Ol'shanskiĭ (Olshanski), “Unitary representations of infinite-dimensional pairs $(G,K)$ and the formalism of R. Howe”, Representation of Lie groups and related topics, Adv. Stud. Contemp. Math., 7, Gordon and Breach, New York, 1990, 269–463  mathscinet  zmath
21. I. I. Piatetskii-Shapiro, Geometry of classical domains and the theory of discrete groups, Fizmatlit, Moscow, 1961, 191 pp.  mathscinet  zmath; English version I. I. Piatetskii-Shapiro, Automorphic functions and the geometry of classical domains, Math. Appl., 8, Gordon and Breach Science Publishers, New York–London–Paris, 1969, viii+264 pp.  mathscinet  zmath
22. V. P. Potapov, “The multiplicative structure of $J$-contractive matrix functions”, Tr. Mosk. Mat. Obshch., 4, GITTL, Moscow, 1955, 125–236  mathnet  mathscinet  zmath; English transl. in Amer. Math. Soc. Transl. Ser. 2, 15, Amer. Math. Soc., Providence, RI, 1960, 131–243  crossref  mathscinet  zmath
23. B. Sz.-Nagy and C. Foiaş, Analyse harmonique des opérateurs de l'espace de Hilbert, Akadémiaí Kiadó, Budapest; Masson et Cie, Paris, 1967, xi+373 pp.  mathscinet  zmath
24. D. P. Želobenko, Compact Lie groups and their representations, Nauka, Moscow, 1970, 664 pp.  mathscinet  zmath; English transl. in Transl. Math. Monogr., 40, Amer. Math. Soc., Providence, RI, 1973, viii+448 pp.  mathscinet  zmath

Citation: Yu. A. Neretin, “Inner functions of matrix argument and conjugacy classes in unitary groups”, Mat. Sb., 213:8 (2022), 26–43; Sb. Math., 213:8 (2022), 1041–1057
Citation in format AMSBIB
\Bibitem{Ner22}
\by Yu.~A.~Neretin
\paper Inner functions of matrix argument and conjugacy classes in unitary groups
\jour Mat. Sb.
\yr 2022
\vol 213
\issue 8
\pages 26--43
\mathnet{http://mi.mathnet.ru/sm9673}
\crossref{https://doi.org/10.4213/sm9673}
\mathscinet{http://mathscinet.ams.org/mathscinet-getitem?mr=4461462}
\adsnasa{https://adsabs.harvard.edu/cgi-bin/bib_query?2022SbMat.213.1041N}
\transl
\jour Sb. Math.
\yr 2022
\vol 213
\issue 8
\pages 1041--1057
\crossref{https://doi.org/10.4213/sm9673e}
\isi{https://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcApp=Publons&SrcAuth=Publons_CEL&DestLinkType=FullRecord&DestApp=WOS_CPL&KeyUT=000992270000002}
\scopus{https://www.scopus.com/record/display.url?origin=inward&eid=2-s2.0-85165906885}
Linking options:
  • https://www.mathnet.ru/eng/sm9673
  • https://doi.org/10.4213/sm9673e
  • https://www.mathnet.ru/eng/sm/v213/i8/p26
  • Citing articles in Google Scholar: Russian citations, English citations
    Related articles in Google Scholar: Russian articles, English articles
    Математический сборник Sbornik: Mathematics
    Statistics & downloads:
    Abstract page:289
    Russian version PDF:23
    English version PDF:39
    Russian version HTML:130
    English version HTML:63
    References:59
    First page:6
     
      Contact us:
     Terms of Use  Registration to the website  Logotypes © Steklov Mathematical Institute RAS, 2024