On the groups of unitriangular automorphisms of relatively free groups

We describe the structure of the group $U_n$ of unitriangular automorphisms of the relatively free group $G_n$ of finite rank $n$ in an arbitrary variety $\mathscr C$ of groups. This enables us to introduce an effective concept of normal form for the elements and present $U_n$ by using generators and defining relations. The cases $n=1,2$ are obvious: $U_1$ is trivial, and $U_2$ is cyclic. For $n\ge3$ we prove the following: If $G_{n-1}$ is a nilpotent group then so is $U_n$. If $G_{n-1}$ is a nilpotent-by-finite group then $U_n$ admits a faithful matrix representation. But if the variety $\mathscr C$ is different from the variety of all groups and $G_{n-1}$ is not nilpotent-by-finite then $U_n$ admits no faithful matrix representation over any field. Thus, we exhaustively classify linearity for the groups of unitriangular automorphisms of finite rank relatively free groups in proper varieties of groups, which complements the results of Olshanskii on the linearity of the full automorphism groups $\mathrm{Aut}G_n$. Moreover, we introduce the concept of length of an automorphism of an arbitrary relatively free group $G_n$ and estimate the length of the inverse automorphism in the case that it is unitriangular.