Thoughts on the development of SKIF-Aurora supercomputer interconnect

This article presents speculations on the experience authors have got while developing the 3D-torus interconnect of SKIF-Aurora supercomputer. Authors have implemented all the levels of network infrastructure from the schematics of network adapters up to the support of communication libraries. Some advantages of self-made networks over the commercially available off-the-shelf solutions are discussed. We show that the last ones can be significantly surpassed in some aspects by careful engineering and implementation. As an example we compare message rate characteristics in our network and in InfiniBand.