Architecture of a distributed computing system with tasks containerization and prioritization

This article describes an architecture of a distributed system, which can speed up the process of obtaining results for such tasks. The system comprises a backend server, control service (supervisor), a set of worker nodes and a database. To abstract from particular languages and tools required for computational algorithms, these algorithms are executed in Docker containers. The system supports several strategies for tasks prioritization to operate efficiently under heavy load introduced by multiple users. To make use of the system, the user only needs to build a Docker image with an encapsulated algorithm, describe the input dataset in a JSON file and upload them via web interface. The system can be deployed in any public cloud. In this article, we provide a detailed description of the system architecture and numerical results obtained from computations on various clouds and local platforms. We show the influence of different prioritization strategies on the duration of computations under a moderate workload.