Machine translation: Indicator-based evaluation of training progress in neural processing

The paper presents data collected while observing training progress of a neural machine translation (NMT) engine. The observed training progress received qualitative evaluation based on a set of indicators. Two hundred and fifty text fragments in Russian were used as experimental material for the study. For the duration of one year, every month these fragments were translated into French using the publicly available Google's NMT engine. The produced translations were recorded and annotated by language experts in a supracorpora database which resulted in a series of 12 annotated translations for each of the 250 Russian fragments. The annotations include labels of translation errors which enables researchers to determine the NMT instability types according to the changes of translation quality or lack thereof. The goal of this paper is to describe the newly developed indicator-based approach and to provide an example of its application to evaluation of a neural network training progress.