Indicator-based evaluation of machine translation instability

The paper presents data collected by tracking performance of a neural machine translation (NMT) engine and results of translation errors analysis. Indicator-based evaluation of NMT instability was carried out as a part of an experiment that involved 250 Russian text fragments. Each month for the duration of one year, these fragments were translated into French using the Google Translate NMT engine. The translations were recorded and annotated in a supracorpora database; the annotations include types of translation errors found in the translations by language experts. This procedure resulted in a series of 12 annotated translations for each of the 250 Russian fragments. The annotations include not only the types of errors found in the translations but also the types of NMT instability which indicate dynamics of translation quality or lack thereof. The paper aims to provide comparative analysis of Google Translate performance that takes into account the temporal variation aspect.