Instability of neural machine translation

The paper describes an experiment focused on studying the instability of neural machine translation (NMT). In the course of a year, an array of text fragments in Russian was repeatedly translated into French. The time step was one month. To produce translations, the Google's NMT system was used. The experiment helps reveal the instability of NMT, i. e., it shows that translations of a given text fragment tend to change with time but not always improving the quality. The generated translations were linguistically annotated, which led to uncovering several different types of the NMT instability. While annotating, a previously designed classification of machine translation errors was employed. It was altered to meet the objectives of the experiment, the ultimate goal of which was to obtain a frequency distribution of different types of the NMT instability. Yet, the first step of the experiment limited itself to only categorizing the NMT instability, and it is this very step that the paper describes. As the empirical data, the experiment uses Russian–French annotations generated in a supracorpora database. Each annotation contains a fragment of the source Russian text, its translation into French, and the description of translation errors occurring there.