The technique allowing for temporal estimation of machine translation instability

The paper presents a technique allowing for temporal estimation of instability of neural machine translation (NMT). This technique gives an opportunity to see how NMT of a given text fragment changes with time. The experiment described in the paper involves 250 Russian text fragments. During a year, each text fragment was repeatedly translated into French. The time step was one month. To produce translations, the Google NMT system was used. All the translations were annotated in a supracorpora database to register the output errors (if there were any). Eventually, for each of 250 text fragments, there was a series of 12 annotated translations. The annotation containing the 12th translation had a heading denoting the degree of NMT instability in relation to the entire series of translations. This heading characterized changes in translation quality or indicated their absence. The paper is aimed to describe both the technique allowing for temporal estimation of NMT instability and results of its application.