Expert evaluation of machine translation: Error classification

The paper considers the error classification applied in the expert evaluation of the machine translation quality. The classification includes common error headings (for errors at the level of grammar, vocabulary, punctuation, etc.) as well as headings that are associated with a specific type of linguistic unit selected for evaluating the machine translation quality. The quality evaluation is performed by experts as they linguistically annotate machine translation outcomes. If, while annotating, an expert finds errors, then headings, necessary to characterize these errors, are included in the annotation. The headings allow one to calculate the relative frequency of machine translation errors for the array of test sentences selected for translation and quality evaluation. The main goal of the paper is to describe the proposed classification of common and *specific errors. The principal difference of the classification from the existing error classifications is that it is aimed at backing interval evaluation for machine translation systems, whose quality of work may vary over time. The headings of the proposed classification allow experts to record both improvements and decreases in machine translation quality at a given time interval.