Loading...

Automatic Evaluation of Machine Translation Using Abstract Meaning Representation

Sadeghieh, Hamid | 2021

859 Viewed
  1. Type of Document: M.Sc. Thesis
  2. Language: Farsi
  3. Document No: 53944 (31)
  4. University: Sharif University of Technology
  5. Department: Languages and Linguistics Center
  6. Advisor(s): Rezae, Saeed; Bahrani, Mohammad
  7. Abstract:
  8. Machine Translation Quality Evaluation, compared to the other issues dealt with in the field of Natural Language Processing, is faced with the challenge that the repetition of the translation process from the same linguistic form in the source language will not necessarily lead to a unique linguistic form in the target language. Therefore, considering the fact that the Abstract Meaning Representation (AMR) graph is the same for all the sentences of similar meaning, this thesis has been an attempt to extend the efficiency of AMR graphs to the area of Machine Translation Quality Evaluation. The main research question dealt with in the present thesis was whether the similarity of the AMR graphs of the candidate and reference translations, regardless of their lexical-syntactic structure, can provide an acceptable estimation of the quality of candidate translations. Thus, the maximum value of F score, calculated from the matching areas of candidate and reference translations’ AMR graphs (referred to as smatch score), has been considered as the semantic similarity index of the two sentences under study, and as a result, a criterion for the quality of machine translation. In order the examine the efficiency of the proposed methods, an experiment was designed and the smatch:std, smatch:-lablel, smatch:-vars, and smatch:-wsd scores were also computed for the sample data used in the Workshop of Machine Translation held in 2018 (WMT18) for seven different languages into English. The results revealed that one can consider smatch scores of AMR graphs of the candidate and reference translations as an estimation of the quality of translation, particularly if it is intended to evaluate the system-level quality of candidate translations. Moreover, it should be added that the proposed methods have revealed an acceptable performance in comparison to the baseline and even the state-of-the-art metrics in most of the language pairs under investigation, particularly in Czech – English language pair. Furthermore, smatch:-label proposed method has been the most successful method in the automated system-level evaluation of translation quality. Ultimately, it can be concluded that although the performance of the proposed methods is affected by the quality level of candidate translations, just like the baseline metrics, the consistency of the correlation of all the proposed methods, except smatch:-vars, has been greater than that of the baseline and state-of-the art metrics under investigation with the human evaluation method
  9. Keywords:
  10. Abstract Meaning Representation ; Semantic Graph Similarity ; Automated Machine Translation Quality Evaluation ; Machine Translation

 Digital Object List

 Bookmark

...see more