Options
Layered: Metric for machine translation evaluation
Journal
Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN
0736587X
Date Issued
2014-01-01
Author(s)
Gautam, Shubham
Bhattacharyya, Pushpak
Abstract
This paper describes the LAYERED metric which is used for the shared WMT14 metrics task. Various metrics exist for MT evaluation: BLEU (Papineni, 2002), METEOR (Alon Lavie, 2007), TER (Snover, 2006) etc., but are found inadequate in quite a few language settings like, for example, in case of free word order languages. In this paper, we propose an MT evaluation scheme that is based on the NLP layers: lexical, syntactic and semantic. We contend that higher layer metrics are after all needed. Results are presented on the corpora of ACL-WMT, 2013 and 2014. We end with a metric which is composed of weighted metrics at individual layers, which correlates very well with human judgment.