[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/lo-2017-meant-2-0-accurate-semantic/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) / MEANT 2.0: Accurate semantic MT evaluation for any output language

# MEANT 2.0: Accurate semantic MT evaluation for any output language

**Authors:** [Chi-kiu Lo](https://md.mova.elamur.ai/free/en/docs/papers/authors/chi-kiu-lo/) · **Year:** 2017 · **Venue:** Proceedings of the Second Conference on Machine Translation (WMT 2017) **Link:** [aclanthology.org/W17-4767](https://aclanthology.org/W17-4767/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**51** citations](https://openalex.org/works?filter=cites:W2760738985) openalex · 2026-10-18[**30** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/lo-2017-meant-2-0-accurate-semantic/) of the 140673 papers whose bibliographies we read[Proceedings of the Second Conference on Machine Translation](https://aclanthology.org/volumes/W17-47/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/W17-4767/)
- [PDF](https://aclanthology.org/W17-4767.pdf)
- [DOI 10.18653/v1/w17-4767](https://doi.org/10.18653/v1/w17-4767)
- [OpenAlex](https://openalex.org/W2760738985)

**Authors (1)**

1. [Lo, Chi-kiu](https://openalex.org/A5049492975) [ORCID](https://orcid.org/0000-0001-8714-7846) · h 16 · 933 citations · 80 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

MEANT 2.0 is the version of the frame-matching metric submitted to the WMT 2017 metrics task. It keeps the matching of predicates and role fillers, but changes how two fillers are compared: an idf-weighted distributional n-gram accuracy, computed from word embeddings trained on a monolingual corpus of the output language. The idf weight lowers the importance of common words; the n-gram accuracy handles reordering. A degraded variant (nosrl) drops the semantic parser and so works for any output language, at lower accuracy. The paper reports that the untrained, non-ensemble metric performs as well as the best earlier WMT participants across output languages, including ensemble and trained ones, and gives timing figures showing a low cost. It is open source.

## Ideas

- I1. idf weights over the verse corpus: a match is weighted by how rare its concept is across all the Bible pairs, so common function concepts weigh less than names when two renderings are compared.
- I2. The parser-free variant as a fallback for Hebrew: match the reader's concept and tag sequences without frames, so the score works before the MACULA-based frames are used.
- I3. An order-sensitive n-gram accuracy over the encoder's chunk sequence. It tests open problem 3 (the encoder should write trees, not strings) by measuring whether chunks keep their neighbours.

## For Mova

The places are `tools/translate` and `profile.rs` (the standard encoder). An idf table is a plain count table and is cheap. The paper's embeddings are a neural component; Mova's version would be count-based, for example co-occurrence counts over chunks in ordered maps.

Topics: [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/w17-4767](https://doi.org/10.18653/v1/w17-4767)

```
@inproceedings{lo2017meant,
  title = {{MEANT 2.0: Accurate semantic MT evaluation for any output language}},
  author = {Lo, Chi-kiu},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4767},
  url = {https://doi.org/10.18653/v1/w17-4767}
}
```

- [← previous MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles](https://md.mova.elamur.ai/free/en/docs/papers/lo-2011-meant-inexpensive-high-accuracy-semi/)
- [next → Multidimensional quality metrics: a flexible system for assessing translation quality](https://md.mova.elamur.ai/free/en/docs/papers/lommel-2013-multidimensional-quality-metrics-flexible-system/)
- [↑ back to Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/)

## References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omri Abend and Ari Rappoport. 2013. UCCA: A Semantics-based Grammatical Annotation Scheme. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers. Association for Computational Linguistics, Potsdam, Germany, pages 1–12. http://www.aclweb.org/anthology/W13-0101. [↗](https://aclanthology.org/W13-0101/)
2. Alexandra Birch, Omri Abend, Ondřej Bojar, and Barry Haddow. 2016. HUME: Human UCCA-Based Evaluation of Machine Translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1264–1274. https://aclweb.org/anthology/D16-1134. Finally, the time used in computing the MEANT score is proportional to the size of the evaluation set and the word embedding model. The scoring step processes around 50 to 100 sentences each second. movements. In 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL-2006). [↗](https://aclanthology.org/D16-1134/) [our card](https://md.mova.elamur.ai/free/en/docs/papers/birch-2016-hume-human-ucca-based-evaluation/)
3. Anders Björkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual semantic role labeling. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL 2009): Shared Task. Association for Computational Linguistics, Boulder, Colorado, pages 43–48. http://www.aclweb.org/anthology/W09-1206. [↗](https://aclanthology.org/W09-1206/)
4. Chi-kiu Lo, Meriem Beloucif, Markus Saers, and Dekai Wu. 2014. XMEANT: Better semantic MT evaluation without reference translations. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, pages 765–771. https://doi.org/10.3115/v1/P14-2124. [↗](https://doi.org/10.3115/v1/p14-2124)
5. Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 Workshop on Statistical Machine Translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Baltimore, Maryland, USA, pages 12– 58. http://www.aclweb.org/anthology/W14-3302. [↗](https://aclanthology.org/W14-3302/)
6. Chi-kiu Lo, Philipp Dowling, and Dekai Wu. 2015. Improving evaluation and optimization of MT systems against MEANT. In Proceedings of the Tenth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Lisbon, Portugal, pages 434–441. http://aclweb.org/anthology/W15-3056. [↗](https://aclanthology.org/W15-3056/)
7. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016a. Findings of the 2016 Conference on Machine Translation. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 131–198. http://www.aclweb.org/anthology/W16-2301. [↗](https://aclanthology.org/W16-2301/)
8. Chi-kiu Lo and Dekai Wu. 2011a. MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pages 220–229. http://aclweb.org/anthology/P11-1023. [↗](https://aclanthology.org/P11-1023/) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lo-2011-meant-inexpensive-high-accuracy-semi/)
9. Chi-Kiu Lo and Dekai Wu. 2011b. SMT Versus AI Redux: How Semantic Fames Evaluate MT More Accurately. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence - Volume Volume Three. AAAI Press, IJCAI’11, pages 1838– 1845. https://doi.org/10.5591/978-1-57735-516-8/IJCAI11-308. [↗](https://doi.org/10.5591/978-1-57735-516-8/ijcai11-308)
10. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 Workshop on Statistical Machine Translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Lisbon, Portugal, pages 1–46. http://aclweb.org/anthology/W15-3001. [↗](https://aclanthology.org/W15-3001/)
11. Chi-kiu Lo and Dekai Wu. 2012. Unsupervised vs. supervised weight estimation for semantic MT evaluation metrics. In Proceedings of the Sixth Workshop on Syntax, Semantics and Structure in Statistical Translation. Association for Computational Linguistics, Jeju, Republic of Korea, pages 49–56. http://www.aclweb.org/anthology/W12-4206. [↗](https://aclanthology.org/W12-4206/)
12. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016b. Results of the WMT16 Metrics Shared Task. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 199–231. http://www.aclweb.org/anthology/W16-2302. [↗](https://aclanthology.org/W16-2302/)
13. Matous Machacek and Ondrej Bojar. 2014. Results of the WMT14 Metrics Shared Task. In Proceedings of the Ninth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Baltimore, Maryland, USA, pages 293– 301. http://www.aclweb.org/anthology/W14-3336. [↗](https://aclanthology.org/W14-3336/)
14. Michael Denkowski and Alon Lavie. 2014. METEOR universal: Language specific translation evaluation for any target language. In 9th Workshop on Statistical Machine Translation (WMT 2014). [find](https://scholar.google.com/scholar?q=Michael+Denkowski+and+Alon+Lavie%2E+2014%2E+METEOR+universal%3A+Language+specific+translation+evaluation+for+any+target+language%2E+In+9th+Workshop+on+Statistical+Machine+Translation+%28WMT+2014%29%2E)
15. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In The second international conference on Human Language Technology Research (HLT ’02). San Diego, California. [find](https://scholar.google.com/scholar?q=George+Doddington%2E+2002%2E+Automatic+evaluation+of+machine+translation+quality+using+n%2Dgram+co%2Doccurrence+statistics%2E+In+The+second+international+conference+on+Human+Language+Technology+Research+%28HLT+%E2%80%990)
16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems. Curran Associates Inc., USA, NIPS’13, pages 3111–3119. [http://dl.acm.org/citation.cfm?id=2999792.2999959](http://dl.acm.org/citation.cfm?id=2999792.2999959). [↗](http://dl.acm.org/citation.cfm?id=2999792.2999959)
17. Gregor Leusch, Nicola Ueffing, and Hermann Ney. 2006. CDer: Efficient MT evaluation using block Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In 40th Annual Meeting of the Association for Computational Linguistics (ACL-02). Philadelphia, Pennsylvania, pages 311–318. https://doi.org/10.3115/1073083.1073135. [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
18. Michael Roth and Kristian Woodsend. 2014. Composition of word representations improves semantic role labelling. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 407–413. http://www.aclweb.org/anthology/D14-1045. [↗](https://aclanthology.org/D14-1045/)
19. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In 7th Biennial Conference Association for Machine Translation in the Americas (AMTA 2006). Cambridge, Massachusetts, pages 223–231. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+7th+Biennial+Conference+Association+for+Mac) [our card](https://md.mova.elamur.ai/free/en/docs/papers/snover-2006-study-translation-edit-rate-targeted/)
20. Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015. Results of the WMT15 Metrics Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation. Association for Computational Linguistics, Lisbon, Portugal, pages 256–273. http://aclweb.org/anthology/W15-3031. [↗](https://aclanthology.org/W15-3031/)
21. Hua-Ping Zhang, Hong-Kui Yu, De-Yi Xiong, and Qun Liu. 2003. HHMM-based Chinese Lexical Analyzer ICTCLAS. In Proceedings of the Second SIGHAN Workshop on Chinese Language Processing. Association for Computational Linguistics, Sapporo, Japan, pages 184–187. [find](https://scholar.google.com/scholar?q=Hua%2DPing+Zhang%2C+Hong%2DKui+Yu%2C+De%2DYi+Xiong%2C+and+Qun+Liu%2E+2003%2E+HHMM%2Dbased+Chinese+Lexical+Analyzer+ICTCLAS%2E+In+Proceedings+of+the+Second+SIGHAN+Workshop+on+Chinese+Language+Processing%2E+Association+for+C)