[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/liu-2005-syntactic-features-evaluation-machine-translation/) [home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) / Syntactic Features for Evaluation of Machine Translation # Syntactic Features for Evaluation of Machine Translation **Authors:** Ding Liu, [Daniel Gildea](https://md.mova.elamur.ai/free/en/docs/papers/authors/daniel-gildea/) · **Year:** 2005 · **Venue:** Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization **Link:** [aclanthology.org/W05-0904](https://aclanthology.org/W05-0904/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only [**90** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/liu-2005-syntactic-features-evaluation-machine-translation/) of the 140673 papers whose bibliographies we read[Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization](https://aclanthology.org/volumes/W05-09/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/W05-0904/) - [PDF](https://aclanthology.org/W05-0904.pdf) **Authors (2)** 1. Ding Liu 2. Daniel Gildea Citations and author metrics from OpenAlex (); authors with the same name are not always told apart. ## What it is about The paper adds syntax to n-gram evaluation. The headword-chain metric (HWCM) matches the head-modifier paths of the reference against the hypothesis. The subtree metric (STM) matches constituent subtrees, and a dependency subtree variant (DSTM) matches head subtrees. All of them need a parser on the hypothesis. At sentence level, most syntactic metrics predict fluency better than BLEU, and HWCM predicts overall sentence judgments better too. At corpus level the kernel-based subtree metric reaches a correlation of about 0.88 with human overall judgments on the 2005 test systems, against about 0.78 for BLEU at its best n-gram length. The authors note that the kernel metric is weak at sentence level, so the two levels can disagree. ## Ideas - I1. Headword-chain matching on meta trees: take the head-modifier chains of the reference's trees, read by the same reader, and check that each chain appears in the output's trees in the same order. This measures open problem 3, whether the encoder writes trees. - I2. Report sentence-level and verse-level-aggregated scores separately, as the paper shows they can disagree. A delta that raises the corpus score but lowers the per-verse agreement is flagged. - I3. A syntactic fluency check on the encoder output, computed as chain matches, to catch order errors that bag-of-concept scores miss. ## For Mova The places are `tools/roundtrip` (UD trees) and `en/src/lin.rs` (order). Mova does not need a parser on the output: the paper's cost is the parsing of noisy MT, and Mova can compare meta trees built from the references instead. Topics: [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/) ## Cite The paper itself, not this card. BibTeX from the library's record of the work. ``` @inproceedings{liu2005syntactic, title = {{Syntactic Features for Evaluation of Machine Translation}}, author = {Liu, Ding and Gildea, Daniel}, booktitle = {Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization}, year = {2005}, url = {https://aclanthology.org/W05-0904/} } ``` - [← previous Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://md.mova.elamur.ai/free/en/docs/papers/li-2018-delete-retrieve-generate-simple-approach/) - [next → Tree-to-String Alignment Template for Statistical Machine Translation](https://md.mova.elamur.ai/free/en/docs/papers/liu-2006-tree-string-alignment-template-statistical/) - [↑ back to Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) ## References 12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Hiyan Alshawi, Srinivas Bangalore, and Shona Douglas. 2000. Learning dependency translation models as collections of finite state head transducers. Computational Linguistics, 26(1):45–60. [find](https://scholar.google.com/scholar?q=Hiyan+Alshawi%2C+Srinivas+Bangalore%2C+and+Shona+Douglas%2E+2000%2E+Learning+dependency+translation+models+as+collections+of+finite+state+head+transducers%2E+Computational+Linguistics%2C+26%281%29%3A45%E2%80%9360%2E) 2. John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2003. Confidence estimation for machine translation. Technical report, Center for Language and Speech Processing, Johns Hopkins University, Baltimore. Summer Workshop Final Report. [find](https://scholar.google.com/scholar?q=John+Blatz%2C+Erin+Fitzgerald%2C+George+Foster%2C+Simona+Gandrabur%2C+Cyril+Goutte%2C+Alex+Kulesza%2C+Alberto+Sanchis%2C+and+Nicola+Ueffing%2E+2003%2E+Confidence+estimation+for+machine+translation%2E+Technical+report%2C+Ce) 3. Eugene Charniak, Kevin Knight, and Kenji Yamada. 2003. Syntax-based language models for machine translation. In Proc. MT Summit IX. [find](https://scholar.google.com/scholar?q=Eugene+Charniak%2C+Kevin+Knight%2C+and+Kenji+Yamada%2E+2003%2E+Syntax%2Dbased+language+models+for+machine+translation%2E+In+Proc%2E+MT+Summit+IX%2E) 4. Michael Collins and Nigel Duffy. 2001. Convolution kernels for natural language. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Michael+Collins+and+Nigel+Duffy%2E+2001%2E+Convolution+kernels+for+natural+language%2E+In+Advances+in+Neural+Information+Processing+Systems%2E) 5. Michael John Collins. 1999. Head-driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania, Philadelphia. [find](https://scholar.google.com/scholar?q=Michael+John+Collins%2E+1999%2E+Head%2Ddriven+Statistical+Models+for+Natural+Language+Parsing%2E+Ph%2ED%2E+thesis%2C+University+of+Pennsylvania%2C+Philadelphia%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/cl-2003-collins-head-driven-statistical-models-natural/) 6. G. Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In In HLT 2002, Human Language Technology Conference, San Diego, CA. [find](https://scholar.google.com/scholar?q=G%2E+Doddington%2E+2002%2E+Automatic+evaluation+of+machine+translation+quality+using+n%2Dgram+co%2Doccurrence+statistics%2E+In+In+HLT+2002%2C+Human+Language+Technology+Conference%2C+San+Diego%2C+CA%2E) 7. Heidi J. Fox. 2002. Phrasal cohesion and statistical machine translation. In In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 304–311. [find](https://scholar.google.com/scholar?q=Heidi+J%2E+Fox%2E+2002%2E+Phrasal+cohesion+and+statistical+machine+translation%2E+In+In+Proceedings+of+the+2002+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP+2002%29%2C+pages+304%E2%80%93311%2E) 8. Alex Kulesza and Stuart M. Shieber. 2004. A learning approach to improving sentence-level MT evaluation. In Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation (TMI), Baltimore, MD, October. [find](https://scholar.google.com/scholar?q=Alex+Kulesza+and+Stuart+M%2E+Shieber%2E+2004%2E+A+learning+approach+to+improving+sentence%2Dlevel+MT+evaluation%2E+In+Proceedings+of+the+10th+International+Conference+on+Theoretical+and+Methodological+Issues+in) 9. Dekang Lin. 2004. A path-based transfer model for machine translation. In Proceedings of the 20th International Conference on Computational Linguistics (COLING-04), pages 625–630, Geneva, Switzerland. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+2004%2E+A+path%2Dbased+transfer+model+for+machine+translation%2E+In+Proceedings+of+the+20th+International+Conference+on+Computational+Linguistics+%28COLING%2D04%29%2C+pages+625%E2%80%93630%2C+Geneva%2C+Switzerland%2E) 10. Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, Anoop Sarkar, Kenji Yamada, Alex Fraser, Shankar Kumar, Libin Shen, David Smith, Katherine Eng, Viren Jain, Zhen Jin, and Dragomir Radev. 2004. A smorgasbord of features for statistical machine translation. In Proceedings of the 2004 Meeting of the North American chapter of the Association for Computational Linguistics (NAACL-04), Boston. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och%2C+Daniel+Gildea%2C+Sanjeev+Khudanpur%2C+Anoop+Sarkar%2C+Kenji+Yamada%2C+Alex+Fraser%2C+Shankar+Kumar%2C+Libin+Shen%2C+David+Smith%2C+Katherine+Eng%2C+Viren+Jain%2C+Zhen+Jin%2C+and+Dragomir+Radev%2E+2004%2E+A+smo) 11. Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In Proceedings of the 41th Annual Conference of the Association for Computational Linguistics (ACL-03). [find](https://scholar.google.com/scholar?q=Franz+Josef+Och%2E+2003%2E+Minimum+error+rate+training+for+statistical+machine+translation%2E+In+Proceedings+of+the+41th+Annual+Conference+of+the+Association+for+Computational+Linguistics+%28ACL%2D03%29%2E) 12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Conference of the Association for Computational Linguistics (ACL-02), Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Conference+of+the+Association+for+C) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)