[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/xu-2020-incorporate-semantic-structures-into-machine/) [home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) / Incorporate Semantic Structures into Machine Translation Evaluation via UCCA # Incorporate Semantic Structures into Machine Translation Evaluation via UCCA **Authors:** Jin Xu, Yinuo Guo, Junfeng Hu · **Year:** 2020 · **Venue:** Proceedings of the Fifth Conference on Machine Translation (WMT 2020) **Link:** [aclanthology.org/2020.wmt-1.104](https://aclanthology.org/2020.wmt-1.104/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) [**7** citations](https://openalex.org/works?filter=cites:W3093157554) openalex · 2026-10-18[**6** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/xu-2020-incorporate-semantic-structures-into-machine/) of the 140673 papers whose bibliographies we read[Proceedings of the Fifth Conference on Machine Translation](https://aclanthology.org/volumes/2020.wmt-1/)[licence](https://creativecommons.org/licenses/by/4.0/) - [Publisher's page](https://aclanthology.org/2020.wmt-1.104/) - [PDF](https://aclanthology.org/2020.wmt-1.104.pdf) - [DOI 10.18653/v1/2020.wmt-1.104](https://doi.org/10.18653/v1/2020.wmt-1.104) - [OpenAlex](https://openalex.org/W3093157554) **Authors (3)** 1. [Jin Xu](https://openalex.org/A5101804966) [ORCID](https://orcid.org/0000-0001-9372-5681) · h 14 · 720 citations · 84 works 2. [Yinuo Guo](https://openalex.org/A5016681207) [ORCID](https://orcid.org/0009-0004-6867-6944) · h 28 · 2168 citations · 88 works 3. [Junfeng Hu](https://openalex.org/A5025524764) [ORCID](https://orcid.org/0000-0003-1409-1495) · h 12 · 606 citations · 68 works Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart. ## What it is about The paper defines semantic core words: words that appear in every good translation of one source sentence. They are found from UCCA units, mainly the main relations of scenes and their participants. The proposed measure, Semantically Weighted Sentence Similarity (SWSS), computes precision and recall over the overlap of core words after stem matching, takes their F1, and then penalises differences between the two UCCA structures and in length. Used as a wrapper around BLEU and METEOR, it improves their correlation with human judgments consistently across the tested settings. The approach borrows the copying idea from neural generation, but the measure itself is count-based. ## Ideas - I1. Core weights from agreement across authors. A concept that every Bible translation of a verse keeps (the divine name, the verb of the main event) gets a high weight in scoring; a concept only one translator keeps gets a low weight. No UCCA is needed: the agreement among KJV, WEB, ASV and JPS is the signal. - I2. Use the core weight as a priority in the lexicon work (open problem 2): a merge that breaks the agreement on a core concept counts as a larger residual than one that breaks a function word. - I3. Penalise the structural difference between two readings of one verse, in the way SWSS penalises UCCA differences, using the number of frame roles that differ in `prag`. ## For Mova The places are `tools/translate` (`meta.rs` for the lexicon, the author delta) and `prag`. The UCCA parser is not part of Mova. The cross-translation vote, which the paper uses for the core words, is the part that fits, because the Bible pairs are already in the repo's data flow. Topics: [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/) ## Cite The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2020.wmt-1.104](https://doi.org/10.18653/v1/2020.wmt-1.104) ``` @inproceedings{xu2020incorporate, title = {{Incorporate Semantic Structures into Machine Translation Evaluation via UCCA}}, author = {Xu, Jin and Guo, Yinuo and Hu, Junfeng}, booktitle = {Proceedings of the Fifth Conference on Machine Translation}, year = {2020}, doi = {10.18653/v1/2020.wmt-1.104}, url = {https://doi.org/10.18653/v1/2020.wmt-1.104} } ``` - [← previous From Shakespeare to Twitter: What are Language Styles all about?](https://md.mova.elamur.ai/free/en/docs/papers/xu-2017-shakespeare-twitter-what-are-language/) - [next → Not an Interlingua, But Close: Comparison of English AMRs to Chinese and Czech](https://md.mova.elamur.ai/free/en/docs/papers/xue-2014-not-interlingua-but-close-comparison/) - [↑ back to Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) ## References 3 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (ucca). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228–238. 0.645 0.512 0.543 0.627 0.564 0.627 0.586 0.649 0.503 0.537 0.626 0.574 0.609 0.583 [find](https://scholar.google.com/scholar?q=Omri+Abend+and+Ari+Rappoport%2E+2013%2E+Universal+conceptual+cognitive+annotation+%28ucca%29%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29) [our card](https://md.mova.elamur.ai/free/en/docs/papers/abend-2013-universal-conceptual-cognitive-annotation-ucca/) 2. Eleftherios Avramidis, Maja Popovic, David Vilar, and Aljoscha Burchardt. 2011. Evaluate with confidence estimation: Machine ranking of translation outputs using grammatical features. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 65–70. Association for Computational Linguistics. *Table 3: Results of ablation experiments. ”+UCCA” is the complete SWSS model combined with Meteor, ”-repr” means the penalties based on UCCA representation (P S , P N , P E ) are removed, ”-len” means the length penalty is removed, and ”None” contains only Meteor without SWSS.* Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 178–186. words is clearly a good and large-scale representation of copy words, according to the results. [find](https://scholar.google.com/scholar?q=Eleftherios+Avramidis%2C+Maja+Popovic%2C+David+Vilar%2C+and+Aljoscha+Burchardt%2E+2011%2E+Evaluate+with+confidence+estimation%3A+Machine+ranking+of+translation+outputs+using+grammatical+features%2E+In+Proceedings+o) [our card](https://md.mova.elamur.ai/free/en/docs/papers/banarescu-2013-abstract-meaning-representation-sembanking/) 3. Alexandra Birch, Omri Abend, Ondřej Bojar, and Barry Haddow. 2016. HUME: Human UCCA-based evaluation of machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1264–1274, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexandra+Birch%2C+Omri+Abend%2C+Ondr%CC%8Cej+Bojar%2C+and+Barry+Haddow%2E+2016%2E+HUME%3A+Human+UCCA%2Dbased+evaluation+of+machine+translation%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Lang) [our card](https://md.mova.elamur.ai/free/en/docs/papers/birch-2016-hume-human-ucca-based-evaluation/)