[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/opitz-2021-decomposable-metric-explainable-evaluation-text/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) / Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

# Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

**Authors:** [Juri Opitz](https://md.mova.elamur.ai/free/en/docs/papers/authors/juri-opitz/), [Anette Frank](https://md.mova.elamur.ai/free/en/docs/papers/authors/anette-frank/) · **Year:** 2021 · **Venue:** EACL 2021 (16th Conference of the European Chapter of ACL, Main Volume) **Link:** [aclanthology.org/2021.eacl-main.129](https://aclanthology.org/2021.eacl-main.129/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**22** citations](https://openalex.org/works?filter=cites:W3072195424) openalex · 2026-10-18[**28** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/opitz-2021-decomposable-metric-explainable-evaluation-text/) of the 140673 papers whose bibliographies we read[Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume](https://aclanthology.org/volumes/2021.eacl-main/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2021.eacl-main.129/)
- [PDF](https://aclanthology.org/2021.eacl-main.129.pdf)
- [DOI 10.18653/v1/2021.eacl-main.129](https://doi.org/10.18653/v1/2021.eacl-main.129)
- [OpenAlex](https://openalex.org/W3072195424)

**Authors (2)**

1. [Juri Opitz](https://openalex.org/A5010534476) [ORCID](https://orcid.org/0000-0001-6892-4574) · h 12 · 521 citations · 92 works
2. [Anette Frank](https://openalex.org/A5023977688) [ORCID](https://orcid.org/0000-0003-4706-9817) · h 28 · 3260 citations · 179 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The paper proposes MF-beta, a metric for AMR-to-text output with two parts combined in an F-beta style. The meaning part (M) parses the generated sentence back to an AMR with a strong parser and scores the graph overlap with the input AMR, using a graded Smatch that gives partial credit for near-matching concepts. The form part (F) scores grammaticality and fluency with a language model, and needs no reference sentence. The beta weight lets a user choose how much to weigh meaning against form. Two pilot studies follow. In the first, the disentangled scores explain system differences that BLEU hides (one system is best on meaning, another on form). In the second, the ranking is stable under two other parsers (Spearman 0.95 for the meaning score) and under dropping the gold graph, which means the metric can be used where no gold AMR exists.

## Ideas

- I1. A two-part score for translator output: meaning as the round-trip divergence in the meta-world, with per-concept grading, and form as a count-based n-gram score of the target text. Report both and state beta with each result.
- I2. Reader stability: score the same outputs with two different readers (`tools/hbo` and the MACULA-based reader) and check the rank correlation. If it is low, the meaning score is measuring the reader, not the translator.
- I3. Graded concept match: compare two meta-worlds by a score per concept, with credit for concepts that share a lexicon merge, rather than exact equality. This is the graded Smatch idea applied to the meta-world.
- I4. Gold ablation: score the output against the reading of the reference text instead of the source, to see whether the score still ranks systems. This is useful where the source reading is noisy, as for the Hebrew side.

## For Mova

The meaning score fits `tools/translate/measure.rs` and `tools/roundtrip`, and the form score fits a count-based n-gram check of the target. The graded match and the reader-stability test are the parts to adopt; the language model is not needed, since the counts are the model.

Topics: [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2021.eacl-main.129](https://doi.org/10.18653/v1/2021.eacl-main.129)

```
@inproceedings{opitz2021decomposable,
  title = {{Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR}},
  author = {Opitz, Juri and Frank, Anette},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.129},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.129}
}
```

- [← previous Multi-lingual Sentence Generation from the PIVOT interlingua](https://md.mova.elamur.ai/free/en/docs/papers/okumura-1991-multi-lingual-sentence-generation-pivot/)
- [next → Weisfeiler-Leman in the Bamboo: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity](https://md.mova.elamur.ai/free/en/docs/papers/opitz-2021-weisfeiler-leman-bamboo-novel-amr/)
- [↑ back to Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/)

## References

56 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Computer Vision – ECCV 2016, pages 382–398, Cham. Springer International Publishing. [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Basura+Fernando%2C+Mark+Johnson%2C+and+Stephen+Gould%2E+2016%2E+Spice%3A+Semantic+propositional+image+caption+evaluation%2E+In+Computer+Vision+%E2%80%93+ECCV+2016%2C+pages+382%E2%80%93398%2C+Cham%2E+Springer+Internatio)
2. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186. [find](https://scholar.google.com/scholar?q=Laura+Banarescu%2C+Claire+Bonial%2C+Shu+Cai%2C+Madalina+Georgescu%2C+Kira+Griffitt%2C+Ulf+Hermjakob%2C+Kevin+Knight%2C+Philipp+Koehn%2C+Martha+Palmer%2C+and+Nathan+Schneider%2E+2013%2E+Abstract+meaning+representation+for+s) [our card](https://md.mova.elamur.ai/free/en/docs/papers/banarescu-2013-abstract-meaning-representation-sembanking/)
3. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender+and+Alexander+Koller%2E+2020%2E+Climbing+towards+NLU%3A+On+meaning%2C+form%2C+and+understanding+in+the+age+of+data%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational)
4. Christopher Bryant and Ted Briscoe. 2018. Language model based grammatical error correction without annotated training data. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 247–253, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Christopher+Bryant+and+Ted+Briscoe%2E+2018%2E+Language+model+based+grammatical+error+correction+without+annotated+training+data%2E+In+Proceedings+of+the+Thirteenth+Workshop+on+Innovative+Use+of+NLP+for+Buil)
5. Deng Cai and Wai Lam. 2020a. AMR parsing via graph-sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290–1301, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Deng+Cai+and+Wai+Lam%2E+2020a%2E+AMR+parsing+via+graph%2Dsequence+iterative+inference%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1290%E2%80%931301%2C+Online%2E+As)
6. Deng Cai and Wai Lam. 2020b. Graph transformer for graph-to-sequence learning. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7464–7471. AAAI Press. [find](https://scholar.google.com/scholar?q=Deng+Cai+and+Wai+Lam%2E+2020b%2E+Graph+transformer+for+graph%2Dto%2Dsequence+learning%2E+In+The+Thirty%2DFourth+AAAI+Conference+on+Artificial+Intelligence%2C+AAAI+2020%2C+The+Thirty%2DSecond+Innovative+Applications+of+)
7. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shu+Cai+and+Kevin+Knight%2E+2013%2E+Smatch%3A+an+evaluation+metric+for+semantic+feature+structures%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Shor)
8. Marine Carpuat. 2013. A semantic evaluation of machine translation lexical choice. In Proceedings of the Seventh Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 1–10, Atlanta, Georgia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marine+Carpuat%2E+2013%2E+A+semantic+evaluation+of+machine+translation+lexical+choice%2E+In+Proceedings+of+the+Seventh+Workshop+on+Syntax%2C+Semantics+and+Structure+in+Statistical+Translation%2C+pages+1%E2%80%9310%2C+Atl)
9. Marco Damonte, Shay B. Cohen, and Giorgio Satta. 2017. An incremental parser for abstract meaning representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 536–546, Valencia, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marco+Damonte%2C+Shay+B%2E+Cohen%2C+and+Giorgio+Satta%2E+2017%2E+An+incremental+parser+for+abstract+meaning+representation%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Association+for+C)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
11. Lucia Donatelli, Michael Regan, William Croft, and Nathan Schneider. 2018. Annotation of tense and aspect semantics for sentential AMR. In Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018), pages 96–108, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lucia+Donatelli%2C+Michael+Regan%2C+William+Croft%2C+and+Nathan+Schneider%2E+2018%2E+Annotation+of+tense+and+aspect+semantics+for+sentential+AMR%2E+In+Proceedings+of+the+Joint+Workshop+on+Linguistic+Annotation%2C+M) [our card](https://md.mova.elamur.ai/free/en/docs/papers/donatelli-2018-annotation-tense-aspect-semantics-sentential/)
12. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The WebNLG challenge: Generating text from RDF data. In Proceedings of the 10th International Conference on Natural Language Generation, pages 124–133, Santiago de Compostela, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Claire+Gardent%2C+Anastasia+Shimorina%2C+Shashi+Narayan%2C+and+Laura+Perez%2DBeltrachini%2E+2017%2E+The+WebNLG+challenge%3A+Generating+text+from+RDF+data%2E+In+Proceedings+of+the+10th+International+Conference+on+Natu) [our card](https://md.mova.elamur.ai/free/en/docs/papers/gardent-2017-webnlg-challenge-generating-text-rdf/)
13. Jonas Groschwitz, Matthias Lindemann, Meaghan Fowlie, Mark Johnson, and Alexander Koller. 2018. AMR dependency parsing with a typed semantic algebra. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1831–1841, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jonas+Groschwitz%2C+Matthias+Lindemann%2C+Meaghan+Fowlie%2C+Mark+Johnson%2C+and+Alexander+Koller%2E+2018%2E+AMR+dependency+parsing+with+a+typed+semantic+algebra%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+A)
14. Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019. Densely connected graph convolutional networks for graph-to-sequence learning. Transactions of the Association for Computational Linguistics, 7:297–312. [find](https://scholar.google.com/scholar?q=Zhijiang+Guo%2C+Yan+Zhang%2C+Zhiyang+Teng%2C+and+Wei+Lu%2E+2019%2E+Densely+connected+graph+convolutional+networks+for+graph%2Dto%2Dsequence+learning%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7)
15. Michael Heilman, Aoife Cahill, Nitin Madnani, Melissa Lopez, Matthew Mulholland, and Joel Tetreault. 2014. Predicting grammaticality on an ordinal scale. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 174–180, Baltimore, Maryland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Michael+Heilman%2C+Aoife+Cahill%2C+Nitin+Madnani%2C+Melissa+Lopez%2C+Matthew+Mulholland%2C+and+Joel+Tetreault%2E+2014%2E+Predicting+grammaticality+on+an+ordinal+scale%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+t)
16. Hans Kamp. 1981. A theory of truth and semantic representation. In J. Groenendijk, Th. Janssen, and M. Stokhof, editors, Formal Methods in the Study of Language, pages 277–322. Mathematisch Centrum. Reprinted in ”Meaning and the Dynamics of Interpretation - Selected Papers of Hans Kamp” (Kamp (2013)) p. 329 -369. [find](https://scholar.google.com/scholar?q=Hans+Kamp%2E+1981%2E+A+theory+of+truth+and+semantic+representation%2E+In+J%2E+Groenendijk%2C+Th%2E+Janssen%2C+and+M%2E+Stokhof%2C+editors%2C+Formal+Methods+in+the+Study+of+Language%2C+pages+277%E2%80%93322%2E+Mathematisch+Centrum%2E+R)
17. Anisia Katinskaia and Sardana Ivanova. 2019. Multiple admissibility: Judging grammaticality using unlabeled data in language learning. In Proceedings of pages 4773–4786, Barcelona, Spain (Online). International Committee on Computational Linguistics. the 7th Workshop on Balto-Slavic Natural Language Processing, pages 12–22, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anisia+Katinskaia+and+Sardana+Ivanova%2E+2019%2E+Multiple+admissibility%3A+Judging+grammaticality+using+unlabeled+data+in+language+learning%2E+In+Proceedings+of+pages+4773%E2%80%934786%2C+Barcelona%2C+Spain+%28Online%29%2E+Int)
18. Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-sequence models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 146–157, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ioannis+Konstas%2C+Srinivasan+Iyer%2C+Mark+Yatskar%2C+Yejin+Choi%2C+and+Luke+Zettlemoyer%2E+2017%2E+Neural+AMR%3A+Sequence%2Dto%2Dsequence+models+for+parsing+and+generation%2E+In+Proceedings+of+the+55th+Annual+Meeting+of) [our card](https://md.mova.elamur.ai/free/en/docs/papers/konstas-2017-neural-amr-sequence-sequence-models/)
19. Maarit Koponen, Leena Salmi, and Markku Nikulin. 2019. A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output. Machine Translation, 33(1-2):61–90. [find](https://scholar.google.com/scholar?q=Maarit+Koponen%2C+Leena+Salmi%2C+and+Markku+Nikulin%2E+2019%2E+A+product+and+process+analysis+of+post%2Deditor+corrections+on+neural%2C+statistical+and+rule%2Dbased+machine+translation+output%2E+Machine+Translation%2C+)
20. Jey Han Lau, Carlos S Armendariz, Shalom Lappin, Matthew Purver, and Chang Shu. 2020. How furiously can colourless green ideas sleep? sentence acceptability in context. arXiv preprint arXiv:2004.00881. [↗](https://arxiv.org/abs/2004.00881)
21. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E+Association+for+Computational+Linguistics%2E)
22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692)
23. Chi-kiu Lo. 2017. MEANT 2.0: Accurate semantic MT evaluation for any output language. In Proceedings of the Second Conference on Machine Translation, pages 589–597, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chi%2Dkiu+Lo%2E+2017%2E+MEANT+2%2E0%3A+Accurate+semantic+MT+evaluation+for+any+output+language%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+pages+589%E2%80%93597%2C+Copenhagen%2C+Denmark%2E+Association+fo) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lo-2017-meant-2-0-accurate-semantic/)
24. Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack’s wife hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5962–5971, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Robert+Logan%2C+Nelson+F%2E+Liu%2C+Matthew+E%2E+Peters%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+2019%2E+Barack%E2%80%99s+wife+hillary%3A+Using+knowledge+graphs+for+fact%2Daware+language+modeling%2E+In+Proceedings+of+the+57th+Annual+)
25. Chunchuan Lyu and Ivan Titov. 2018. AMR parsing as graph prediction with latent alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 397–407, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chunchuan+Lyu+and+Ivan+Titov%2E+2018%2E+AMR+parsing+as+graph+prediction+with+latent+alignment%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Pa)
26. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nitika+Mathur%2C+Timothy+Baldwin%2C+and+Trevor+Cohn%2E+2020%2E+Tangled+up+in+BLEU%3A+Reevaluating+the+evaluation+of+automatic+machine+translation+evaluation+metrics%2E+In+Proceedings+of+the+58th+Annual+Meeting+of) [our card](https://md.mova.elamur.ai/free/en/docs/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/)
27. Jonathan May and Jay Priyadarshi. 2017. SemEval-2017 task 9: Abstract meaning representation parsing and generation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 536–545, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jonathan+May+and+Jay+Priyadarshi%2E+2017%2E+SemEval%2D2017+task+9%3A+Abstract+meaning+representation+parsing+and+generation%2E+In+Proceedings+of+the+11th+International+Workshop+on+Semantic+Evaluation+%28SemEval%2D2)
28. Preksha Nema and Mitesh M. Khapra. 2018. Towards a better metric for evaluating question generation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3950–3959, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Preksha+Nema+and+Mitesh+M%2E+Khapra%2E+2018%2E+Towards+a+better+metric+for+evaluating+question+generation+systems%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+)
29. Rik van Noord and Johan Bos. 2017. The meaning factory at SemEval-2017 task 9: Producing AMRs with neural semantic parsing. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 929–933, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rik+van+Noord+and+Johan+Bos%2E+2017%2E+The+meaning+factory+at+SemEval%2D2017+task+9%3A+Producing+AMRs+with+neural+semantic+parsing%2E+In+Proceedings+of+the+11th+International+Workshop+on+Semantic+Evaluation+%28Se)
30. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jekaterina+Novikova%2C+Ondr%CC%8Cej+Dus%CC%8Cek%2C+Amanda+Cercas+Curry%2C+and+Verena+Rieser%2E+2017%2E+Why+we+need+new+evaluation+metrics+for+NLG%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Lan)
31. Juri Opitz. 2020. AMR quality rating with a lightweight CNN. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 235–247, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Juri+Opitz%2E+2020%2E+AMR+quality+rating+with+a+lightweight+CNN%2E+In+Proceedings+of+the+1st+Conference+of+the+Asia%2DPacific+Chapter+of+the+Association+for+Computational+Linguistics+and+the+10th+Internationa)
32. Juri Opitz and Anette Frank. 2019. Automatic accuracy prediction for AMR parsing. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 212–223, Minneapolis, Minnesota. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Juri+Opitz+and+Anette+Frank%2E+2019%2E+Automatic+accuracy+prediction+for+AMR+parsing%2E+In+Proceedings+of+the+Eighth+Joint+Conference+on+Lexical+and+Computational+Semantics+%28%2ASEM+2019%29%2C+pages+212%E2%80%93223%2C+Minne)
33. Manuel Mager, Ramón Fernandez Astudillo, Tahira Naseem, Md Arafat Sultan, Young-Suk Lee, Radu Florian, and Salim Roukos. 2020. GPT-too: A language-model-first approach for AMR-to-text generation. In Proc. of ACL, pages 1846–1852, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Manuel+Mager%2C+Ramo%CC%81n+Fernandez+Astudillo%2C+Tahira+Naseem%2C+Md+Arafat+Sultan%2C+Young%2DSuk+Lee%2C+Radu+Florian%2C+and+Salim+Roukos%2E+2020%2E+GPT%2Dtoo%3A+A+language%2Dmodel%2Dfirst+approach+for+AMR%2Dto%2Dtext+generation%2E+In+)
34. Juri Opitz, Anette Frank, and Letitia Parcalabescu. 2020. Amr similarity metrics from principles. Transactions of the Association for Computational Linguistics, 8(0):522–538. [find](https://scholar.google.com/scholar?q=Juri+Opitz%2C+Anette+Frank%2C+and+Letitia+Parcalabescu%2E+2020%2E+Amr+similarity+metrics+from+principles%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%280%29%3A522%E2%80%93538%2E)
35. Emma Manning, Shira Wein, and Nathan Schneider. 2020. A human evaluation of AMR-to-English generation systems. In Proceedings of the 28th International Conference on Computational Linguistics, Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. [find](https://scholar.google.com/scholar?q=Emma+Manning%2C+Shira+Wein%2C+and+Nathan+Schneider%2E+2020%2E+A+human+evaluation+of+AMR%2Dto%2DEnglish+generation+systems%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+Martha+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/manning-2020-human-evaluation-amr-english-generation/)
36. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
37. Adam Poliak, Yonatan Belinkov, James Glass, and Benjamin Van Durme. 2018. On the evaluation of semantic phenomena in neural machine translation using natural language inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 513–523, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Yonatan+Belinkov%2C+James+Glass%2C+and+Benjamin+Van+Durme%2E+2018%2E+On+the+evaluation+of+semantic+phenomena+in+neural+machine+translation+using+natural+language+inference%2E+In+Proceedings+of+the+)
38. Alexander Popov. 2017. Word sense disambiguation with recurrent neural networks. In Proceedings of the Student Research Workshop Associated with RANLP 2017, pages 25–34, Varna. INCOMA Ltd. [find](https://scholar.google.com/scholar?q=Alexander+Popov%2E+2017%2E+Word+sense+disambiguation+with+recurrent+neural+networks%2E+In+Proceedings+of+the+Student+Research+Workshop+Associated+with+RANLP+2017%2C+pages+25%E2%80%9334%2C+Varna%2E+INCOMA+Ltd%2E)
39. Maja Popović. 2015. chrF: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2015%2E+chrF%3A+character+n%2Dgram+f%2Dscore+for+automatic+MT+evaluation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+392%E2%80%93395%2C+Lisbon%2C+Portugal%2E+Association+f) [our card](https://md.mova.elamur.ai/free/en/docs/papers/popovic-2015-chrf-character-n-gram-f/)
40. Maja Popović. 2016. chrF deconstructed: beta parameters and n-gram weights. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 499–504, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2016%2E+chrF+deconstructed%3A+beta+parameters+and+n%2Dgram+weights%2E+In+Proceedings+of+the+First+Conference+on+Machine+Translation%3A+Volume+2%2C+Shared+Task+Papers%2C+pages+499%E2%80%93504%2C+Berlin%2C+Germany) [our card](https://md.mova.elamur.ai/free/en/docs/papers/popovic-2016-chrf-deconstructed-beta-parameters-n/)
41. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
42. Leonardo F. R. Ribeiro, Claire Gardent, and Iryna Gurevych. 2019. Enhancing AMR-to-text generation with dual graph representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3183–3194, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Leonardo+F%2E+R%2E+Ribeiro%2C+Claire+Gardent%2C+and+Iryna+Gurevych%2E+2019%2E+Enhancing+AMR%2Dto%2Dtext+generation+with+dual+graph+representations%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natura)
43. C. J. van Rijsbergen. 1979. Information Retrieval. Butterworth. [find](https://scholar.google.com/scholar?q=C%2E+J%2E+van+Rijsbergen%2E+1979%2E+Information+Retrieval%2E+Butterworth%2E)
44. Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2020. A survey of evaluation metrics used for nlg systems. arXiv preprint arXiv:2008.12009. [↗](https://arxiv.org/abs/2008.12009)
45. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. [↗](https://arxiv.org/abs/1910.01108)
46. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696. [↗](https://arxiv.org/abs/2004.04696)
47. Anastaisa Shimorina, Claire Gardent, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: report on human evaluation. Technical report, Technical report, Université de Lorraine, Nancy, France. [find](https://scholar.google.com/scholar?q=Anastaisa+Shimorina%2C+Claire+Gardent%2C+Shashi+Narayan%2C+and+Laura+Perez%2DBeltrachini%2E+2017%2E+The+webnlg+challenge%3A+report+on+human+evaluation%2E+Technical+report%2C+Technical+report%2C+Universite%CC%81+de+Lorraine%2C+N)
48. Linfeng Song, Xiaochang Peng, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2017. AMR-to-text generation with synchronous node replacement grammar. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 7–13, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Linfeng+Song%2C+Xiaochang+Peng%2C+Yue+Zhang%2C+Zhiguo+Wang%2C+and+Daniel+Gildea%2E+2017%2E+AMR%2Dto%2Dtext+generation+with+synchronous+node+replacement+grammar%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Associ) [our card](https://md.mova.elamur.ai/free/en/docs/papers/song-2017-amr-text-generation-synchronous-node/)
49. Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for AMR-to-text generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1616– 1626, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Linfeng+Song%2C+Yue+Zhang%2C+Zhiguo+Wang%2C+and+Daniel+Gildea%2E+2018%2E+A+graph%2Dto%2Dsequence+model+for+AMR%2Dto%2Dtext+generation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Ling)
50. Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015. Results of the WMT15 metrics shared task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256– 273, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Milos%CC%8C+Stanojevic%CC%81%2C+Amir+Kamran%2C+Philipp+Koehn%2C+and+Ondr%CC%8Cej+Bojar%2E+2015%2E+Results+of+the+WMT15+metrics+shared+task%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+256%E2%80%93+2)
51. Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. arXiv preprint arXiv:2005.03642. [↗](https://arxiv.org/abs/2005.03642)
52. Tianming Wang, Xiaojun Wan, and Hanqi Jin. 2020a. Amr-to-text generation with graph transformer. Transactions of the Association for Computational Linguistics, 8(0):19–33. [find](https://scholar.google.com/scholar?q=Tianming+Wang%2C+Xiaojun+Wan%2C+and+Hanqi+Jin%2E+2020a%2E+Amr%2Dto%2Dtext+generation+with+graph+transformer%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%280%29%3A19%E2%80%9333%2E)
53. Tianming Wang, Xiaojun Wan, and Shaowei Yao. 2020b. Better amr-to-text generation with graph structure reconstruction. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3919–3925. International Joint Conferences on Artificial Intelligence Organization. Main track. [find](https://scholar.google.com/scholar?q=Tianming+Wang%2C+Xiaojun+Wan%2C+and+Shaowei+Yao%2E+2020b%2E+Better+amr%2Dto%2Dtext+generation+with+graph+structure+reconstruction%2E+In+Proceedings+of+the+Twenty%2DNinth+International+Joint+Conference+on+Artificial+I)
54. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
55. Wangchunshu Zhou and Ke Xu. 2020. Learning to compare for better training and evaluation of open domain natural language generation models. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 9717– 9724. AAAI Press. [find](https://scholar.google.com/scholar?q=Wangchunshu+Zhou+and+Ke+Xu%2E+2020%2E+Learning+to+compare+for+better+training+and+evaluation+of+open+domain+natural+language+generation+models%2E+In+The+Thirty%2DFourth+AAAI+Conference+on+Artificial+Intellige)
56. Wanzheng Zhu and Suma Bhat. 2020. Gruen for evaluating linguistic quality of generated text. arXiv preprint arXiv:2010.02498. F1 score grammaticality LM all poor/perfect all 0.80 0.79 0.80 0.66 0.74 0.73 0.72 0.72 0.80 0.76 0.80 0.69 0.71 0.70 0.72 0.72 GPT2 GPT2-distill BERT RoBERTa *Table 4: Results for assessing the F orm score prediction (corpus-level) of different LMs for NLG-generated sentences against humans judgements (separated by grammaticality and fluency); all: all 12k generated sentences vs. ’poor/perfect’: the 5k instances of best/worst generations in both grammaticality and fluency.* *Figure 4: A critical issue and its alleviation.* [↗](https://arxiv.org/abs/2010.02498)