[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/manning-2021-referenceless-parsing-based-evaluation-amr/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Referenceless Parsing-Based Evaluation of AMR-to-English Generation

# Referenceless Parsing-Based Evaluation of AMR-to-English Generation

**Authors:** [Emma Manning](https://md.mova.elamur.ai/free/en/docs/papers/authors/emma-manning/) and [Nathan Schneider](https://md.mova.elamur.ai/free/en/docs/papers/authors/nathan-schneider/) · **Year:** 2021 · **Venue:** Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems **Link:** [aclanthology.org/2021.eval4nlp-1.12](https://aclanthology.org/2021.eval4nlp-1.12/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**1** citations](https://openalex.org/works?filter=cites:W3213775472) openalex · 2026-10-18[**8** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/manning-2021-referenceless-parsing-based-evaluation-amr/) of the 140673 papers whose bibliographies we read[Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems](https://aclanthology.org/volumes/2021.eval4nlp-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2021.eval4nlp-1.12/)
- [PDF](https://aclanthology.org/2021.eval4nlp-1.12.pdf)
- [DOI 10.18653/v1/2021.eval4nlp-1.12](https://doi.org/10.18653/v1/2021.eval4nlp-1.12)
- [OpenAlex](https://openalex.org/W3213775472)

**Authors (2)**

1. [Emma Manning](https://openalex.org/A5006919541) h 3 · 45 citations · 12 works
2. [Nathan Schneider](https://openalex.org/A5014008069) [ORCID](https://orcid.org/0000-0002-2745-0533) · h 28 · 6684 citations · 159 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

Reference-based metrics such as BLEU compare a generated sentence with one human sentence, which is a poor fit for a meaning representation that many sentences can express. The paper tests a referenceless alternative for AMR-to-English generation: parse the generated sentence back into AMR and compare that graph with the input AMR using Smatch. Against the human adequacy judgements of Manning et al. (2020), the sentence-level correlation is 0.358 with the JAMR parser, 0.462 with Lyu and Titov, and 0.495 with Cai and Lam; the reference metrics reach 0.52 (BLEU) and 0.69 (BLEURT). When the parses of 406 sentences were corrected by hand, the correlation rose from 0.49 to 0.66 on the same sample. Parser misses on lower-case text, such as unrecognised names, are a main source of error. Because Smatch uses random restarts, the authors fixed the seed and raised the restarts to make the score reproducible.

## Ideas

- I1. Referenceless frame match: parse each generated sentence with the Mova pipeline into frames and score the overlap with the input frames, with no randomness. Check it against the open human judgements from the 2020 paper.
- I2. Recasing before parsing: lower-case output breaks named-entity recognition. Add a rule-based recaser that capitalises tokens found in the world's name list, then test whether the frame match rises on lower-case outputs.
- I3. Parse audit: correct a sample of Mova parses by hand, as the paper does, and report the gap between automatic and corrected scores before making any claim.
- I4. Determinism: the seeded, many-restart score is a model for any graph-matching measure in `tools/roundtrip`; two runs must give the same numbers.

## For Mova

Chapters: [07. Punctuation, capital letters, abbreviations and numbers](https://md.mova.elamur.ai/free/en/docs/languages/en/07-punctuation-capitals-numbers/) (capitals and proper names), [05. The writing system](https://md.mova.elamur.ai/free/en/docs/languages/en/05-writing-system/), [20. Syntax: the Universal Dependencies frame](https://md.mova.elamur.ai/free/en/docs/languages/en/20-syntax-overview/). Code: `tools/roundtrip` (the frame comparison), `en/src/tok.rs`, `en/src/parse.rs`. Measure: correlation with the human adequacy scores, with and without recasing, one run and a second identical run.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2021.eval4nlp-1.12](https://doi.org/10.18653/v1/2021.eval4nlp-1.12)

```
@inproceedings{manning2021referenceless,
  title = {{Referenceless Parsing-Based Evaluation of AMR-to-English Generation}},
  author = {Manning, Emma and Schneider, Nathan},
  booktitle = {Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems},
  year = {2021},
  doi = {10.18653/v1/2021.eval4nlp-1.12},
  url = {https://doi.org/10.18653/v1/2021.eval4nlp-1.12}
}
```

- [← previous A Human Evaluation of AMR-to-English Generation Systems](https://md.mova.elamur.ai/free/en/docs/papers/manning-2020-human-evaluation-amr-english-generation/)
- [next → Generating Typed Dependency Parses from Phrase Structure Parses](https://md.mova.elamur.ai/free/en/docs/papers/map-2006-marneffe-generating-typed-dependency-parses-phrase/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

16 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Laura+Banarescu%2C+Claire+Bonial%2C+Shu+Cai%2C+Madalina+Georgescu%2C+Kira+Griffitt%2C+Ulf+Hermjakob%2C+Kevin+Knight%2C+Philipp+Koehn%2C+Martha+Palmer%2C+and+Nathan+Schneider%2E+2013%2E+Abstract+Meaning+Representation+for+s) [our card](https://md.mova.elamur.ai/free/en/docs/papers/banarescu-2013-abstract-meaning-representation-sembanking/)
2. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+METEOR%3A+An+automatic+metric+for+MT+evaluation+with+improved+correlation+with+human+judgments%2E+In+Proceedings+of+the+ACL+Workshop+on+Intrinsic+and+Extrinsic+Eva) [our card](https://md.mova.elamur.ai/free/en/docs/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
3. Deng Cai and Wai Lam. 2020. AMR parsing via graph-sequence iterative inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1290–1301, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Deng+Cai+and+Wai+Lam%2E+2020%2E+AMR+parsing+via+graph%2Dsequence+iterative+inference%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1290%E2%80%931301%2C+Online%2E+Ass)
4. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shu+Cai+and+Kevin+Knight%2E+2013%2E+Smatch%3A+an+evaluation+metric+for+semantic+feature+structures%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Shor)
5. Jeffrey Flanigan, Chris Dyer, Noah A. Smith, and Jaime Carbonell. 2016. CMU at SemEval-2016 task 8: Graph-based AMR parsing with infinite ramp loss. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1202–1206, San Diego, California. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeffrey+Flanigan%2C+Chris+Dyer%2C+Noah+A%2E+Smith%2C+and+Jaime+Carbonell%2E+2016%2E+CMU+at+SemEval%2D2016+task+8%3A+Graph%2Dbased+AMR+parsing+with+infinite+ramp+loss%2E+In+Proceedings+of+the+10th+International+Workshop+o)
6. Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A discriminative graph-based parser for the Abstract Meaning Representation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426– 1436, Baltimore, Maryland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeffrey+Flanigan%2C+Sam+Thomson%2C+Jaime+Carbonell%2C+Chris+Dyer%2C+and+Noah+A%2E+Smith%2E+2014%2E+A+discriminative+graph%2Dbased+parser+for+the+Abstract+Meaning+Representation%2E+In+Proceedings+of+the+52nd+Annual+Meet)
7. Chunchuan Lyu and Ivan Titov. 2018. AMR parsing as graph prediction with latent alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 397–407, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chunchuan+Lyu+and+Ivan+Titov%2E+2018%2E+AMR+parsing+as+graph+prediction+with+latent+alignment%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Pa)
8. Emma Manning, Shira Wein, and Nathan Schneider. 2020. A human evaluation of AMR-to-English generation systems. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4773–4786, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emma+Manning%2C+Shira+Wein%2C+and+Nathan+Schneider%2E+2020%2E+A+human+evaluation+of+AMR%2Dto%2DEnglish+generation+systems%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+pages+4) [our card](https://md.mova.elamur.ai/free/en/docs/papers/manning-2020-human-evaluation-amr-english-generation/)
9. Juri Opitz and Anette Frank. 2021. Towards a decomposable metric for explainable evaluation of text generation from AMR. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1504–1518, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Juri+Opitz+and+Anette+Frank%2E+2021%2E+Towards+a+decomposable+metric+for+explainable+evaluation+of+text+generation+from+AMR%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+Associatio) [our card](https://md.mova.elamur.ai/free/en/docs/papers/opitz-2021-decomposable-metric-explainable-evaluation-text/)
10. Juri Opitz, Letitia Parcalabescu, and Anette Frank. 2020. AMR similarity metrics from principles. Transactions of the Association for Computational Linguistics, 8:522–538. [find](https://scholar.google.com/scholar?q=Juri+Opitz%2C+Letitia+Parcalabescu%2C+and+Anette+Frank%2E+2020%2E+AMR+similarity+metrics+from+principles%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A522%E2%80%93538%2E)
11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
12. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2017%2E+chrF%2B%2B%3A+words+helping+character+n%2Dgrams%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+pages+612%E2%80%93618%2C+Copenhagen%2C+Denmark%2E+Association+for+Computational+Linguist) [our card](https://md.mova.elamur.ai/free/en/docs/papers/popovic-2017-chrf-words-helping-character-n/)
13. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
14. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas, pages 223–231, Cambridge. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+Study+of+Translation+Edit+Rate+with+Targeted+Human+Annotation%2E+In+Proceedings+of+the+7th+Conference+of+the+As) [our card](https://md.mova.elamur.ai/free/en/docs/papers/snover-2006-study-translation-edit-rate-targeted/)
15. Linfeng Song and Daniel Gildea. 2019. SemBleu: A robust metric for AMR parsing evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4547– 4552, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Linfeng+Song+and+Daniel+Gildea%2E+2019%2E+SemBleu%3A+A+robust+metric+for+AMR+parsing+evaluation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4547%E2%80%93+4552%2C)
16. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In ICLR. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+BERTScore%3A+Evaluating+Text+Generation+with+BERT%2E+In+ICLR%2E)