[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/belz-2011-first-surface-realisation-shared-task/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) / The First Surface Realisation Shared Task: Overview and Evaluation Results

# The First Surface Realisation Shared Task: Overview and Evaluation Results

**Authors:** [Anja Belz](https://md.mova.elamur.ai/free/en/docs/papers/authors/anja-belz/), [Michael White](https://md.mova.elamur.ai/free/en/docs/papers/authors/michael-white/), Dominic Espinosa, Eric Kow, Deirdre Hogan, [Amanda Stent](https://md.mova.elamur.ai/free/en/docs/papers/authors/amanda-stent/) · **Year:** 2011 · **Venue:** ENLG 2011 (13th European Workshop on Natural Language Generation) **Link:** [aclanthology.org/W11-2832](https://aclanthology.org/W11-2832/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**105** citations](https://openalex.org/works?filter=cites:W2125417976) openalex · 2026-10-19[**68** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/belz-2011-first-surface-realisation-shared-task/) of the 140673 papers whose bibliographies we read[Proceedings of the 13th European Workshop on Natural Language Generation](https://aclanthology.org/volumes/W11-28/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/W11-2832/)
- [PDF](https://aclanthology.org/W11-2832.pdf)
- [OpenAlex](https://openalex.org/W2125417976)

**Authors (6)**

1. [Anja Belz](https://openalex.org/A5010118897) [ORCID](https://orcid.org/0000-0002-0552-8096) · [Semantic Scholar](https://www.semanticscholar.org/author/1792458) · h 24 · 2159 citations · 135 works
2. [Michael White](https://openalex.org/A5025147385) [ORCID](https://orcid.org/0000-0002-3062-1719) · [Semantic Scholar](https://www.semanticscholar.org/author/48312837) · h 35 · 12011 citations · 223 works
3. [Dominic Espinosa](https://openalex.org/A5078535853) [Semantic Scholar](https://www.semanticscholar.org/author/144797578) · h 5 · 181 citations · 6 works
4. [Eric Kow](https://openalex.org/A5053478598) [Semantic Scholar](https://www.semanticscholar.org/author/1726297) · h 13 · 599 citations · 58 works
5. [Deirdre Hogan](https://openalex.org/A5057165717) [Semantic Scholar](https://www.semanticscholar.org/author/49494454) · h 10 · 519 citations · 32 works
6. [Amanda Stent](https://openalex.org/A5053693571) [ORCID](https://orcid.org/0000-0002-1532-5903) · [Semantic Scholar](https://www.semanticscholar.org/author/1690152) · h 31 · 4359 citations · 157 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

The paper reports the first surface realisation shared task. Teams received a common input (a shallow track and a deep track, both derived from Penn Treebank parses) and produced English sentences. Five teams submitted six systems, and the corpus sentences were scored as a human topline. Systems were judged automatically with BLEU and related metrics, and by human raters on clarity, readability and meaning similarity. The statistical dependency realisers performed best. The two systems that used a hand-written grammar or a reversible grammar did poorly, and the authors trace much of this to converting the common input into the form their realisers expected, which the report treats as a large obstacle. The paper also notes that BLEU scores from earlier, non-shared work could not be compared, because each used a different input.

## Ideas

- I1. Report input conversion coverage separately from realiser quality. The report shows that a weak result can come from the conversion of the input rather than from the realiser. The translator's residual per layer already separates these; keep that split in every table.
- I2. Use one common input for all encoder variants, so that comparisons between author deltas run on the same meta-world and cannot differ by conversion.
- I3. Add the reference text as a row in every table, as the human topline does, so the gap to the ceiling is always visible.
- I4. Split human-style judgements into clarity, readability and meaning similarity. For Mova's normal mode, report exact match, chrF and a meaning-only score on the same rows, so that a fluent but wrong output shows as a meaning drop.

## For Mova

The ideas fit `tools/translate/measure.rs` and the layout of its README tables. The shared-task design, a common-ground input for comparing systems, is the same argument behind the translator's meta-world, and the paper's warning about conversion cost is a reason to measure the reader separately.

Topics: [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{belz2011first,
  title = {{The First Surface Realisation Shared Task: Overview and Evaluation Results}},
  author = {Belz, Anja and White, Michael and Espinosa, Dominic and Kow, Eric and Hogan, Deirdre and Stent, Amanda},
  booktitle = {Proceedings of the 13th European Workshop on Natural Language Generation},
  year = {2011},
  url = {https://aclanthology.org/W11-2832/}
}
```

- [← previous Machine Translation Using Isomorphic UCGs](https://md.mova.elamur.ai/free/en/docs/papers/beaven-1988-machine-translation-isomorphic-ucgs/)
- [next → Die Aufgabe des Übersetzers (The Task of the Translator)](https://md.mova.elamur.ai/free/en/docs/papers/benjamin-1923-die-aufgabe-des-ubersetzers/)
- [↑ back to Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/)

## References

17 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proc. of the ACL-05 Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+METEOR%3A+An+Automatic+Metric+for+MT+Evaluation+with+Improved+Correlation+with+Human+Judgments%2E+In+Proc%2E+of+the+ACL%2D05+Workshop+on+Intrinsic+and+Extrinsic+Evalua) [our card](https://md.mova.elamur.ai/free/en/docs/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
2. Anja Belz and Eric Kow. 2011. Discrete vs. continuous rating scales for language evaluation in nlp. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL-HLT’11). [find](https://scholar.google.com/scholar?q=Anja+Belz+and+Eric+Kow%2E+2011%2E+Discrete+vs%2E+continuous+rating+scales+for+language+evaluation+in+nlp%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%2DHLT%E2%80%99)
3. A. Belz, M. White, J. van Genabith, D. Hogan, and A. Stent. 2010. Finding common ground: Towards a Clarity Mean Readability Mean Homogeneous subsets Corpus 88.06 A STUMABA - D 64.91 B OSU 22.83 C Homogeneous subsets A B C Corpus 83.28 60.30 22.71 STUMABA - D OSU Table 9: SR-Task, Deep Track: Results for Clarity, Readability and Meaning Similarity evaluations, in terms of means and homogeneous subsets determined by post-hoc Tukey’s HSD (sig. < 0.05). Surface Realisation shared task. In Proc. of INLG’10, pages 267–271. [find](https://scholar.google.com/scholar?q=A%2E+Belz%2C+M%2E+White%2C+J%2E+van+Genabith%2C+D%2E+Hogan%2C+and+A%2E+Stent%2E+2010%2E+Finding+common+ground%3A+Towards+a+Clarity+Mean+Readability+Mean+Homogeneous+subsets+Corpus+88%2E06+A+STUMABA+%2D+D+64%2E91+B+OSU+22%2E83+C+Homo)
4. Bernd Bohnet, Leo Wanner, Simon Mille, and Alicia Burga. 2010. Broad coverage multilingual deep sentence generation with a stochastic multi-level realizer. In Proceedings of the 23rd International Conference on Computational Linguistics, Beijing, China. [find](https://scholar.google.com/scholar?q=Bernd+Bohnet%2C+Leo+Wanner%2C+Simon+Mille%2C+and+Alicia+Burga%2E+2010%2E+Broad+coverage+multilingual+deep+sentence+generation+with+a+stochastic+multi%2Dlevel+realizer%2E+In+Proceedings+of+the+23rd+International+Con) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bohnet-2010-broad-coverage-multilingual-deep-sentence/)
5. A. Cahill and J. van Genabith. 2006. Robust PCFG-based generation using automatically acquired LFG approximations. In Proc. ACL’06, pages 1033–44. [find](https://scholar.google.com/scholar?q=A%2E+Cahill+and+J%2E+van+Genabith%2E+2006%2E+Robust+PCFG%2Dbased+generation+using+automatically+acquired+LFG+approximations%2E+In+Proc%2E+ACL%E2%80%9906%2C+pages+1033%E2%80%9344%2E)
6. Charles Callaway. 2003. Evaluating coverage for large symbolic NLG grammars. In Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI 2003), pages 811–817. [find](https://scholar.google.com/scholar?q=Charles+Callaway%2E+2003%2E+Evaluating+coverage+for+large+symbolic+NLG+grammars%2E+In+Proceedings+of+the+18th+International+Joint+Conference+on+Artificial+Intelligence+%28IJCAI+2003%29%2C+pages+811%E2%80%93817%2E)
7. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the EMNLP 2011 Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Michael+Denkowski+and+Alon+Lavie%2E+2011%2E+Meteor+1%2E3%3A+Automatic+Metric+for+Reliable+Optimization+and+Evaluation+of+Machine+Translation+Systems%2E+In+Proceedings+of+the+EMNLP+2011+Workshop+on+Statistical+M)
8. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to-dependency conversion for english. In Joakim Nivre, Heiki-Jaan Kaalep, Kadri Muischnek, and Mare Koit, editors, Proceedings of NODALIDA 2007, pages 105–112, Tartu, Estonia. [find](https://scholar.google.com/scholar?q=Richard+Johansson+and+Pierre+Nugues%2E+2007%2E+Extended+constituent%2Dto%2Ddependency+conversion+for+english%2E+In+Joakim+Nivre%2C+Heiki%2DJaan+Kaalep%2C+Kadri+Muischnek%2C+and+Mare+Koit%2C+editors%2C+Proceedings+of+NODALI) [our card](https://md.mova.elamur.ai/free/en/docs/papers/map-2007-johansson-extended-constituent-dependency-conversion-english/)
9. I. Langkilde. 2002. An empirical verification of coverage and correctness for a general-purpose sentence generator. In Proc. 2nd International Natural Language Generation Conference (INLG ’02). [find](https://scholar.google.com/scholar?q=I%2E+Langkilde%2E+2002%2E+An+empirical+verification+of+coverage+and+correctness+for+a+general%2Dpurpose+sentence+generator%2E+In+Proc%2E+2nd+International+Natural+Language+Generation+Conference+%28INLG+%E2%80%9902%29%2E)
10. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1995. The PENN treebank: Annotating predicate argument structure. Distributed on the PENN Treebank Release 2 CD-ROM, Linguistic Data Consortium. [find](https://scholar.google.com/scholar?q=Mitchell+Marcus%2C+Grace+Kim%2C+Mary+Ann+Marcinkiewicz%2C+Robert+MacIntyre%2C+Ann+Bies%2C+Mark+Ferguson%2C+Karen+Katz%2C+and+Britta+Schasberger%2E+1995%2E+The+PENN+treebank%3A+Annotating+predicate+argument+structure%2E+Dis) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1994-marcus-penn-treebank-annotating-predicate-argument/)
11. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The nombank project: An interim report. In NAACL/HLT Workshop Frontiers in Corpus Annotation. [find](https://scholar.google.com/scholar?q=Adam+Meyers%2C+Ruth+Reeves%2C+Catherine+Macleod%2C+Rachel+Szekely%2C+Veronika+Zielinska%2C+Brian+Young%2C+and+Ralph+Grishman%2E+2004%2E+The+nombank+project%3A+An+interim+report%2E+In+NAACL%2FHLT+Workshop+Frontiers+in+Corpu)
12. Hiroko Nakanishi, Yusuke Miyao, and Jun’ichi Tsujii. 2005. Probabilistic models for disambiguation of an hpsg-based chart generator. In Proceedings of the 9th International Workshop on Parsing Technology (Parsing’05), pages 93–102. Association for Computational Linguistics. Meaning Similarity Mean Homogeneous subsets Corpus 99.86 A STUMABA - D 72.54 B OSU 32.44 C Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: A corpus annotated with semantic roles. In Computational Linguistics Journal, pages 71–105. [find](https://scholar.google.com/scholar?q=Hiroko+Nakanishi%2C+Yusuke+Miyao%2C+and+Jun%E2%80%99ichi+Tsujii%2E+2005%2E+Probabilistic+models+for+disambiguation+of+an+hpsg%2Dbased+chart+generator%2E+In+Proceedings+of+the+9th+International+Workshop+on+Parsing+Technol)
13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+40th+Annual+Meeting+of+the+Association+for+Computat) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
14. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proc. of the Association for Machine Translation in the Americas (AMTA-06). [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+Proc%2E+of+the+Association+for+Machine+Transl) [our card](https://md.mova.elamur.ai/free/en/docs/papers/snover-2006-study-translation-edit-rate-targeted/)
15. Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluı́s Màrquez, and Joakim Nivre. 2008. The conll-2008 shared task on joint parsing of syntactic and semantic dependencies. In Proceedings of the Twelfth Conference on Computational Natural Language Learning, Manchester, UK. [find](https://scholar.google.com/scholar?q=Mihai+Surdeanu%2C+Richard+Johansson%2C+Adam+Meyers%2C+Llu%C4%B1%CC%81s+Ma%CC%80rquez%2C+and+Joakim+Nivre%2E+2008%2E+The+conll%2D2008+shared+task+on+joint+parsing+of+syntactic+and+semantic+dependencies%2E+In+Proceedings+of+the+Twelf) [our card](https://md.mova.elamur.ai/free/en/docs/papers/surdeanu-2008-conll-2008-shared-task-joint/)
16. Michael White and Rajakrishnan Rajkumar. 2009. Perceptron reranking for CCG realisation. In Proceedings of the 2009 Conference on Empririal Methods in Natural Language Processing (EMNLP’09), pages 410– 419. [find](https://scholar.google.com/scholar?q=Michael+White+and+Rajakrishnan+Rajkumar%2E+2009%2E+Perceptron+reranking+for+CCG+realisation%2E+In+Proceedings+of+the+2009+Conference+on+Empririal+Methods+in+Natural+Language+Processing+%28EMNLP%E2%80%9909%29%2C+pages+410)
17. H. Zhong and A. Stent. 2005. Building surface realizers automatically from corpora. In A. Belz and S. Varges, editors, Proceedings of UCNLG’05, pages 49–54. [find](https://scholar.google.com/scholar?q=H%2E+Zhong+and+A%2E+Stent%2E+2005%2E+Building+surface+realizers+automatically+from+corpora%2E+In+A%2E+Belz+and+S%2E+Varges%2C+editors%2C+Proceedings+of+UCNLG%E2%80%9905%2C+pages+49%E2%80%9354%2E)