[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/callaway-2005-types-distributions-errors-wide-coverage/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) / The Types and Distributions of Errors in a Wide Coverage Surface Realizer Evaluation

# The Types and Distributions of Errors in a Wide Coverage Surface Realizer Evaluation

**Authors:** Charles B. Callaway · **Year:** 2005 · **Venue:** Proceedings of the Tenth European Workshop on Natural Language Generation (ENLG-05) **Link:** [aclanthology.org/W05-1619](https://aclanthology.org/W05-1619/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**5** citations](https://openalex.org/works?filter=cites:W2786261708) openalex · 2026-10-19[**2** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/callaway-2005-types-distributions-errors-wide-coverage/) of the 140673 papers whose bibliographies we read[Proceedings of the Tenth European Workshop on Natural Language Generation (ENLG-05)](https://aclanthology.org/volumes/W05-16/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/W05-1619/)
- [PDF](https://aclanthology.org/W05-1619.pdf)
- [OpenAlex](https://openalex.org/W2786261708)

**Authors (1)**

1. [Callaway, Charles](https://openalex.org/A5046972209) [Semantic Scholar](https://www.semanticscholar.org/author/1730221) · h 18 · 1837 citations · 81 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

A study of why a wide-coverage realiser makes errors when it is scored on a treebank. The realiser is FUF/SURGE, scored on Penn Treebank section 23 after a transformer turns trees into its input. The newer version covers 98.5% of sentences with 69.3% exact matches, against 82.8% and 57.5% for the statistical HALogen, and has a higher simple string accuracy. The paper then asks where the remaining errors come from. The author generated 4,240 sentences from sections 20 to 22 and hand-checked 629 that had large accuracy penalties, classifying each error twice: first its source (corpus annotation, transformer, or grammar), then the grammar rule behind it. Most failing sentences trace back to the transformer, not the grammar; about 29% trace to syntax. Among syntactic constructions, mixed noun-phrase modifiers are the most frequent, while topicalisation has the highest average penalty per instance (about 73 edit-distance points). Rare constructions such as it-clefts are found with regular expressions, and the corpus tags for them are often wrong, so a treebank is a noisy test of rare phenomena.

## Ideas

- I1. Separate error sources in Mova's roundtrip report: corpus or annotation error, reader and transformer error, and encoder error, as the paper does. Estimate the corpus floor first by hand-checking a sample, so the topline is known before the rules are blamed.
- I2. Find rare constructions by pattern, and keep them as a probe set. The paper finds it-clefts and wh-clefts with regular expressions over text. In KJV and WEB, run the same kind of pattern as a held-out probe, not a training source, and count the correct and incorrect outputs.
- I3. Report the average penalty per error class next to the counts. Topicalisation was rare but costly; the translator's report should say which class costs most per verse, not only how many verses fail.
- I4. Coverage beside exact match. The paper's gain came mostly from coverage. Give the translator's normal-mode report the share of verses translated without a fallback next to the exact-match rate.

## For Mova

The measures belong in `tools/roundtrip` (divergence classes) and `tools/translate/src/measure.rs`. The ideas touch the residual-by-class measure (open problem 2) and the reader's noise (open problem 1). The paper's counts are for English newspaper text and do not transfer; its method does.

Topics: [Treebanks](https://md.mova.elamur.ai/free/en/docs/papers/topics/treebanks/) · [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{callaway2005types,
  title = {{The Types and Distributions of Errors in a Wide Coverage Surface Realizer Evaluation}},
  author = {Callaway, Charles},
  booktitle = {Proceedings of the Tenth European Workshop on Natural Language Generation (ENLG-05)},
  year = {2005},
  url = {https://aclanthology.org/W05-1619/}
}
```

- [← previous Deterministic natural language generation from meaning representations for machine translation](https://md.mova.elamur.ai/free/en/docs/papers/butler-2016-deterministic-natural-language-generation-meaning/)
- [next → Factorising AMR generation through syntax](https://md.mova.elamur.ai/free/en/docs/papers/cao-2019-factorising-amr-generation-through-syntax/)
- [↑ back to Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/)

## References

18 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Bangalore and O. Rambow. Exploiting a probabilistic hierarchical model for generation. In COLING–2000: Proceedings of the 18th International Conference on Computational Linguistics, Saarbruecken, Germany, 2000. [find](https://scholar.google.com/scholar?q=S%2E+Bangalore+and+O%2E+Rambow%2E+Exploiting+a+probabilistic+hierarchical+model+for+generation%2E+In+COLING%E2%80%932000%3A+Proceedings+of+the+18th+International+Conference+on+Computational+Linguistics%2C+Saarbruecken%2C+G)
2. John A. Bateman. KPML: The KOMET-penman (multilingual) development environment. Technical Report Release 0.8, Institut für Integrierte Publikations- und Informationssysteme (IPSI), GMD, Darmstadt, 1995. [find](https://scholar.google.com/scholar?q=John+A%2E+Bateman%2E+KPML%3A+The+KOMET%2Dpenman+%28multilingual%29+development+environment%2E+Technical+Report+Release+0%2E8%2C+Institut+fu%CC%88r+Integrierte+Publikations%2D+und+Informationssysteme+%28IPSI%29%2C+GMD%2C+Darmstadt%2C+19)
3. Charles B. Callaway. Evaluating coverage for large symbolic NLG grammars. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, pages 811–817, Acapulco, Mexico, August 2003. [find](https://scholar.google.com/scholar?q=Charles+B%2E+Callaway%2E+Evaluating+coverage+for+large+symbolic+NLG+grammars%2E+In+Proceedings+of+the+Eighteenth+International+Joint+Conference+on+Artificial+Intelligence%2C+pages+811%E2%80%93817%2C+Acapulco%2C+Mexico%2C+A)
4. Charles Callaway. Wide coverage symbolic surface realization. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 125–128, Barcelona, Spain, July 2004. [find](https://scholar.google.com/scholar?q=Charles+Callaway%2E+Wide+coverage+symbolic+surface+realization%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+125%E2%80%93128%2C+Barcelona%2C+Spain%2C+July+2004%2E)
5. M. Dickinson and D. Meurers. Detecting errors in part-of-speech annotation. In Proceedings of the 10th Conference of the European Chapter of the ACL, Budapest, Hungary, April 2003. [find](https://scholar.google.com/scholar?q=M%2E+Dickinson+and+D%2E+Meurers%2E+Detecting+errors+in+part%2Dof%2Dspeech+annotation%2E+In+Proceedings+of+the+10th+Conference+of+the+European+Chapter+of+the+ACL%2C+Budapest%2C+Hungary%2C+April+2003%2E)
6. George Doddington. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the 2002 Conference on Human Language Technology, San Diego, CA, March 2002. [find](https://scholar.google.com/scholar?q=George+Doddington%2E+Automatic+evaluation+of+machine+translation+quality+using+n%2Dgram+co%2Doccurrence+statistics%2E+In+Proceedings+of+the+2002+Conference+on+Human+Language+Technology%2C+San+Diego%2C+CA%2C+March+2)
7. Christine Doran. Incorporating Punctuation into the Sentence Grammar: A Lexicalized Tree Adjoining Grammar Perspective. PhD thesis, University of Pennsylvania, Philadelphia, PA, 1998. [find](https://scholar.google.com/scholar?q=Christine+Doran%2E+Incorporating+Punctuation+into+the+Sentence+Grammar%3A+A+Lexicalized+Tree+Adjoining+Grammar+Perspective%2E+PhD+thesis%2C+University+of+Pennsylvania%2C+Philadelphia%2C+PA%2C+1998%2E)
8. Michael Elhadad. FUF: The universal unifier user manual version 5.0. Technical Report CUCS-038-91, Dept. of Computer Science, Columbia University, 1991. Total SSA Penalty 601 8.99% 47 0.70% 1080 16.15% 1197 17.90% 560 8.37% 307 4.59% 542 8.11% 1231 18.41% 317 4.74% 805 12.04% 6687 100.0% Avg. Penalty 50.08 15.67 41.54 18.70 56.00 34.11 41.69 42.45 19.81 73.18 34.65 [find](https://scholar.google.com/scholar?q=Michael+Elhadad%2E+FUF%3A+The+universal+unifier+user+manual+version+5%2E0%2E+Technical+Report+CUCS%2D038%2D91%2C+Dept%2E+of+Computer+Science%2C+Columbia+University%2C+1991%2E+Total+SSA+Penalty+601+8%2E99%25+47+0%2E70%25+1080+16%2E15)
9. Georgia Green. Pragmatic motivation and exploitation of syntactic rules. Technical report, University of Illinois, [http://www.linguistics.uiuc.edu/g-green/441/prmoex/index.html](http://www.linguistics.uiuc.edu/g-green/441/prmoex/index.html), 2001. [↗](http://www.linguistics.uiuc.edu/g-green/441/prmoex/index.html)
10. Michael Halliday. System and Function in Language. Oxford University Press, Oxford, 1976. [find](https://scholar.google.com/scholar?q=Michael+Halliday%2E+System+and+Function+in+Language%2E+Oxford+University+Press%2C+Oxford%2C+1976%2E)
11. Irene Langkilde and Kevin Knight. Generation that exploits corpus-based statistical knowledge. In COLING-ACL-98: Proceedings of the Joint 36th Meeting of the ACL andthe 17th International COL-ING, pages 704–710, Montréal, Canada, August 1998. [find](https://scholar.google.com/scholar?q=Irene+Langkilde+and+Kevin+Knight%2E+Generation+that+exploits+corpus%2Dbased+statistical+knowledge%2E+In+COLING%2DACL%2D98%3A+Proceedings+of+the+Joint+36th+Meeting+of+the+ACL+andthe+17th+International+COL%2DING%2C+pag)
12. Irene Langkilde-Geary. An empirical verification of coverage and correctness for a general-purpose sentence generator. In Second International Natural Language Generation Conference, Harriman, NY, July 2002. [find](https://scholar.google.com/scholar?q=Irene+Langkilde%2DGeary%2E+An+empirical+verification+of+coverage+and+correctness+for+a+general%2Dpurpose+sentence+generator%2E+In+Second+International+Natural+Language+Generation+Conference%2C+Harriman%2C+NY%2C+Jul)
13. Benoit Lavoie and Owen Rambow. A fast and portable realizer for text generation systems. In Proceedings of the 5th Conference on Applied Natural Language Processing, 1997. [find](https://scholar.google.com/scholar?q=Benoit+Lavoie+and+Owen+Rambow%2E+A+fast+and+portable+realizer+for+text+generation+systems%2E+In+Proceedings+of+the+5th+Conference+on+Applied+Natural+Language+Processing%2C+1997%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lavoie-1997-fast-portable-realizer-text-generation/)
14. M. Marcus, B. Santorini, and M. Marcinkiewicz. Building a large annotated corpus of English: The PennTreeBank. Computational Linguistics, 26(2), 1993. [find](https://scholar.google.com/scholar?q=M%2E+Marcus%2C+B%2E+Santorini%2C+and+M%2E+Marcinkiewicz%2E+Building+a+large+annotated+corpus+of+English%3A+The+PennTreeBank%2E+Computational+Linguistics%2C+26%282%29%2C+1993%2E)
15. Igor A. Mel’cuk. Dependency Syntax: Theory and Practice. SUNY Publications, 1988. [find](https://scholar.google.com/scholar?q=Igor+A%2E+Mel%E2%80%99cuk%2E+Dependency+Syntax%3A+Theory+and+Practice%2E+SUNY+Publications%2C+1988%2E)
16. K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. BLEU: A method for automatic evaluation of MT. Technical Report RC22176, IBM Research, New York, September 2001. [find](https://scholar.google.com/scholar?q=K%2E+Papineni%2C+S%2E+Roukos%2C+T%2E+Ward%2C+and+W%2E+J%2E+Zhu%2E+BLEU%3A+A+method+for+automatic+evaluation+of+MT%2E+Technical+Report+RC22176%2C+IBM+Research%2C+New+York%2C+September+2001%2E)
17. Adwait Ratnaparkhi. Trainable methods for surface natural language generation. In Proceedings of the First North American Conference of the ACL, Seattle, WA, May 2000. [find](https://scholar.google.com/scholar?q=Adwait+Ratnaparkhi%2E+Trainable+methods+for+surface+natural+language+generation%2E+In+Proceedings+of+the+First+North+American+Conference+of+the+ACL%2C+Seattle%2C+WA%2C+May+2000%2E)
18. Michael White and Ted Caldwell. EXEMPLARS: A practical, extensible framework for dynamic text generation. In Proceedings of the Ninth International Workshop on NLG, pages 266–275, Niagara-on-the-Lake, Ontario, August 1998. [find](https://scholar.google.com/scholar?q=Michael+White+and+Ted+Caldwell%2E+EXEMPLARS%3A+A+practical%2C+extensible+framework+for+dynamic+text+generation%2E+In+Proceedings+of+the+Ninth+International+Workshop+on+NLG%2C+pages+266%E2%80%93275%2C+Niagara%2Don%2Dthe%2DLake%2C)