[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/enud-1999-carroll-corpus-annotation-parser-evaluation/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Corpus Annotation for Parser Evaluation

# Corpus Annotation for Parser Evaluation

**Authors:** [Ted Briscoe](https://md.mova.elamur.ai/free/en/docs/papers/authors/ted-briscoe/), [John Carroll](https://md.mova.elamur.ai/free/en/docs/papers/authors/john-carroll/), [Guido Minnen](https://md.mova.elamur.ai/free/en/docs/papers/authors/guido-minnen/) · **Year:** 1999 · **Venue:** Proceedings of the EACL Workshop on Linguistically Interpreted Corpora (LINC), Bergen, Norway **Link:** [arxiv.org/abs/cs/9907013](https://arxiv.org/abs/cs/9907013) · **Licence:** arXiv non-exclusive licence — summary only

[**85** citations](https://openalex.org/works?filter=cites:W1625716732) openalex · 2026-10-18[**28** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/enud-1999-carroll-corpus-annotation-parser-evaluation/) of the 140673 papers whose bibliographies we read[Proceedings of the EACL99 workshop on Linguistically Interpreted Corpora (LINC), Bergen, Norway, June 12](https://openalex.org/S4306400194)[licence](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

- [Publisher's page](https://arxiv.org/abs/cs/9907013)
- [PDF](https://arxiv.org/pdf/cs/9907013)
- [arXiv cs/9907013](https://arxiv.org/abs/cs/9907013)
- [OpenAlex](https://openalex.org/W1625716732)

**Authors (3)**

1. [John Carroll](https://openalex.org/A5101474712) [ORCID](https://orcid.org/0000-0003-3744-4919) · h 24 · 2447 citations · 120 works
2. [Guido Minnen](https://openalex.org/A5073035830) h 8 · 869 citations · 28 works
3. [Ted Briscoe](https://openalex.org/A5030189297) h 42 · 7666 citations · 222 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

Bracket-based parser scores (PARSEVAL) are hard to compare across grammars, because the bracketing of a treebank and the output of a parser differ in ways that are not errors. The authors propose an annotation scheme that records grammatical relations between heads and dependents (subject, object, modifier and so on), organised in a hierarchy so that a shallow parser may leave a label underspecified. They mark up a corpus of 500 sentences (10K words) taken from SUSANNE, with 9.72 relations per sentence and around 95% inter-annotator agreement. A robust lexicalised parser scores an F-score of 75.1 on the general dependent relation, 86.5 on non-clausal subjects and 50.7 on indirect objects.

## Ideas

- I1. `en parse-eval`: add a per-label table (precision, recall, F) for the UD labels on EWT test, in the paper's style, with the general `dep` score first. Measure which labels lower LAS most, and start with the lowest-F label.
- I2. Hierarchical diagnostic: count a predicted label as a near miss when it shares a parent with the gold label (`nsubj` and `csubj`, `obl` and `nmod`). Report this beside LAS as a diagnostic only, never as a tuning target. Measure: the gap between LAS and the near-miss score on EWT test.
- I3. Control in `en/src/expert.rs`: a rule that adds a second, enhanced edge when one nominal is the subject of two verbs (the paper's example of control). Trees stay single-headed; the extra edge is reported only. Measure: `nsubj` F on EWT test with and without the enhanced edges, and the number of such cases.

## For Mova

The chapters are [20. Syntax: the Universal Dependencies frame](https://md.mova.elamur.ai/free/en/docs/languages/en/20-syntax-overview/) and `23-complementation.md` (control). The code is `en/src/parse.rs` with the `en parse-eval` command, and `en/src/expert.rs` for the control rule. The paper's corpus is not in Mova's data, and its numbers belong to a 1999 parser, so they are not comparable. Every figure must come from EWT test, which is measured once per release.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv cs/9907013](https://arxiv.org/abs/cs/9907013)

```
@inproceedings{carroll1999corpus,
  title = {{Corpus Annotation for Parser Evaluation}},
  author = {Carroll, John and Minnen, Guido and Briscoe, Ted},
  booktitle = {Proceedings of the EACL99 workshop on Linguistically Interpreted Corpora (LINC), Bergen, Norway, June 12},
  year = {1999},
  eprint = {cs/9907013},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/cs/9907013}
}
```

- [← previous Rhetorical Structure Theory: Toward a functional theory of text organization](https://md.mova.elamur.ai/free/en/docs/papers/enud-1988-mann-rhetorical-structure-theory-functional/)
- [next → Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics?](https://md.mova.elamur.ai/free/en/docs/papers/enud-2011-manning-part-speech-tagging-97-100/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

24 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Atwell, E. (1996) Comparative evaluation of grammatical annotation models. In R. Sutcliffe, H. Koch & A. McElligott (Eds.), Industrial Parsing of Software Manuals, 25–46. Amsterdam: Rodopi. [find](https://scholar.google.com/scholar?q=Atwell%2C+E%2E+%281996%29+Comparative+evaluation+of+grammatical+annotation+models%2E+In+R%2E+Sutcliffe%2C+H%2E+Koch+%26+A%2E+McElligott+%28Eds%2E%29%2C+Industrial+Parsing+of+Software+Manuals%2C+25%E2%80%9346%2E+Amsterdam%3A+Rodopi%2E)
2. Bies, A., Ferguson, M., Katz, K., MacIntyre, R., Tredinnick, V., Kim, G., Marcinkiewicz, M., Schasberger, B. (1995) Bracketing guidelines for Treebank II style Penn Treebank Project. Technical Report, CIS, University of Pennsylvania, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Bies%2C+A%2E%2C+Ferguson%2C+M%2E%2C+Katz%2C+K%2E%2C+MacIntyre%2C+R%2E%2C+Tredinnick%2C+V%2E%2C+Kim%2C+G%2E%2C+Marcinkiewicz%2C+M%2E%2C+Schasberger%2C+B%2E+%281995%29+Bracketing+guidelines+for+Treebank+II+style+Penn+Treebank+Project%2E+Technical+Report%2C)
3. Carpenter, B. & Manning, C. (1997) Probabilistic parsing using left corner language models. In Proceedings of the 5th ACL/SIGPARSE International Workshop on Parsing Technologies. MIT, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Carpenter%2C+B%2E+%26+Manning%2C+C%2E+%281997%29+Probabilistic+parsing+using+left+corner+language+models%2E+In+Proceedings+of+the+5th+ACL%2FSIGPARSE+International+Workshop+on+Parsing+Technologies%2E+MIT%2C+Cambridge%2C+MA%2E)
4. Carroll, J., Briscoe E. & Sanfilippo, A. (1998) Parser evaluation: a survey and a new proposal. In Proceedings of the International Conference on Language Resources and Evaluation, 447–454. Granada, Spain. Available online at <ftp://ftp.cogs.susx.ac.uk/pub/users/johnca/ lre98-final.ps>. [find](https://scholar.google.com/scholar?q=Carroll%2C+J%2E%2C+Briscoe+E%2E+%26+Sanfilippo%2C+A%2E+%281998%29+Parser+evaluation%3A+a+survey+and+a+new+proposal%2E+In+Proceedings+of+the+International+Conference+on+Language+Resources+and+Evaluation%2C+447%E2%80%93454%2E+Granada%2C+S)
5. Carroll, J., Minnen, G. & Briscoe E. (1998) Can subcategorisation probabilities help a statistical parser?. In Proceedings of the 6th ACL/SIGDAT Workshop on Very Large Corpora. Montreal, Canada. Available online at <http://xxx.lanl.gov/abs/cmp-lg/9806013>. [↗](https://arxiv.org/abs/cmp-lg/9806013)
6. Charniak, E. (1996) Tree-bank grammars. In Proceedings of the 13th National Conference on Artificial Intelligence, AAAI-96, 1031–1036. . [find](https://scholar.google.com/scholar?q=Charniak%2C+E%2E+%281996%29+Tree%2Dbank+grammars%2E+In+Proceedings+of+the+13th+National+Conference+on+Artificial+Intelligence%2C+AAAI%2D96%2C+1031%E2%80%931036%2E+%2E)
7. Collins, M. (1996) A new statistical parser based on bigram lexical dependencies. In Proceedings of the 34th Meeting of the Association for Computational Linguistics, 184–191. Santa Cruz, CA. [find](https://scholar.google.com/scholar?q=Collins%2C+M%2E+%281996%29+A+new+statistical+parser+based+on+bigram+lexical+dependencies%2E+In+Proceedings+of+the+34th+Meeting+of+the+Association+for+Computational+Linguistics%2C+184%E2%80%93191%2E+Santa+Cruz%2C+CA%2E)
8. Gaizauskas, R. (1998) Special issue on evaluation. Computer Speech & Language. [find](https://scholar.google.com/scholar?q=Gaizauskas%2C+R%2E+%281998%29+Special+issue+on+evaluation%2E+Computer+Speech+%26+Language%2E)
9. Gaizauskas, R., Hepple M. & Huyck, C. (1998) Modifying existing annotated corpora for general comparative evaluation of parsing. In Proceedings of the LRE Workshop on Evaluation of Parsing Systems. Granada, Spain. [find](https://scholar.google.com/scholar?q=Gaizauskas%2C+R%2E%2C+Hepple+M%2E+%26+Huyck%2C+C%2E+%281998%29+Modifying+existing+annotated+corpora+for+general+comparative+evaluation+of+parsing%2E+In+Proceedings+of+the+LRE+Workshop+on+Evaluation+of+Parsing+Systems%2E+Gr)
10. Grishman, R., Macleod, C. & Sterling, J. (1992) Evaluating parsing strategies using standardized parse files. In Proceedings of the 3rd ACL Conference on Applied Natural Language Processing, 156–161. Trento, Italy. [find](https://scholar.google.com/scholar?q=Grishman%2C+R%2E%2C+Macleod%2C+C%2E+%26+Sterling%2C+J%2E+%281992%29+Evaluating+parsing+strategies+using+standardized+parse+files%2E+In+Proceedings+of+the+3rd+ACL+Conference+on+Applied+Natural+Language+Processing%2C+156%E2%80%93161%2E+)
11. Harrison, P., Abney, S., Black, E., Flickinger, D., Gdaniec, C., Grishman, R., Hindle, D., Ingria, B., Marcus, M., Santorini, B. & Strzalkowski, T. (1991) Evaluating syntax performance of parser/grammars of English. In Proceedings of the Workshop on Evaluating Natural Language Processing Systems. ACL. [find](https://scholar.google.com/scholar?q=Harrison%2C+P%2E%2C+Abney%2C+S%2E%2C+Black%2C+E%2E%2C+Flickinger%2C+D%2E%2C+Gdaniec%2C+C%2E%2C+Grishman%2C+R%2E%2C+Hindle%2C+D%2E%2C+Ingria%2C+B%2E%2C+Marcus%2C+M%2E%2C+Santorini%2C+B%2E+%26+Strzalkowski%2C+T%2E+%281991%29+Evaluating+syntax+performance+of+parser%2Fgramm)
12. Leech, G. & Garside, R. (1991) Running a grammar factory: the production of syntactically analysed corpora or ‘treebanks’. In S. Johansson & A. Stenstrom (Eds.), English Computer Corpora: Selected Papers and Bibliography, Mouton de Gruyter, Berlin. [find](https://scholar.google.com/scholar?q=Leech%2C+G%2E+%26+Garside%2C+R%2E+%281991%29+Running+a+grammar+factory%3A+the+production+of+syntactically+analysed+corpora+or+%E2%80%98treebanks%E2%80%99%2E+In+S%2E+Johansson+%26+A%2E+Stenstrom+%28Eds%2E%29%2C+English+Computer+Corpora%3A+Selected+Pap)
13. Lehmann, S., Oepen, S., Regnier-Prost, S., Netter, K., Lux, V., Klein, J., Falkedal, K., Fouvry, F., Estival, D., Dauphin, E., Compagnion, H., Baur, J., Balkan, L. & Arnold, D. (1996) tsnlp — test suites for natural language processing. In Proceedings of the International Conference on Computational Linguistics, COLING-96, 711–716. Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Lehmann%2C+S%2E%2C+Oepen%2C+S%2E%2C+Regnier%2DProst%2C+S%2E%2C+Netter%2C+K%2E%2C+Lux%2C+V%2E%2C+Klein%2C+J%2E%2C+Falkedal%2C+K%2E%2C+Fouvry%2C+F%2E%2C+Estival%2C+D%2E%2C+Dauphin%2C+E%2E%2C+Compagnion%2C+H%2E%2C+Baur%2C+J%2E%2C+Balkan%2C+L%2E+%26+Arnold%2C+D%2E+%281996%29+tsnlp+%E2%80%94+test+sui)
14. Lin, D. (1995) A dependency-based method for evaluating broad-coverage parsers. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, 1420–1425. Montreal, Canada. [find](https://scholar.google.com/scholar?q=Lin%2C+D%2E+%281995%29+A+dependency%2Dbased+method+for+evaluating+broad%2Dcoverage+parsers%2E+In+Proceedings+of+the+14th+International+Joint+Conference+on+Artificial+Intelligence%2C+1420%E2%80%931425%2E+Montreal%2C+Canada%2E)
15. Lin, D. (1996) Dependency-based parser evaluation: a study with a software manual corpus. In R. Sutcliffe, H-D. Koch & A. McElligott (Eds.), Industrial Parsing of Software Manuals, 13–24. Amsterdam, The Netherlands: Rodopi. [find](https://scholar.google.com/scholar?q=Lin%2C+D%2E+%281996%29+Dependency%2Dbased+parser+evaluation%3A+a+study+with+a+software+manual+corpus%2E+In+R%2E+Sutcliffe%2C+H%2DD%2E+Koch+%26+A%2E+McElligott+%28Eds%2E%29%2C+Industrial+Parsing+of+Software+Manuals%2C+13%E2%80%9324%2E+Amsterdam%2C+T)
16. Marcus, M., Santorini, B. & Marcinkiewicz (1993) Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2), 313–330. [find](https://scholar.google.com/scholar?q=Marcus%2C+M%2E%2C+Santorini%2C+B%2E+%26+Marcinkiewicz+%281993%29+Building+a+large+annotated+corpus+of+English%3A+The+Penn+Treebank%2E+Computational+Linguistics%2C+19%282%29%2C+313%E2%80%93330%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
17. Roland, D. & Jurafsky, D. (1998) How verb subcategorization frequencies are affected by corpus choice. In Proceedings of the 17th International Conference on Computational Linguistics (COLING-ACL’98), 1122–1128. Montreal, Canada. [find](https://scholar.google.com/scholar?q=Roland%2C+D%2E+%26+Jurafsky%2C+D%2E+%281998%29+How+verb+subcategorization+frequencies+are+affected+by+corpus+choice%2E+In+Proceedings+of+the+17th+International+Conference+on+Computational+Linguistics+%28COLING%2DACL%E2%80%9998%29%2C)
18. Rubio, A. (Ed.) (1998) International Conference on Language Resources and Evaluation. Granada, Spain. [find](https://scholar.google.com/scholar?q=Rubio%2C+A%2E+%28Ed%2E%29+%281998%29+International+Conference+on+Language+Resources+and+Evaluation%2E+Granada%2C+Spain%2E)
19. Sampson, G. (1995) English for the Computer. Oxford, UK: Oxford University Press. [find](https://scholar.google.com/scholar?q=Sampson%2C+G%2E+%281995%29+English+for+the+Computer%2E+Oxford%2C+UK%3A+Oxford+University+Press%2E)
20. Sanfilippo, A., Barnett, R., Calzolari, N., Flores, S., Hellwig, P., Leech, P., Melero, M., Montemagni, S., Odijk, J., Pirrelli, V., Teufel, S., Villegas M. & Zaysser, L. (1996) Subcategorization Standards. Report of the EAGLES Lexicon/Syntax Interest Group. Available through eagles@ilc.pi.cnr.it. [find](https://scholar.google.com/scholar?q=Sanfilippo%2C+A%2E%2C+Barnett%2C+R%2E%2C+Calzolari%2C+N%2E%2C+Flores%2C+S%2E%2C+Hellwig%2C+P%2E%2C+Leech%2C+P%2E%2C+Melero%2C+M%2E%2C+Montemagni%2C+S%2E%2C+Odijk%2C+J%2E%2C+Pirrelli%2C+V%2E%2C+Teufel%2C+S%2E%2C+Villegas+M%2E+%26+Zaysser%2C+L%2E+%281996%29+Subcategorization+Stan)
21. Sekine, S. (1997) The domain dependence of parsing. In Proceedings of the 5th ACL Conference on Applied Natural Language Processing, 96–102. Washington, DC. [find](https://scholar.google.com/scholar?q=Sekine%2C+S%2E+%281997%29+The+domain+dependence+of+parsing%2E+In+Proceedings+of+the+5th+ACL+Conference+on+Applied+Natural+Language+Processing%2C+96%E2%80%93102%2E+Washington%2C+DC%2E)
22. Srinivas, B., Doran, C., Hockey B. & Joshi A. (1996) An approach to robust partial parsing and evaluation metrics. In Proceedings of the ESSLLI’96 Workshop on Robust Parsing. Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Srinivas%2C+B%2E%2C+Doran%2C+C%2E%2C+Hockey+B%2E+%26+Joshi+A%2E+%281996%29+An+approach+to+robust+partial+parsing+and+evaluation+metrics%2E+In+Proceedings+of+the+ESSLLI%E2%80%9996+Workshop+on+Robust+Parsing%2E+Prague%2C+Czech+Republic%2E)
23. Srinivas, B., Doran, C. & Kulick, S. (1995) Heuristics and parse ranking. In Proceedings of the 4th ACL/SIGPARSE International Workshop on Parsing Technologies. Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Srinivas%2C+B%2E%2C+Doran%2C+C%2E+%26+Kulick%2C+S%2E+%281995%29+Heuristics+and+parse+ranking%2E+In+Proceedings+of+the+4th+ACL%2FSIGPARSE+International+Workshop+on+Parsing+Technologies%2E+Prague%2C+Czech+Republic%2E)
24. van Rijsbergen, C. (1979) Information Retrieval. Butterworth, London. [find](https://scholar.google.com/scholar?q=van+Rijsbergen%2C+C%2E+%281979%29+Information+Retrieval%2E+Butterworth%2C+London%2E)