[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/lingbulk-2008-foster-parser-evaluation-bnc-evaluating-4/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics

# Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics

**Authors:** [Jennifer Foster](https://md.mova.elamur.ai/free/en/docs/papers/authors/jennifer-foster/) and [Josef van Genabith](https://md.mova.elamur.ai/free/en/docs/papers/authors/josef-van-genabith/) · **Year:** 2008 · **Venue:** Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08) **Link:** [aclanthology.org/L08-1115](https://aclanthology.org/L08-1115/) · **Licence:** copyrighted (ELRA, all rights reserved; listed as [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) by the ACL Anthology) — summary only

[**11** citations](https://openalex.org/works?filter=cites:W1590397582) openalex · 2026-10-19[**7** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/lingbulk-2008-foster-parser-evaluation-bnc-evaluating-4/) of the 140673 papers whose bibliographies we read[Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)](https://aclanthology.org/volumes/L08-1/)[licence](https://creativecommons.org/licenses/by-nc/4.0/)

- [Publisher's page](https://aclanthology.org/L08-1115/)
- [PDF](https://aclanthology.org/L08-1115.pdf)
- [DOI 10.63317/2kobxa2925pm](https://doi.org/10.63317/2kobxa2925pm)
- [OpenAlex](https://openalex.org/W1590397582)

**Authors (2)**

1. [Jennifer Foster](https://openalex.org/A5066497689) [ORCID](https://orcid.org/0000-0002-7789-4853) · [Semantic Scholar](https://www.semanticscholar.org/author/144116340) · h 23 · 2298 citations · 125 works
2. [Josef van Genabith](https://openalex.org/A5049194403) [ORCID](https://orcid.org/0000-0003-1322-7944) · [Semantic Scholar](https://www.semanticscholar.org/author/7519068) · h 38 · 5517 citations · 366 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

The paper builds a new test set of 1,000 hand-corrected British National Corpus parse trees, chosen so that each sentence has a verb that is unseen in the WSJ training sections. Four versions of the Charniak parser are scored on it with three metrics: Parseval, Leaf-Ancestor and a dependency score (LAS, UAS, label accuracy). Reranking and self-training on BNC parses help, the BNC self-trained parser best of all: Parseval F rises from 82.5 to 85.4, LAS from 82.5 to 84.2, UAS from 85.8 to 87.4, and Leaf-Ancestor from 0.8807 to 0.8900. Per-label F-scores show that ADV, DEP, IOBJ, PRN and PRT are weak for all four parsers, and that coordination is weaker for the newspaper self-trained parser.

## Ideas

- I1. A difficult-by-design test subset: select EWT sentences that contain a verb form not seen in `en` training, using `en lexicon` for the check, and measure `en parse` LAS on this subset against the full EWT test. The paper's selection rule is the model here. Test only, since the subset is used for measurement.
- I2. Per-label F table in `en ud-eval`: report F-score for each dependency label, as the paper's breakdown does. Measure first whether the same labels are weak in Mova on EWT test before any model change.
- I3. Path-to-root lineage score: a Leaf-Ancestor analogue for dependency trees. For each word, compare the gold and predicted path to the root by edit distance, and report the mean beside LAS in `en parse-eval`. Measure on EWT test and check that it ranks models the same way as LAS.
- I4. Self-training on target-genre parses: add silver `en` parses of raw open text from the target genre to the perceptron training data, and test on a held-out sample of that genre. The gain must beat a control that adds the same number of sentences from the original genre.

## For Mova

The code is `en/src/parse.rs` and the `en parse-eval` and `en ud-eval` commands. The chapters are [21. The simple clause](https://md.mova.elamur.ai/free/en/docs/languages/en/21-clause-structure/) and `37-register-and-variation.md`. The numbers are Charniak-parser figures on a BNC set, so they are not Mova numbers; Mova's own reference is LAS 79.8 on EWT test. The BNC is non-commercial and the test set is test only, never training data.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.63317/2kobxa2925pm](https://doi.org/10.63317/2kobxa2925pm)

```
@inproceedings{foster2008parser,
  title = {{Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics}},
  author = {Foster, Jennifer and Genabith, Josef van},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/2kobxa2925pm},
  url = {https://doi.org/10.63317/2kobxa2925pm}
}
```

- [← previous Tree-distance and some other variants of evalb](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2008-emms-tree-distance-some-other-variants/)
- [next → Towards Semi-Automated Annotation for Prepositional Phrase Attachment](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2010-rosenthal-semi-automated-annotation-prepositional-phrase/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ann Bies, Mark Ferguson, Karen Katz, and Robert MacIntyre. 1995. Bracketing guidelines for Treebank II style, Penn Treebank project. Technical Report Tech Report MS-CIS-95-06, University of Pennsylvania, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Ann+Bies%2C+Mark+Ferguson%2C+Karen+Katz%2C+and+Robert+MacIntyre%2E+1995%2E+Bracketing+guidelines+for+Treebank+II+style%2C+Penn+Treebank+project%2E+Technical+Report+Tech+Report+MS%2DCIS%2D95%2D06%2C+University+of+Pennsylvan)
2. Ezra Black, Steve Abney, Dan Flickinger, Claudia Gdaniec, Robert Grishman, Philip Harrison, Donald Hindle, Robert Ingria, Fred Jelinek, Judith Klavans, Mark Liberman, Mitchell Marcus, Salim Roukos, Beatrice Santorini, and Tomek Strzalkowski. 1991. A procedure for quantitatively comparing the syntactic coverage of english grammars. In Proceedings of the 1991 DARPA Speech and Natural Language Workshop, pages 306– 311. [find](https://scholar.google.com/scholar?q=Ezra+Black%2C+Steve+Abney%2C+Dan+Flickinger%2C+Claudia+Gdaniec%2C+Robert+Grishman%2C+Philip+Harrison%2C+Donald+Hindle%2C+Robert+Ingria%2C+Fred+Jelinek%2C+Judith+Klavans%2C+Mark+Liberman%2C+Mitchell+Marcus%2C+Salim+Roukos%2C+Be)
3. Lou Burnard. 2000. User reference guide for the British National Corpus. Technical report, Oxford University Computing Services. [find](https://scholar.google.com/scholar?q=Lou+Burnard%2E+2000%2E+User+reference+guide+for+the+British+National+Corpus%2E+Technical+report%2C+Oxford+University+Computing+Services%2E)
4. John Carroll, Anette Frank, Dekang Lin, Detlef Prescher, and Hans Uszkoreit, editors. 2002. Proceedings of the “Beyond Parseval - Towards Improved Evaluation Measures for Parsing Systems” Workshop at the 3rd International Conference on Linguistic Resources and Evaluation (LREC-02), Las Palmas, Gran Canaria. [find](https://scholar.google.com/scholar?q=John+Carroll%2C+Anette+Frank%2C+Dekang+Lin%2C+Detlef+Prescher%2C+and+Hans+Uszkoreit%2C+editors%2E+2002%2E+Proceedings+of+the+%E2%80%9CBeyond+Parseval+%2D+Towards+Improved+Evaluation+Measures+for+Parsing+Systems%E2%80%9D+Workshop+at+)
5. Eugene Charniak and Mark Johnson. 2005. Course-to-fine n-best-parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the ACL (ACL-05), pages 173–180, Ann Arbor, Michigan, June. [find](https://scholar.google.com/scholar?q=Eugene+Charniak+and+Mark+Johnson%2E+2005%2E+Course%2Dto%2Dfine+n%2Dbest%2Dparsing+and+maxent+discriminative+reranking%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+ACL+%28ACL%2D05%29%2C+pages+173%E2%80%93180%2C+Ann+Arbor%2C+Mich)
6. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the Annual Meeting of the North American Association for Computational Linguistics (NAACL-00), pages 132–139, Seattle, Washington. [find](https://scholar.google.com/scholar?q=Eugene+Charniak%2E+2000%2E+A+maximum%2Dentropy%2Dinspired+parser%2E+In+Proceedings+of+the+Annual+Meeting+of+the+North+American+Association+for+Computational+Linguistics+%28NAACL%2D00%29%2C+pages+132%E2%80%93139%2C+Seattle%2C+Washi)
7. Grzegorz Chrupała, Nicolas Stroppa, Josef van Genabith, and Georgiana Dinu. 2007. Better training for function labeling. In Proceedings of the Recent Advances in Natural Language Processing Conference (RANLP-07), pages 133–138, Borovets, Bulgaria. [find](https://scholar.google.com/scholar?q=Grzegorz+Chrupa%C5%82a%2C+Nicolas+Stroppa%2C+Josef+van+Genabith%2C+and+Georgiana+Dinu%2E+2007%2E+Better+training+for+function+labeling%2E+In+Proceedings+of+the+Recent+Advances+in+Natural+Language+Processing+Conference)
8. Jennifer Foster, Joachim Wagner, Djamé Seddah, and Josef van Genabith. 2007. Adapting WSJ-trained parsers to the British National Corpus using in-domain self-training. In Proceedings of the Tenth International Workshop on Parsing Technologies (IWPT-07), pages 33–35, Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Jennifer+Foster%2C+Joachim+Wagner%2C+Djame%CC%81+Seddah%2C+and+Josef+van+Genabith%2E+2007%2E+Adapting+WSJ%2Dtrained+parsers+to+the+British+National+Corpus+using+in%2Ddomain+self%2Dtraining%2E+In+Proceedings+of+the+Tenth+Int) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2007-foster-adapting-wsj-trained-parsers-british/)
9. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to-dependency conversion for English. In Joakim Nivre, Heiki-Jaan Kaalep, Kadri Muischnek, and Mare Koit, editors, Proceedings of NODALIDA 2007, pages 105–112, Tartu, Estonia. [find](https://scholar.google.com/scholar?q=Richard+Johansson+and+Pierre+Nugues%2E+2007%2E+Extended+constituent%2Dto%2Ddependency+conversion+for+English%2E+In+Joakim+Nivre%2C+Heiki%2DJaan+Kaalep%2C+Kadri+Muischnek%2C+and+Mare+Koit%2C+editors%2C+Proceedings+of+NODALI) [our card](https://md.mova.elamur.ai/free/en/docs/papers/map-2007-johansson-extended-constituent-dependency-conversion-english/)
10. Sandra Kübler and Heike Telljohann. 2002. Towards a dependency-oriented evaluation for partial parsing. In Carroll et al. (Carroll et al., 2002), pages 9–16. [find](https://scholar.google.com/scholar?q=Sandra+Ku%CC%88bler+and+Heike+Telljohann%2E+2002%2E+Towards+a+dependency%2Doriented+evaluation+for+partial+parsing%2E+In+Carroll+et+al%2E+%28Carroll+et+al%2E%2C+2002%29%2C+pages+9%E2%80%9316%2E)
11. Dekang Lin. 1998. Dependency-based evaluation of MINIPAR. In John Carroll, editor, Proceedings of The Evaluation of Parsing Systems Workshop at the 3rd International Conference on Linguistic Resources and Evaluation (LREC), Cognitive Science Research Papers 489, pages 48–56. University of Sussex, Brighton, England. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2E+1998%2E+Dependency%2Dbased+evaluation+of+MINIPAR%2E+In+John+Carroll%2C+editor%2C+Proceedings+of+The+Evaluation+of+Parsing+Systems+Workshop+at+the+3rd+International+Conference+on+Linguistic+Resources)
12. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: Annotating Predicate Argument Structure. In Proceedings of the ARPA Workshop on Human Language Technology, pages 110–115, Princeton, NJ. [find](https://scholar.google.com/scholar?q=Mitchell+Marcus%2C+Grace+Kim%2C+Mary+Ann+Marcinkiewicz%2C+Robert+MacIntyre%2C+Ann+Bies%2C+Mark+Ferguson%2C+Karen+Katz%2C+and+Britta+Schasberger%2E+1994%2E+The+Penn+Treebank%3A+Annotating+Predicate+Argument+Structure%2E+In+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1994-marcus-penn-treebank-annotating-predicate-argument/)
13. David McClosky, Eugene Charniak, and Mark Johnson. 2006a. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference and North American chapter of the ACL annual meeting (HLT-NAACL-06), pages 152–159, New York, June. [find](https://scholar.google.com/scholar?q=David+McClosky%2C+Eugene+Charniak%2C+and+Mark+Johnson%2E+2006a%2E+Effective+self%2Dtraining+for+parsing%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+and+North+American+chapter+of+the+ACL+annual+m) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-mcclosky-effective-self-training-parsing/)
14. David McClosky, Eugene Charniak, and Mark Johnson. 2006b. Reranking and self-training for parser adaptation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL (COLING-ACL-06), pages 337–344, Sydney, Australia, July. [find](https://scholar.google.com/scholar?q=David+McClosky%2C+Eugene+Charniak%2C+and+Mark+Johnson%2E+2006b%2E+Reranking+and+self%2Dtraining+for+parser+adaptation%2E+In+Proceedings+of+the+21st+International+Conference+on+Computational+Linguistics+and+44th+A) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-mcclosky-reranking-self-training-parser-adaptation/)
15. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan Mac Donald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Johan+Hall%2C+Sandra+Ku%CC%88bler%2C+Ryan+Mac+Donald%2C+Jens+Nilsson%2C+Sebastian+Riedel%2C+and+Deniz+Yuret%2E+2007%2E)
16. The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915–932, Prague, Czech Republic, June. [find](https://scholar.google.com/scholar?q=The+CoNLL+2007+shared+task+on+dependency+parsing%2E+In+Proceedings+of+the+CoNLL+Shared+Task+Session+of+EMNLP%2DCoNLL+2007%2C+pages+915%E2%80%93932%2C+Prague%2C+Czech+Republic%2C+June%2E)
17. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 433–440, Sydney, Australia. [find](https://scholar.google.com/scholar?q=Slav+Petrov%2C+Leon+Barrett%2C+Romain+Thibaux%2C+and+Dan+Klein%2E+2006%2E+Learning+accurate%2C+compact%2C+and+interpretable+tree+annotation%2E+In+Proceedings+of+the+21st+International+Conference+on+Computational+Ling) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-petrov-learning-accurate-compact-interpretable-tree/)
18. Ines Rehbein and Josef van Genabith. 2007. Treebank annotation schemes and parser evaluation for German. In Proceedings of the Joint EMNLP-CoNLL 2007, pages 630–639, Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Ines+Rehbein+and+Josef+van+Genabith%2E+2007%2E+Treebank+annotation+schemes+and+parser+evaluation+for+German%2E+In+Proceedings+of+the+Joint+EMNLP%2DCoNLL+2007%2C+pages+630%E2%80%93639%2C+Prague%2C+Czech+Republic%2E)
19. Geoffrey Sampson and Anna Babarczy. 2002. A test of the leaf-ancestor metric for parse accuracy. In Carroll et al. (Carroll et al., 2002). [find](https://scholar.google.com/scholar?q=Geoffrey+Sampson+and+Anna+Babarczy%2E+2002%2E+A+test+of+the+leaf%2Dancestor+metric+for+parse+accuracy%2E+In+Carroll+et+al%2E+%28Carroll+et+al%2E%2C+2002%29%2E)
20. Joachim Wagner, Djamé Seddah, Jennifer Foster, and Josef van Genabith. 2007. C-structures and f-structures for the British National Corpus. In Proceedings of the 12th International Workshop on Lexical Functional Grammar, Stanford, CA. [find](https://scholar.google.com/scholar?q=Joachim+Wagner%2C+Djame%CC%81+Seddah%2C+Jennifer+Foster%2C+and+Josef+van+Genabith%2E+2007%2E+C%2Dstructures+and+f%2Dstructures+for+the+British+National+Corpus%2E+In+Proceedings+of+the+12th+International+Workshop+on+Lexica)
21. Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proceedings of the 8th International Workshop on Parsing Technologies (IWPT-03), pages 195–206, Nancy, France. [find](https://scholar.google.com/scholar?q=Hiroyasu+Yamada+and+Yuji+Matsumoto%2E+2003%2E+Statistical+dependency+analysis+with+support+vector+machines%2E+In+Proceedings+of+the+8th+International+Workshop+on+Parsing+Technologies+%28IWPT%2D03%29%2C+pages+195%E2%80%9320) [our card](https://md.mova.elamur.ai/free/en/docs/papers/parsers-2003-yamada-statistical-dependency-analysis-support-vector/)