[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/bank-2004-ringger-penn-treebank-evaluate-non-treebank/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Using the Penn Treebank to Evaluate Non-Treebank Parsers

# Using the Penn Treebank to Evaluate Non-Treebank Parsers

**Authors:** Eric K. Ringger, Robert C. Moore, Lucy Vanderwende, Hisami Suzuki and [Eugene Charniak](https://md.mova.elamur.ai/free/en/docs/papers/authors/eugene-charniak/) · **Year:** 2004 · **Venue:** Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004) **Link:** [aclanthology.org/L04-1115](https://aclanthology.org/L04-1115/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology; ELRA: all rights reserved) — summary only

[**3** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/bank-2004-ringger-penn-treebank-evaluate-non-treebank/) of the 140673 papers whose bibliographies we read[Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)](https://aclanthology.org/volumes/L04-1/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/L04-1115/)
- [PDF](https://aclanthology.org/L04-1115.pdf)

**Authors (5)**

1. Eric K. Ringger
2. Robert C. Moore
3. Eugene Charniak
4. Lucy Vanderwende
5. Hisami Suzuki

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

The question is how to compare a parser that was never trained on the Penn Treebank (NLPWin, built at Microsoft Research) with Treebank-trained parsers such as Charniak's, when the two attach modifiers in a different order and use different brackets. The authors score both on unlabelled precision and recall of maximal projections of heads (MPH), a bracket set that focuses on "maximal projections of heads" (p. 1). Head labels for the gold trees come from Charniak's head rules, plus a small set of reassignment rules; single-word brackets, the sentence bracket and punctuation are dropped, and unary nodes are flattened. On WSJ section 23, NLPWin reaches P 74.32 and R 74.25, and Charniak's parser P 86.18 and R 85.72. On the restricted Brown test BR1 the gap is smaller (NLPWin P 69.54, R 70.56; Charniak P 77.71, R 77.73). The authors argue that raw brackets would reward high PP attachment, and that a dependency tree with ordered dependents is isomorphic to an MPH tree, so the same metric could score dependency parsers.

## Ideas

- I1. `tools/parsecmp`: add an MPH score next to the labelled scores. In a UD tree, each word and all its dependents form one span, which is the same bracket the paper's MPH trees give, so the metric needs no head-rule set. Score the `en` output and the constituency runs already listed in `parsecmp` (`stanza-con`, `benepar`, `supar-con`) against the same Haiku reference, and check whether the MPH ranking of systems agrees with the LAS ranking. Non-projective trees give spans with gaps; report their count separately.
- I2. Negative control for that metric: in a copy of the reference, move every PP from verb attachment to noun attachment by a fixed rule. MPH precision and recall must fall on the same sentences. A metric that stays flat under this change is not measuring attachment.
- I3. Bias check, as the paper warns about high attachment: on EWT test, count how often `en` attaches a PP to the verb and to the noun, and how often gold does, and report the MPH score on the PP subset alone. This shows whether the SLM has the bias the paper describes, and in which direction.

## For Mova

The chapters are [22. Phrases: noun, adjective, adverb and prepositional phrases](https://md.mova.elamur.ai/free/en/docs/languages/en/22-phrases/) (NP structure, PP attachment) and [21. The simple clause](https://md.mova.elamur.ai/free/en/docs/languages/en/21-clause-structure/). The code is `tools/parsecmp` (the `SYSTEMS` and `CONSTITUENCY` lists) and `en parse-eval`. The Penn Treebank is LDC data under non-commercial terms: use it, if at all, for tests only, never in a release. The EWT measurements are CC BY-SA, so they are measurements, not material carried into a build.

Topics: [Dependency parsing](https://md.mova.elamur.ai/free/en/docs/papers/topics/dependency-parsing/) · [Treebanks](https://md.mova.elamur.ai/free/en/docs/papers/topics/treebanks/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{ringger2004using,
  title = {{Using the Penn Treebank to Evaluate Non-Treebank Parsers}},
  author = {Ringger, Eric K. and Moore, Robert C. and Charniak, Eugene and Vanderwende, Lucy and Suzuki, Hisami},
  booktitle = {Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)},
  year = {2004},
  url = {https://aclanthology.org/L04-1115/}
}
```

- [← previous A Dependency Treebank for English](https://md.mova.elamur.ai/free/en/docs/papers/bank-2002-rambow-dependency-treebank-english/)
- [next → Fully Parsing the Penn Treebank](https://md.mova.elamur.ai/free/en/docs/papers/bank-2006-gabbard-fully-parsing-penn-treebank/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bies, A., M. Ferguson, K. Katz, R. MacIntyre, V. Tredinnick, G. Kim, M.A. Marcinkiewicz, B. Schasberger. (1995) "Bracketing Guidelines for Treebank II Style Penn Treebank Project". TR, University of Pennsylvania. WSJ 23 BR 0 BR 1 8903 84.82 85.77 13131 74.25 74.32 15163 85.72 86.18 8489 71.55 70.49 9415 79.33 79.66 8217 70.56 69.54 9054 77.73 77.71 41.23 0.32 80.91 27.11 0.7 68.34 45.16 0.29 82.41 32.95 0.67 73 45.75 0.4 80.55 32.8 0.67 73.2 44.3 0.47 79.05 97.03 90.44 97.14 91.3 95.65 90.85 94.5 [find](https://scholar.google.com/scholar?q=Bies%2C+A%2E%2C+M%2E+Ferguson%2C+K%2E+Katz%2C+R%2E+MacIntyre%2C+V%2E+Tredinnick%2C+G%2E+Kim%2C+M%2EA%2E+Marcinkiewicz%2C+B%2E+Schasberger%2E+%281995%29+%22Bracketing+Guidelines+for+Treebank+II+Style+Penn+Treebank+Project%22%2E+TR%2C+University+of+P)
2. Black, E et al. (1991) “A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars”. In Proceedings of the February 1991 DARPA Speech and Natural Language Workshop. pp. 306-311. [find](https://scholar.google.com/scholar?q=Black%2C+E+et+al%2E+%281991%29+%E2%80%9CA+Procedure+for+Quantitatively+Comparing+the+Syntactic+Coverage+of+English+Grammars%E2%80%9D%2E+In+Proceedings+of+the+February+1991+DARPA+Speech+and+Natural+Language+Workshop%2E+pp%2E+306%2D31)
3. Charniak, E. (2000) “A Maximum-Entropy-Inspired Parser”. In Proceedings of NAACL-2000. pp. 132–139. [find](https://scholar.google.com/scholar?q=Charniak%2C+E%2E+%282000%29+%E2%80%9CA+Maximum%2DEntropy%2DInspired+Parser%E2%80%9D%2E+In+Proceedings+of+NAACL%2D2000%2E+pp%2E+132%E2%80%93139%2E)
4. Collins, M. (1999) “Head-Driven Statistical Models for Natural Language Parsing”. Ph.D. thesis, U. Pennsylvania. [find](https://scholar.google.com/scholar?q=Collins%2C+M%2E+%281999%29+%E2%80%9CHead%2DDriven+Statistical+Models+for+Natural+Language+Parsing%E2%80%9D%2E+Ph%2ED%2E+thesis%2C+U%2E+Pennsylvania%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/cl-2003-collins-head-driven-statistical-models-natural/)
5. Collins, M. & N. Duffy. (2002) “New Ranking Algorithms for Parsing and Tagging: Kernels over Discrete Structures, and the Voted Perceptron”. In Proceedings of ACL 2002. [find](https://scholar.google.com/scholar?q=Collins%2C+M%2E+%26+N%2E+Duffy%2E+%282002%29+%E2%80%9CNew+Ranking+Algorithms+for+Parsing+and+Tagging%3A+Kernels+over+Discrete+Structures%2C+and+the+Voted+Perceptron%E2%80%9D%2E+In+Proceedings+of+ACL+2002%2E)
6. Gaizauskas, R., M. Hepple, & C. Huyck. (1998a) “A Scheme for Comparative Evaluation of Diverse Parsing Systems”. In Proceedings of 1st International Conference on Language Resources and Evaluation (LREC'98), pp. 143-149. [find](https://scholar.google.com/scholar?q=Gaizauskas%2C+R%2E%2C+M%2E+Hepple%2C+%26+C%2E+Huyck%2E+%281998a%29+%E2%80%9CA+Scheme+for+Comparative+Evaluation+of+Diverse+Parsing+Systems%E2%80%9D%2E+In+Proceedings+of+1st+International+Conference+on+Language+Resources+and+Evaluation+%28LR)
7. Gaizauskas, R., M. Hepple, & C. Huyck. (1998b) “Modifying Existing Annotated Corpora for General Comparative Evaluation of Parsing”. In Proceedings of the LREC'98 Workshop on Evaluation of Parsing Systems. [find](https://scholar.google.com/scholar?q=Gaizauskas%2C+R%2E%2C+M%2E+Hepple%2C+%26+C%2E+Huyck%2E+%281998b%29+%E2%80%9CModifying+Existing+Annotated+Corpora+for+General+Comparative+Evaluation+of+Parsing%E2%80%9D%2E+In+Proceedings+of+the+LREC%2798+Workshop+on+Evaluation+of+Parsing+Sys)
8. Heidorn, G. (2000). “Intelligent writing assistance”. In Dale et al. Handbook of Natural Language Processing, Marcel Dekker. [find](https://scholar.google.com/scholar?q=Heidorn%2C+G%2E+%282000%29%2E+%E2%80%9CIntelligent+writing+assistance%E2%80%9D%2E+In+Dale+et+al%2E+Handbook+of+Natural+Language+Processing%2C+Marcel+Dekker%2E)
9. Kucera, H. & W.N. Francis. (1967) Computational Analysis of Present-Day American English. Providence: Brown University Press. [find](https://scholar.google.com/scholar?q=Kucera%2C+H%2E+%26+W%2EN%2E+Francis%2E+%281967%29+Computational+Analysis+of+Present%2DDay+American+English%2E+Providence%3A+Brown+University+Press%2E)
10. Lin, D. (1995) “A dependency-based method for evaluating broad-coverage parsers”. In Proceedings of IJCAI-95, pp.1420-1425. [find](https://scholar.google.com/scholar?q=Lin%2C+D%2E+%281995%29+%E2%80%9CA+dependency%2Dbased+method+for+evaluating+broad%2Dcoverage+parsers%E2%80%9D%2E+In+Proceedings+of+IJCAI%2D95%2C+pp%2E1420%2D1425%2E)
11. Lin, D. (1998) “Dependency-based Evaluation of MINIPAR”. In Workshop on the Evaluation of Parsing Systems. [find](https://scholar.google.com/scholar?q=Lin%2C+D%2E+%281998%29+%E2%80%9CDependency%2Dbased+Evaluation+of+MINIPAR%E2%80%9D%2E+In+Workshop+on+the+Evaluation+of+Parsing+Systems%2E)
12. Magerman, D. (1995) “Statistical decision-tree models for parsing”. In ACL 33, pp. 276–283. [find](https://scholar.google.com/scholar?q=Magerman%2C+D%2E+%281995%29+%E2%80%9CStatistical+decision%2Dtree+models+for+parsing%E2%80%9D%2E+In+ACL+33%2C+pp%2E+276%E2%80%93283%2E)
13. Marcus, M., B. Santorini, M.A. Marcinkiewicz. (1993) “Building a Large Annotated Corpus of English: The Penn Treebank”. Computational Linguistics 19 (2), 313-330. [find](https://scholar.google.com/scholar?q=Marcus%2C+M%2E%2C+B%2E+Santorini%2C+M%2EA%2E+Marcinkiewicz%2E+%281993%29+%E2%80%9CBuilding+a+Large+Annotated+Corpus+of+English%3A+The+Penn+Treebank%E2%80%9D%2E+Computational+Linguistics+19+%282%29%2C+313%2D330%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
14. Riezler, S., T. King, R. Kaplan, R. Crouch, J. Maxwell III, M. Johnson. (2002) “Parsing the Wall Street Journal using a Lexical-Functional Grammar and Discriminative Estimation Techniques”. In Proceedings of ACL 2002. pp. 271-278. [find](https://scholar.google.com/scholar?q=Riezler%2C+S%2E%2C+T%2E+King%2C+R%2E+Kaplan%2C+R%2E+Crouch%2C+J%2E+Maxwell+III%2C+M%2E+Johnson%2E+%282002%29+%E2%80%9CParsing+the+Wall+Street+Journal+using+a+Lexical%2DFunctional+Grammar+and+Discriminative+Estimation+Techniques%E2%80%9D%2E+In+Proceed)
15. Sekine, S. & M. Collins. (1997) EvalB: a bracket scoring program. URL: [http://www.cs.nyu.edu/cs/projects/proteus/evalb/](http://www.cs.nyu.edu/cs/projects/proteus/evalb/) [↗](http://www.cs.nyu.edu/cs/projects/proteus/evalb/)