[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/missing4-2007-mcdonald-characterizing-errors-data-driven-dependency/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Characterizing the Errors of Data-Driven Dependency Parsing Models

# Characterizing the Errors of Data-Driven Dependency Parsing Models

**Authors:** [Ryan McDonald](https://md.mova.elamur.ai/free/en/docs/papers/authors/ryan-mcdonald/) and [Joakim Nivre](https://md.mova.elamur.ai/free/en/docs/papers/authors/joakim-nivre/) · **Year:** 2007 · **Venue:** Proceedings of the 2007 Joint Conference on EMNLP and CoNLL (EMNLP-CoNLL) **Link:** [aclanthology.org/D07-1013](https://aclanthology.org/D07-1013/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**141** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/missing4-2007-mcdonald-characterizing-errors-data-driven-dependency/) of the 140673 papers whose bibliographies we read[Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)](https://aclanthology.org/volumes/D07-1/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/D07-1013/)
- [PDF](https://aclanthology.org/D07-1013.pdf)

**Authors (2)**

1. Ryan McDonald
2. Joakim Nivre

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

The paper compares the two top systems of the CoNLL-X shared task. MSTParser is a global, exhaustive graph-based model with maximum spanning tree decoding and arc-pair features; MaltParser is a local, greedy transition-based model with an SVM classifier. Both reach almost the same labeled accuracy over 13 languages (about 80.8 and 80.8 LAS when the output files are pooled), but they make different kinds of errors. The authors break the errors down by arc length, distance to the root, number of siblings, degree of non-projectivity, part of speech and dependency type. MSTParser is more precise for arcs far from the root and for long arcs; MaltParser is better on short arcs and on nouns and pronouns. A simple oracle over the two outputs reaches 84.5 to 86.9% against about 81% for each system.

## Ideas

- I1. Error profile by distance to the root in `en parse-eval` and in `gparse` (Eisner decoding, `gparse/src/eisner.rs`): precision and recall of arcs at distance 1, 2, 3–4 and 5 or more, on EWT test; the question is whether the arc-standard parser fragments the tree near the root as the paper suggests for greedy parsers.
- I2. Root-modifier check: count predicted root arcs that are not verbs or auxiliaries in `en explain` output over EWT test; the paper reads low root-modifier precision with stable recall as a sign of fragmented parses.
- I3. Oracle diagnostic (test only, not shipped): take the `en` and `gparse` outputs on EWT test and measure the oracle LAS and the share of sentences where one of them is fully correct; if the gap is large, a combiner through `tools/parsecmp` is worth a later decision.
- I4. Non-projectivity split: report LAS separately for sentences with a gold arc of degree 1 or more; the arc-standard parser in `en/src/parse.rs` cannot build those arcs, so the number is its ceiling on EWT and GUM test (GUM is test only).

## For Mova

The relevant chapters are [20. Syntax: the Universal Dependencies frame](https://md.mova.elamur.ai/free/en/docs/languages/en/20-syntax-overview/) (the main clause and root) and `21-clause-structure.md`. The code is `en/src/parse.rs` (arc-standard, beam 8), `gparse/src/eisner.rs`, and the `en parse-eval` and `tools/parsecmp` tools. Measure first: EWT test LAS by distance to the root for both parsers, against the current baseline LAS 80.24. The paper's lesson is that an error analysis by structure says more than one accuracy number.

Topics: [Dependency parsing](https://md.mova.elamur.ai/free/en/docs/papers/topics/dependency-parsing/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{mcdonald2007characterizing,
  title = {{Characterizing the Errors of Data-Driven Dependency Parsing Models}},
  author = {McDonald, Ryan and Nivre, Joakim},
  booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  year = {2007},
  url = {https://aclanthology.org/D07-1013/}
}
```

- [← previous Deep Biaffine Attention for Neural Dependency Parsing](https://md.mova.elamur.ai/free/en/docs/papers/missing-2016-dozat-deep-biaffine-attention-neural-dependency/)
- [next → Analyzing the Impact of Spelling Errors on POS-Tagging and Chunking in Learner English](https://md.mova.elamur.ai/free/en/docs/papers/mizumoto-2017-analyzing-impact-spelling-errors-pos/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Böhmová, J. Hajič, E. Hajičová, and B. Hladká. 2003. The PDT: A 3-level annotation scenario. In A. Abeillé, editor, Treebanks: Building and Using Parsed Corpora, chapter 7. Kluwer Academic Publishers. [find](https://scholar.google.com/scholar?q=A%2E+Bo%CC%88hmova%CC%81%2C+J%2E+Hajic%CC%8C%2C+E%2E+Hajic%CC%8Cova%CC%81%2C+and+B%2E+Hladka%CC%81%2E+2003%2E+The+PDT%3A+A+3%2Dlevel+annotation+scenario%2E+In+A%2E+Abeille%CC%81%2C+editor%2C+Treebanks%3A+Building+and+Using+Parsed+Corpora%2C+chapter+7%2E+Kluwer+Academic+P)
2. S. Buchholz and E. Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proc. CoNLL. [find](https://scholar.google.com/scholar?q=S%2E+Buchholz+and+E%2E+Marsi%2E+2006%2E+CoNLL%2DX+shared+task+on+multilingual+dependency+parsing%2E+In+Proc%2E+CoNLL%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/parsers-2006-buchholz-conll-x-shared-task-multilingual/)
3. Hal Daumé III, John Langford, and Daniel Marcu. 2006. Search-based structured prediction. In Submission. [find](https://scholar.google.com/scholar?q=Hal+Daume%CC%81+III%2C+John+Langford%2C+and+Daniel+Marcu%2E+2006%2E+Search%2Dbased+structured+prediction%2E+In+Submission%2E)
4. Y. Ding and M. Palmer. 2004. Synchronous dependency insertion grammars: A grammar formalism for syntax based statistical MT. In Workshop on Recent Advances in Dependency Grammars (COLING). [find](https://scholar.google.com/scholar?q=Y%2E+Ding+and+M%2E+Palmer%2E+2004%2E+Synchronous+dependency+insertion+grammars%3A+A+grammar+formalism+for+syntax+based+statistical+MT%2E+In+Workshop+on+Recent+Advances+in+Dependency+Grammars+%28COLING%29%2E)
5. R. Hudson. 1984. Word Grammar. Blackwell. [find](https://scholar.google.com/scholar?q=R%2E+Hudson%2E+1984%2E+Word+Grammar%2E+Blackwell%2E)
6. H. Maruyama. 1990. Structural disambiguation with constraint propagation. In Proc. ACL. [find](https://scholar.google.com/scholar?q=H%2E+Maruyama%2E+1990%2E+Structural+disambiguation+with+constraint+propagation%2E+In+Proc%2E+ACL%2E)
7. R. McDonald, K. Crammer, and F. Pereira. 2005a. Online large-margin training of dependency parsers. In Proc. ACL. [find](https://scholar.google.com/scholar?q=R%2E+McDonald%2C+K%2E+Crammer%2C+and+F%2E+Pereira%2E+2005a%2E+Online+large%2Dmargin+training+of+dependency+parsers%2E+In+Proc%2E+ACL%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/narrative-2005-mcdonald-online-large-margin-training-dependency/)
8. R. McDonald, F. Pereira, K. Ribarov, and J. Hajič. 2005b. Non-projective dependency parsing using spanning tree algorithms. In Proc. HLT/EMNLP. [find](https://scholar.google.com/scholar?q=R%2E+McDonald%2C+F%2E+Pereira%2C+K%2E+Ribarov%2C+and+J%2E+Hajic%CC%8C%2E+2005b%2E+Non%2Dprojective+dependency+parsing+using+spanning+tree+algorithms%2E+In+Proc%2E+HLT%2FEMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/narrative-2005-mcdonald-non-projective-dependency-parsing-spanning/)
9. R. McDonald, K. Lerman, and F. Pereira. 2006. Multilingual dependency analysis with a two-stage discriminative parser. In Proc. CoNLL. [find](https://scholar.google.com/scholar?q=R%2E+McDonald%2C+K%2E+Lerman%2C+and+F%2E+Pereira%2E+2006%2E+Multilingual+dependency+analysis+with+a+two%2Dstage+discriminative+parser%2E+In+Proc%2E+CoNLL%2E)
10. I.A. Mel’čuk. 1988. Dependency Syntax: Theory and Practice. State University of New York Press. [find](https://scholar.google.com/scholar?q=I%2EA%2E+Mel%E2%80%99c%CC%8Cuk%2E+1988%2E+Dependency+Syntax%3A+Theory+and+Practice%2E+State+University+of+New+York+Press%2E)
11. J. Nivre and J. Nilsson. 2005. Pseudo-projective dependency parsing. In Proc. ACL. [find](https://scholar.google.com/scholar?q=J%2E+Nivre+and+J%2E+Nilsson%2E+2005%2E+Pseudo%2Dprojective+dependency+parsing%2E+In+Proc%2E+ACL%2E)
12. J. Nivre, J. Hall, J. Nilsson, G. Eryigit, and S. Marinov. 2006. Labeled pseudo-projective dependency parsing with support vector machines. In Proc. CoNLL. [find](https://scholar.google.com/scholar?q=J%2E+Nivre%2C+J%2E+Hall%2C+J%2E+Nilsson%2C+G%2E+Eryigit%2C+and+S%2E+Marinov%2E+2006%2E+Labeled+pseudo%2Dprojective+dependency+parsing+with+support+vector+machines%2E+In+Proc%2E+CoNLL%2E)
13. J. Nivre. 2003. An efficient algorithm for projective dependency parsing. In Proc. IWPT. [find](https://scholar.google.com/scholar?q=J%2E+Nivre%2E+2003%2E+An+efficient+algorithm+for+projective+dependency+parsing%2E+In+Proc%2E+IWPT%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/parsers-2003-nivre-efficient-algorithm-projective-dependency-parsing/)
14. J. Nivre. 2006. Constraints on non-projective dependency parsing. In Proc. EACL. [find](https://scholar.google.com/scholar?q=J%2E+Nivre%2E+2006%2E+Constraints+on+non%2Dprojective+dependency+parsing%2E+In+Proc%2E+EACL%2E)
15. K. Sagae and A. Lavie. 2006. Parser combination by reparsing. In Proc. HLT/NAACL. [find](https://scholar.google.com/scholar?q=K%2E+Sagae+and+A%2E+Lavie%2E+2006%2E+Parser+combination+by+reparsing%2E+In+Proc%2E+HLT%2FNAACL%2E)
16. P. Sgall, E. Hajičová, and J. Panevová. 1986. The Meaning of the Sentence in Its Pragmatic Aspects. Reidel. [find](https://scholar.google.com/scholar?q=P%2E+Sgall%2C+E%2E+Hajic%CC%8Cova%CC%81%2C+and+J%2E+Panevova%CC%81%2E+1986%2E+The+Meaning+of+the+Sentence+in+Its+Pragmatic+Aspects%2E+Reidel%2E)
17. R. Snow, D. Jurafsky, and A. Y. Ng. 2004. Learning syntactic patterns for automatic hypernym discovery. In Proc. NIPS. [find](https://scholar.google.com/scholar?q=R%2E+Snow%2C+D%2E+Jurafsky%2C+and+A%2E+Y%2E+Ng%2E+2004%2E+Learning+syntactic+patterns+for+automatic+hypernym+discovery%2E+In+Proc%2E+NIPS%2E)