[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/bank-2013-sogaard-empirical-study-differences-between-conversion/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / An Empirical Study of Differences between Conversion Schemes and Annotation Guidelines

# An Empirical Study of Differences between Conversion Schemes and Annotation Guidelines

**Authors:** [Anders Søgaard](https://md.mova.elamur.ai/free/en/docs/papers/authors/anders-sogaard/) · **Year:** 2013 · **Venue:** Proceedings of the Second International Conference on Dependency Linguistics (DepLing 2013) **Link:** [aclanthology.org/W13-3733](https://aclanthology.org/W13-3733/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**3** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/bank-2013-sogaard-empirical-study-differences-between-conversion/) of the 140673 papers whose bibliographies we read[Proceedings of the Second International Conference on Dependency Linguistics (DepLing 2013)](https://aclanthology.org/volumes/W13-37/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/W13-3733/)
- [PDF](https://aclanthology.org/W13-3733.pdf)

**Authors (1)**

1. Søgaard, Anders

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

Many dependency treebanks are converted from constituency treebanks, and different conversion schemes make different choices about heads and relation labels. The author measures how far such schemes differ with a generalized tree-edit distance, and compares them with learnability and with downstream performance. Experiments convert the Penn Treebank with four schemes (Yamada, CoNLL, Stanford and LTH). Parser accuracy on section 23 differs clearly for the Stanford scheme (LAS 81.36) against 87.52 to 88.99 for the others, but the parser's own bias makes many differences invisible to the parser. Learnability of a scheme does not predict its downstream score, but the choice of scheme does change the results of downstream applications.

## Ideas

- I1. Scheme noise floor for `en`: build two of the mova-dialect conversions of EWT that differ only in head choice, and report the LAS spread between them, so that every claimed `en` gain is compared with this spread rather than with zero.
- I2. Tree-edit agreement in `treeagree`: compute a generalized tree-edit distance between the silver trees from `annot` and the Stanza or spaCy trees per sentence, and use it as one more agreement gate beside R1 to R8 (measure how many sentences the new gate changes on EWT dev).
- I3. Do not use learnability as a selection rule: when choosing among conversions or among `gparse` checkpoints, select on downstream LAS (the `prag`/`world` output on ESLSpok) and not on training loss, as the paper finds loss does not track downstream results.

## For Mova

The conversions are in `en/src/convert.rs` and the dialect in [The Mova dialect: a UD metadialect, treebanks, and converters](https://md.mova.elamur.ai/free/en/docs/docs/dialect/), the silver checks in `tools/treeagree` and the witnesses in `tools/parsecmp`. The chapter that the ideas touch is [20. Syntax: the Universal Dependencies frame](https://md.mova.elamur.ai/free/en/docs/languages/en/20-syntax-overview/). Measure first: the LAS spread between two conversions of the same EWT trees.

Topics: [Treebanks](https://md.mova.elamur.ai/free/en/docs/papers/topics/treebanks/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{sogaard2013empirical,
  title = {{An Empirical Study of Differences between Conversion Schemes and Annotation Guidelines}},
  author = {Søgaard, Anders},
  booktitle = {Proceedings of the Second International Conference on Dependency Linguistics (DepLing 2013)},
  year = {2013},
  url = {https://aclanthology.org/W13-3733/}
}
```

- [← previous Predicting Conjunct Propagation and Other Extended Stanford Dependencies](https://md.mova.elamur.ai/free/en/docs/papers/bank-2013-nyblom-predicting-conjunct-propagation-other-extended/)
- [next → Validation Issues induced by an Automatic Pre-Annotation Mechanism in the Building of Non-projective Dependency Treebanks](https://md.mova.elamur.ai/free/en/docs/papers/bank-2014-lacroix-validation-issues-induced-automatic-pre/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

34 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bernd Bohnet. 2010. Top accuracy and fast dependency parsing is not a contradiction. In COLING. [find](https://scholar.google.com/scholar?q=Bernd+Bohnet%2E+2010%2E+Top+accuracy+and+fast+dependency+parsing+is+not+a+contradiction%2E+In+COLING%2E)
2. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: controlling for optimizer instability. In ACL. [find](https://scholar.google.com/scholar?q=Jonathan+H%2E+Clark%2C+Chris+Dyer%2C+Alon+Lavie%2C+and+Noah+A%2E+Smith%2E+2011%2E+Better+hypothesis+testing+for+statistical+machine+translation%3A+controlling+for+optimizer+instability%2E+In+ACL%2E)
3. Mike Collins. 1999. Head-driven statistical models for natural language parsing. Ph.D. thesis, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=Mike+Collins%2E+1999%2E+Head%2Ddriven+statistical+models+for+natural+language+parsing%2E+Ph%2ED%2E+thesis%2C+University+of+Pennsylvania%2E)
4. Michael Collins. 2002. Discriminative training methods for Hidden Markov Models. In EMNLP. [find](https://scholar.google.com/scholar?q=Michael+Collins%2E+2002%2E+Discriminative+training+methods+for+Hidden+Markov+Models%2E+In+EMNLP%2E)
5. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning, 20(3):273– 297. [find](https://scholar.google.com/scholar?q=Corinna+Cortes+and+Vladimir+Vapnik%2E+1995%2E+Support%2Dvector+networks%2E+Machine+Learning%2C+20%283%29%3A273%E2%80%93+297%2E)
6. Koby Crammer and Yoram Singer. 2003. Ultraconservative algorithms for multiclass problems. In JMLR. [find](https://scholar.google.com/scholar?q=Koby+Crammer+and+Yoram+Singer%2E+2003%2E+Ultraconservative+algorithms+for+multiclass+problems%2E+In+JMLR%2E)
7. Jakob Elming and Martin Haulrich. 2011. Reordering by parsing. In Proceedings of International Workshop on Using Linguistic Information for Hybrid Machine Translation (LIHMT-2011). [find](https://scholar.google.com/scholar?q=Jakob+Elming+and+Martin+Haulrich%2E+2011%2E+Reordering+by+parsing%2E+In+Proceedings+of+International+Workshop+on+Using+Linguistic+Information+for+Hybrid+Machine+Translation+%28LIHMT%2D2011%29%2E)
8. Jakob Elming, Anders Johannsen, Sigrid Klerke, Emanuele Lapponi, Hector Martinez Alonso, and Anders Søgaard. 2013. Down-stream effects of tree-to-dependency conversions. In NAACL. [find](https://scholar.google.com/scholar?q=Jakob+Elming%2C+Anders+Johannsen%2C+Sigrid+Klerke%2C+Emanuele+Lapponi%2C+Hector+Martinez+Alonso%2C+and+Anders+S%C3%B8gaard%2E+2013%2E+Down%2Dstream+effects+of+tree%2Dto%2Ddependency+conversions%2E+In+NAACL%2E)
9. Michel Galley and Christopher Manning. 2009. Quadratic-time dependency parsing for machine translation. In ACL. [find](https://scholar.google.com/scholar?q=Michel+Galley+and+Christopher+Manning%2E+2009%2E+Quadratic%2Dtime+dependency+parsing+for+machine+translation%2E+In+ACL%2E)
10. Angelina Ivanova, Stephan Oepen, Lilja Øvrelid, and Dan Flickinger. 2012. Who did what to whom? a contrastive study of syntactico-semantic dependencies. In LAW. [find](https://scholar.google.com/scholar?q=Angelina+Ivanova%2C+Stephan+Oepen%2C+Lilja+%C3%98vrelid%2C+and+Dan+Flickinger%2E+2012%2E+Who+did+what+to+whom%3F+a+contrastive+study+of+syntactico%2Dsemantic+dependencies%2E+In+LAW%2E)
11. Richard Johansson and Alessandro Moschitti. 2010. Syntactic and semantic structure for opinion expression detection. In CoNLL. [find](https://scholar.google.com/scholar?q=Richard+Johansson+and+Alessandro+Moschitti%2E+2010%2E+Syntactic+and+semantic+structure+for+opinion+expression+detection%2E+In+CoNLL%2E)
12. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to-dependency conversion for English. In NODALIDA. [find](https://scholar.google.com/scholar?q=Richard+Johansson+and+Pierre+Nugues%2E+2007%2E+Extended+constituent%2Dto%2Ddependency+conversion+for+English%2E+In+NODALIDA%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/map-2007-johansson-extended-constituent-dependency-conversion-english/)
13. Richard Johansson and Pierre Nugues. Dependency-based syntactic-semantic analysis with propbank and nombank. In CoNLL. [find](https://scholar.google.com/scholar?q=Richard+Johansson+and+Pierre+Nugues%2E+Dependency%2Dbased+syntactic%2Dsemantic+analysis+with+propbank+and+nombank%2E+In+CoNLL%2E)
14. Richard Johansson. 2013. Training parsers on incompatible treebanks. In NAACL. [find](https://scholar.google.com/scholar?q=Richard+Johansson%2E+2013%2E+Training+parsers+on+incompatible+treebanks%2E+In+NAACL%2E)
15. Mahesh Joshi and Carolyn Penstein-Rose. 2009. Generalizing dependency features for opinion mining. In ACL. [find](https://scholar.google.com/scholar?q=Mahesh+Joshi+and+Carolyn+Penstein%2DRose%2E+2009%2E+Generalizing+dependency+features+for+opinion+mining%2E+In+ACL%2E)
16. Kevin Knight and Daniel Marcu. 2002. Summarization beyond sentence extraction: a probabilistic approach to sentence compression. Artificial Intelligence, 139:91–107. [find](https://scholar.google.com/scholar?q=Kevin+Knight+and+Daniel+Marcu%2E+2002%2E+Summarization+beyond+sentence+extraction%3A+a+probabilistic+approach+to+sentence+compression%2E+Artificial+Intelligence%2C+139%3A91%E2%80%93107%2E)
17. Emanuele Lapponi, Erik Velldal, Lilja Øvrelid, and Jonathon Read. 2012. UiO2: Sequence-labeling negation using dependency features. In *SEM. [find](https://scholar.google.com/scholar?q=Emanuele+Lapponi%2C+Erik+Velldal%2C+Lilja+%C3%98vrelid%2C+and+Jonathon+Read%2E+2012%2E+UiO2%3A+Sequence%2Dlabeling+negation+using+dependency+features%2E+In+%2ASEM%2E)
18. Alon Lavie and Abhaya Agarwal. 2007. Meteor: an automatic metric for mt evaluation with high levels of correlation with human judgments. In WMT. [find](https://scholar.google.com/scholar?q=Alon+Lavie+and+Abhaya+Agarwal%2E+2007%2E+Meteor%3A+an+automatic+metric+for+mt+evaluation+with+high+levels+of+correlation+with+human+judgments%2E+In+WMT%2E)
19. Wei-Hao Lin and Alexander Hauptmann. 2006. Are these documents written from different perspectives? In COLING-ACL. [find](https://scholar.google.com/scholar?q=Wei%2DHao+Lin+and+Alexander+Hauptmann%2E+2006%2E+Are+these+documents+written+from+different+perspectives%3F+In+COLING%2DACL%2E)
20. Mitchell Marcus, Mary Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=Mitchell+Marcus%2C+Mary+Marcinkiewicz%2C+and+Beatrice+Santorini%2E+1993%2E+Building+a+large+annotated+corpus+of+English%3A+the+Penn+Treebank%2E+Computational+Linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
21. Ryan McDonald and Joakim Nivre. 2007. Characterizing the errors of data-driven dependency parsers. In EMNLP-CoNLL. [find](https://scholar.google.com/scholar?q=Ryan+McDonald+and+Joakim+Nivre%2E+2007%2E+Characterizing+the+errors+of+data%2Ddriven+dependency+parsers%2E+In+EMNLP%2DCoNLL%2E)
22. Ryan McDonald. 2006. Discriminative sentence compression with soft syntactic evidence. In EACL. [find](https://scholar.google.com/scholar?q=Ryan+McDonald%2E+2006%2E+Discriminative+sentence+compression+with+soft+syntactic+evidence%2E+In+EACL%2E)
23. Roser Morante and Eduardo Blanco. 2012. *sem 2012 shared task: Resolving the scope and focus of negation. In *SEM. [find](https://scholar.google.com/scholar?q=Roser+Morante+and+Eduardo+Blanco%2E+2012%2E+%2Asem+2012+shared+task%3A+Resolving+the+scope+and+focus+of+negation%2E+In+%2ASEM%2E)
24. Roser Morante and Caroline Sporleder. 2012. Modality and negation: An introduction to the special issue. Computational linguistics, 38(2):223–260. [find](https://scholar.google.com/scholar?q=Roser+Morante+and+Caroline+Sporleder%2E+2012%2E+Modality+and+negation%3A+An+introduction+to+the+special+issue%2E+Computational+linguistics%2C+38%282%29%3A223%E2%80%93260%2E)
25. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan Mc-Donald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Johan+Hall%2C+Sandra+Ku%CC%88bler%2C+Ryan+Mc%2DDonald%2C+Jens+Nilsson%2C+Sebastian+Riedel%2C+and+Deniz+Yuret%2E+2007%2E)
26. The CoNLL 2007 Shared Task on Dependency Parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915–932, Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=The+CoNLL+2007+Shared+Task+on+Dependency+Parsing%2E+In+Proceedings+of+the+CoNLL+Shared+Task+Session+of+EMNLP%2DCoNLL+2007%2C+pages+915%E2%80%93932%2C+Prague%2C+Czech+Republic%2E)
27. Kishore Papineni, Salim Roukus, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukus%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
28. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 Shared Task on Parsing the Web. In Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL). [find](https://scholar.google.com/scholar?q=Slav+Petrov+and+Ryan+McDonald%2E+2012%2E+Overview+of+the+2012+Shared+Task+on+Parsing+the+Web%2E+In+Notes+of+the+First+Workshop+on+Syntactic+Analysis+of+Non%2DCanonical+Language+%28SANCL%29%2E)
29. Roy Schwartz, Omri Abend, and Ari Rappoport. 2012. Learnability-based syntactic annotation design. In 2008. COLING. [find](https://scholar.google.com/scholar?q=Roy+Schwartz%2C+Omri+Abend%2C+and+Ari+Rappoport%2E+2012%2E+Learnability%2Dbased+syntactic+annotation+design%2E+In+2008%2E+COLING%2E)
30. Anders Søgaard and Martin Haulrich. 2010. On the derivation perplexity of treebanks. In TLT. [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard+and+Martin+Haulrich%2E+2010%2E+On+the+derivation+perplexity+of+treebanks%2E+In+TLT%2E)
31. Reut Tsarfaty, Joakim Nivre, and Evelina Andersson. 2012. Cross-framework evaluation for statistical parsing. In EACL. [find](https://scholar.google.com/scholar?q=Reut+Tsarfaty%2C+Joakim+Nivre%2C+and+Evelina+Andersson%2E+2012%2E+Cross%2Dframework+evaluation+for+statistical+parsing%2E+In+EACL%2E)
32. Erik Velldal, Lilja Øvrelid, Jonathon Read, and Stephan Oepen. 2012. Speculation and negation: Rules, rankers, and the role of synta. Computational linguistics, 38(2):369–410. [find](https://scholar.google.com/scholar?q=Erik+Velldal%2C+Lilja+%C3%98vrelid%2C+Jonathon+Read%2C+and+Stephan+Oepen%2E+2012%2E+Speculation+and+negation%3A+Rules%2C+rankers%2C+and+the+role+of+synta%2E+Computational+linguistics%2C+38%282%29%3A369%E2%80%93410%2E)
33. Peng Xu, Jaeho Kang, Michael Ringgaard, and Franz Och. 2009. Using a dependency parser to improve SMT for subject-object-verb languages. In Proceedings of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT) 2009, pages 245–253, Boulde… [find](https://scholar.google.com/scholar?q=Peng+Xu%2C+Jaeho+Kang%2C+Michael+Ringgaard%2C+and+Franz+Och%2E+2009%2E+Using+a+dependency+parser+to+improve+SMT+for+subject%2Dobject%2Dverb+languages%2E+In+Proceedings+of+the+North+American+Chapter+of+the+Association)
34. Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proceedings of the 8th International Workshop on Parsing Technologies, pages 195–206, Nancy, France. [find](https://scholar.google.com/scholar?q=Hiroyasu+Yamada+and+Yuji+Matsumoto%2E+2003%2E+Statistical+dependency+analysis+with+support+vector+machines%2E+In+Proceedings+of+the+8th+International+Workshop+on+Parsing+Technologies%2C+pages+195%E2%80%93206%2C+Nancy%2C+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/parsers-2003-yamada-statistical-dependency-analysis-support-vector/)