[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/marneffe-2017-assessing-annotation-consistency-universal-dependencies/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / Assessing the Annotation Consistency of the Universal Dependencies Corpora

# Assessing the Annotation Consistency of the Universal Dependencies Corpora

**Authors:** [Marie-Catherine de Marneffe](https://md.mova.elamur.ai/free/en/docs/papers/authors/marie-catherine-de-marneffe/), Matias Grioni, [Jenna Kanerva](https://md.mova.elamur.ai/free/en/docs/papers/authors/jenna-kanerva/), [Filip Ginter](https://md.mova.elamur.ai/free/en/docs/papers/authors/filip-ginter/) · **Year:** 2017 · **Venue:** Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017) **Link:** [aclanthology.org/W17-6514](https://aclanthology.org/W17-6514/) (ACL Anthology W17-6514) **License of the paper:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology) — [creativecommons.org/licenses/by/4.0](https://creativecommons.org/licenses/by/4.0/)

[**11** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/marneffe-2017-assessing-annotation-consistency-universal-dependencies/) of the 140673 papers whose bibliographies we read[Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017)](https://aclanthology.org/volumes/W17-65/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/W17-6514/)
- [PDF](https://aclanthology.org/W17-6514.pdf)

**Authors (4)**

1. Marie-Catherine de Marneffe
2. Matias Grioni
3. Jenna Kanerva
4. Filip Ginter

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## Summary

UD treebanks are produced quickly and for many languages, so manual quality control is hard; the paper looks for a way to find inconsistent annotation automatically. It builds on the "variation nuclei" method of Boyd et al. (2008): pairs of words with the same left and right context that are linked by different dependency relations in different sentences signal that one of the occurrences is probably wrong. The method is reproduced on UD v2 for English, French and Finnish, and extended by comparing lemmas instead of word forms and by adding large automatically parsed corpora (parsebanks). The lemma variant works well for English and French (precision around 62–65%) but produces many false alarms for Finnish, where case changes syntactic role; using word forms helps there. Parsebanks surface new errors but add only moderate recall. The authors release a web tool for manual review of the flagged cases.

## How Mova uses it

- [How to find annotation errors automatically](https://md.mova.elamur.ai/free/en/docs/en/seeds/errors/consistency-detection-methods/) — method 1 ("variation nuclei") among the automatic error-detection methods Mova catalogues, with the paper's figures (266 suspicious lemma pairs in English UD v2, 62% precision; French 65%, Finnish 19%).
- The same principle underlies Mova's expert-rule practice: a rule that fires often on gold data is either wrong or is catching gold-standard errors, and violations on EWT 2.18 have indeed exposed treebank errors.
- Mova does not run a variation-nuclei detector itself; the paper is a method reference.

## Effectiveness in Mova

Background reading for the error-detection seed note. Not measured separately.

---

👨‍🔬💥

## More ideas (a second reading)

- I1. `en ud-check`: add a report of variation nuclei for the English treebanks. A nucleus is a pair of lemmas with the same left and right lemma context and different relations. Check 100 flagged pairs by hand on EWT train, and report precision as the paper does.
- I2. Lemma against form: in the same report, run the check on lemmas and on word forms. The paper shows that lemmas help English and fail for Finnish, so the rule must be set per language, not globally.
- I3. Parsebank check: parse a raw English collection with `en` and flag contexts where the `en` analysis and the treebank disagree at least five times, as the paper does. Measure how many flags are real errors on a hand-checked sample of 50, before any change to `treebanks/en/ewt`.

Topics: [Universal Dependencies](https://md.mova.elamur.ai/free/en/docs/papers/topics/universal-dependencies/) · [Treebanks](https://md.mova.elamur.ai/free/en/docs/papers/topics/treebanks/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{marneffe2017assessing,
  title = {{Assessing the Annotation Consistency of the Universal Dependencies Corpora}},
  author = {Marneffe, Marie-Catherine de and Grioni, Matias and Kanerva, Jenna and Ginter, Filip},
  booktitle = {Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017)},
  year = {2017},
  url = {https://aclanthology.org/W17-6514/}
}
```

- [← previous The Stanford Typed Dependencies Representation](https://md.mova.elamur.ai/free/en/docs/papers/marneffe-2008-stanford-typed-dependencies-representation/)
- [next → Annotating Second Language in Universal Dependencies: a Review of Current Practices and Directions for Harmonized Guidelines](https://md.mova.elamur.ai/free/en/docs/papers/masciolini-2025-annotating-second-language-universal-dependencies/)
- [↑ back to papers](https://md.mova.elamur.ai/free/en/docs/papers/)

## References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bharat Ram Ambati, Rahul Agarwal, Mridul Gupta, Samar Husain, and Dipti Misra Sharma. 2011. Error detection for treebank validation. Proceedings of the 9th Workshop on Asian Language Resources, pages 23–30. [find](https://scholar.google.com/scholar?q=Bharat+Ram+Ambati%2C+Rahul+Agarwal%2C+Mridul+Gupta%2C+Samar+Husain%2C+and+Dipti+Misra+Sharma%2E+2011%2E+Error+detection+for+treebank+validation%2E+Proceedings+of+the+9th+Workshop+on+Asian+Language+Resources%2C+pages+)
2. Alena Böhmová, Jan Hajič, Eva Hajičová, and Barbora Hladká. 2003. The Prague dependency treebank. In Treebanks, pages 103–127. Springer. [find](https://scholar.google.com/scholar?q=Alena+Bo%CC%88hmova%CC%81%2C+Jan+Hajic%CC%8C%2C+Eva+Hajic%CC%8Cova%CC%81%2C+and+Barbora+Hladka%CC%81%2E+2003%2E+The+Prague+dependency+treebank%2E+In+Treebanks%2C+pages+103%E2%80%93127%2E+Springer%2E)
3. Guillaume Bonfante, Bruno Guillaume, Mathieu Morey, and Guy Perrier. 2011. Modular graph rewriting to compute semantics. In Proceedings of the Ninth International Conference on Computational Semantics, pages 65–74. [find](https://scholar.google.com/scholar?q=Guillaume+Bonfante%2C+Bruno+Guillaume%2C+Mathieu+Morey%2C+and+Guy+Perrier%2E+2011%2E+Modular+graph+rewriting+to+compute+semantics%2E+In+Proceedings+of+the+Ninth+International+Conference+on+Computational+Semantics)
4. Adriane Boyd, Markus Dickinson, and W Detmar Meurers. 2008. On detecting errors in dependency treebanks. Research on Language & Computation, 6(2):113–137. [find](https://scholar.google.com/scholar?q=Adriane+Boyd%2C+Markus+Dickinson%2C+and+W+Detmar+Meurers%2E+2008%2E+On+detecting+errors+in+dependency+treebanks%2E+Research+on+Language+%26+Computation%2C+6%282%29%3A113%E2%80%93137%2E)
5. Koenraad De Smedt, Victoria Rosén, and Paul Meurer. 2016. Studying consistency in UD treebanks with INESS-Search. In Fourteenth Workshop on Treebanks and Linguistic Theories (TLT14). [find](https://scholar.google.com/scholar?q=Koenraad+De+Smedt%2C+Victoria+Rose%CC%81n%2C+and+Paul+Meurer%2E+2016%2E+Studying+consistency+in+UD+treebanks+with+INESS%2DSearch%2E+In+Fourteenth+Workshop+on+Treebanks+and+Linguistic+Theories+%28TLT14%29%2E)
6. Markus Dickinson and W. Detmar Meurers. 2003a. Detecting errors in part-of-speech annotation. In Proceedings of the Tenth Conference on European Chapter of the Association for Computational Linguistics, pages 107–114. [find](https://scholar.google.com/scholar?q=Markus+Dickinson+and+W%2E+Detmar+Meurers%2E+2003a%2E+Detecting+errors+in+part%2Dof%2Dspeech+annotation%2E+In+Proceedings+of+the+Tenth+Conference+on+European+Chapter+of+the+Association+for+Computational+Linguistic)
7. Markus Dickinson and W. Detmar Meurers. 2003b. Detecting inconsistencies in treebanks. In Proceedings of the Second Workshop on Treebanks and Linguistic Theories (TLT-03). [find](https://scholar.google.com/scholar?q=Markus+Dickinson+and+W%2E+Detmar+Meurers%2E+2003b%2E+Detecting+inconsistencies+in+treebanks%2E+In+Proceedings+of+the+Second+Workshop+on+Treebanks+and+Linguistic+Theories+%28TLT%2D03%29%2E)
8. Markus Dickinson and W. Detmar Meurers. 2005. Detecting errors in discontinuous structural annotation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 322–329. [find](https://scholar.google.com/scholar?q=Markus+Dickinson+and+W%2E+Detmar+Meurers%2E+2005%2E+Detecting+errors+in+discontinuous+structural+annotation%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages)
9. Eleazar Eskin. 2000. Detecting errors within a corpus using anomaly detection. In Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, pages 148– 153. [find](https://scholar.google.com/scholar?q=Eleazar+Eskin%2E+2000%2E+Detecting+errors+within+a+corpus+using+anomaly+detection%2E+In+Proceedings+of+the+1st+North+American+Chapter+of+the+Association+for+Computational+Linguistics+Conference%2C+pages+148%E2%80%93+)
10. Martin Forst, Núria Bertomeu, Berthold Crysmann, Frederik Fouvry, Silvia Hansen-Schirra, and Valia Kordoni. 2004. Towards a dependency-based gold standard for German parsers – the TiGer Dependency Bank. In Proceedings of the COL-ING Workshop on Linguistically Interpreted Corpora (LINC04), Geneva, Switzerland. [find](https://scholar.google.com/scholar?q=Martin+Forst%2C+Nu%CC%81ria+Bertomeu%2C+Berthold+Crysmann%2C+Frederik+Fouvry%2C+Silvia+Hansen%2DSchirra%2C+and+Valia+Kordoni%2E+2004%2E+Towards+a+dependency%2Dbased+gold+standard+for+German+parsers+%E2%80%93+the+TiGer+Dependency+Ba)
11. Markus Gärtner, Gregor Thiele, Wolfgang Seeker, Anders Björkelund, and Jonas Kuhn. 2013. ICARUS – An extensible graphical search tool for dependency treebanks. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. [find](https://scholar.google.com/scholar?q=Markus+Ga%CC%88rtner%2C+Gregor+Thiele%2C+Wolfgang+Seeker%2C+Anders+Bjo%CC%88rkelund%2C+and+Jonas+Kuhn%2E+2013%2E+ICARUS+%E2%80%93+An+extensible+graphical+search+tool+for+dependency+treebanks%2E+In+Proceedings+of+the+51st+Annual+Meet)
12. Filip Ginter, Jan Hajič, Juhani Luotolahti, Milan Straka, and Daniel Zeman. 2017. CoNLL 2017 shared task - automatically annotated raw texts and word embeddings. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University. [find](https://scholar.google.com/scholar?q=Filip+Ginter%2C+Jan+Hajic%CC%8C%2C+Juhani+Luotolahti%2C+Milan+Straka%2C+and+Daniel+Zeman%2E+2017%2E+CoNLL+2017+shared+task+%2D+automatically+annotated+raw+texts+and+word+embeddings%2E+LINDAT%2FCLARIN+digital+library+at+the+)
13. Juhani Luotolahti, Jenna Kanerva, Sampo Pyysalo, and Filip Ginter. 2015. SETS: Scalable and efficient tree search in dependency graphs. In NAACL Demo, pages 51–55. [find](https://scholar.google.com/scholar?q=Juhani+Luotolahti%2C+Jenna+Kanerva%2C+Sampo+Pyysalo%2C+and+Filip+Ginter%2E+2015%2E+SETS%3A+Scalable+and+efficient+tree+search+in+dependency+graphs%2E+In+NAACL+Demo%2C+pages+51%E2%80%9355%2E)
14. Joakim Nivre, Jens Nilsson, and Johan Hall. 2006. Talbanken05: A Swedish treebank with phrase structure and dependency annotation. In Proceedings of the fifth international conference on Language Resources and Evaluation (LREC-06). [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Jens+Nilsson%2C+and+Johan+Hall%2E+2006%2E+Talbanken05%3A+A+Swedish+treebank+with+phrase+structure+and+dependency+annotation%2E+In+Proceedings+of+the+fifth+international+conference+on+Language+Reso)
15. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), pages 1659–1666. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Marie%2DCatherine+de+Marneffe%2C+Filip+Ginter%2C+Yoav+Goldberg%2C+Jan+Hajic%2C+Christopher+D+Manning%2C+Ryan+McDonald%2C+Slav+Petrov%2C+Sampo+Pyysalo%2C+Natalia+Silveira%2C+et+al%2E+2016%2E+Universal+dependenci) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2016-nivre-universal-dependencies-v1-multilingual-treebank/)
16. Jan Štěpánek and Petr Pajas. 2010. Querying diverse treebanks in a uniform way. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010). [find](https://scholar.google.com/scholar?q=Jan+S%CC%8Cte%CC%8Cpa%CC%81nek+and+Petr+Pajas%2E+2010%2E+Querying+diverse+treebanks+in+a+uniform+way%2E+In+Proceedings+of+the+7th+International+Conference+on+Language+Resources+and+Evaluation+%28LREC+2010%29%2E)
17. Milan Straka, Jan Hajič, and Jana Straková. 2016. UD-Pipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, pos tagging and parsing. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Paris, France, May. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Milan+Straka%2C+Jan+Hajic%CC%8C%2C+and+Jana+Strakova%CC%81%2E+2016%2E+UD%2DPipe%3A+Trainable+pipeline+for+processing+CoNLL%2DU+files+performing+tokenization%2C+morphological+analysis%2C+pos+tagging+and+parsing%2E+In+Proceedings+of)
18. Gregor Thiele, Wolfgang Seeker, Markus Gärtner, Anders Björkelund, and Jonas Kuhn. 2014. A graphical interface for automatic error mining in corpora. In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 57–60. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gregor+Thiele%2C+Wolfgang+Seeker%2C+Markus+Ga%CC%88rtner%2C+Anders+Bjo%CC%88rkelund%2C+and+Jonas+Kuhn%2E+2014%2E+A+graphical+interface+for+automatic+error+mining+in+corpora%2E+In+Proceedings+of+the+Demonstrations+at+the+14th)
19. Hans van Halteren. 2000. The detection of inconsistency in manually tagged text. In Proceedings of the 2nd Workshop on Linguistically Interpreted Corpora. [find](https://scholar.google.com/scholar?q=Hans+van+Halteren%2E+2000%2E+The+detection+of+inconsistency+in+manually+tagged+text%2E+In+Proceedings+of+the+2nd+Workshop+on+Linguistically+Interpreted+Corpora%2E)