[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/kalpakchi-2021-minor-changes-make-difference-case/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / Minor changes make a difference: a case study on the consistency of UD-based dependency parsers

# Minor changes make a difference: a case study on the consistency of UD-based dependency parsers

**Authors:** Dmytro Kalpakchi, Johan Boye · **Year:** 2021 · **Venue:** Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021) **Link:** [aclanthology.org/2021.udw-1.8](https://aclanthology.org/2021.udw-1.8/) (ACL Anthology 2021.udw-1.8) **License of the paper:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology) — [creativecommons.org/licenses/by/4.0](https://creativecommons.org/licenses/by/4.0/)

[**0** citations](https://openalex.org/works?filter=cites:W3216978656) openalex · 2026-10-18[**2** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/kalpakchi-2021-minor-changes-make-difference-case/) of the 140673 papers whose bibliographies we read[Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021)](https://aclanthology.org/volumes/2021.udw-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2021.udw-1.8/)
- [PDF](https://aclanthology.org/2021.udw-1.8.pdf)
- [arXiv 2111.15413](https://arxiv.org/abs/2111.15413)
- [OpenAlex](https://openalex.org/W3216978656)

**Authors (2)**

1. [Dmytro Kalpakchi](https://openalex.org/A5054645028) [ORCID](https://orcid.org/0000-0001-7327-3059) · h 3 · 53 citations · 27 works
2. [Johan Boye](https://openalex.org/A5041634703) [ORCID](https://orcid.org/0000-0003-2600-7668) · h 17 · 707 citations · 89 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## Summary

The paper asks whether off-the-shelf UD parsers stay consistent when a sentence is changed only trivially. The authors take treebank sentences containing a four-digit number (mostly a year) in English, Swedish, Russian and Ukrainian, substitute 50 random values for that number, and parse every variant with pretrained Stanza models. Whole-sentence parse quality is compared with a tree-kernel similarity, and groups of identical errors are found as maximal cliques. Standard metrics (UAS, LAS, BLEX) barely move, yet a large share of variant bundles (about 44% on average) show errors that differ from one variant to another. Two retraining fixes are tried: augmenting the training data with re-numbered sentences, and replacing numbers with a placeholder token; only augmentation helps noticeably. The authors also point to a likely temporal bias in treebanks and warn applications that rely on UD trees.

## How Mova uses it

- [How to find annotation errors automatically](https://md.mova.elamur.ai/free/en/docs/en/seeds/errors/consistency-detection-methods/) — listed as method 5 ("robustness to small changes") among automatic ways to find annotation errors: places where the parse flips under a trivial substitution are candidates for errors.
- [nummod — only quantity and only before the noun](https://md.mova.elamur.ai/free/en/docs/en/seeds/errors/numbered-entities-nummod/) — cited as evidence that numbers are unstable for parsers, as background to the rules `en.nominal.nummod-before-noun` and `en.nominal.nummod-num` (numbers after a noun are identifiers, not counts).
- Used as background reading only; Mova has no perturbation-based checker built from this paper.

## Effectiveness in Mova

Background reading behind one seed note and one detection idea; no component implements the method. Not measured separately.

---

👨‍🔬💥

Topics: [Dependency parsing](https://md.mova.elamur.ai/free/en/docs/papers/topics/dependency-parsing/) · [Universal Dependencies](https://md.mova.elamur.ai/free/en/docs/papers/topics/universal-dependencies/) · [Treebanks](https://md.mova.elamur.ai/free/en/docs/papers/topics/treebanks/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 2111.15413](https://arxiv.org/abs/2111.15413)

```
@inproceedings{kalpakchi2021minor,
  title = {{Minor changes make a difference: a case study on the consistency of UD-based dependency parsers}},
  author = {Kalpakchi, Dmytro and Boye, Johan},
  booktitle = {Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021)},
  year = {2021},
  eprint = {2111.15413},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.15413}
}
```

- [← previous Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction](https://md.mova.elamur.ai/free/en/docs/papers/jie-2022-learning-reason-deductively-math-word/)
- [next → Parsing the Switch: LLM-Based UD Annotation for Complex Code-Switched and Low-Resource Languages](https://md.mova.elamur.ai/free/en/docs/papers/kellert-2025-parsing-switch-llm-based-ud/)
- [↑ back to papers](https://md.mova.elamur.ai/free/en/docs/papers/)

## References

11 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Coen Bron and Joep Kerbosch. 1973. Algorithm 457: finding all cliques of an undirected graph. Communications of the ACM, 16(9):575–577. [find](https://scholar.google.com/scholar?q=Coen+Bron+and+Joep+Kerbosch%2E+1973%2E+Algorithm+457%3A+finding+all+cliques+of+an+undirected+graph%2E+Communications+of+the+ACM%2C+16%289%29%3A575%E2%80%93577%2E)
2. Aditi Chaudhary, Antonios Anastasopoulos, Adithya Pratapa, David R. Mortensen, Zaid Sheikh, Yulia Tsvetkov, and Graham Neubig. 2020. Automatic extraction of rules governing morphological agreement. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pag… [find](https://scholar.google.com/scholar?q=Aditi+Chaudhary%2C+Antonios+Anastasopoulos%2C+Adithya+Pratapa%2C+David+R%2E+Mortensen%2C+Zaid+Sheikh%2C+Yulia+Tsvetkov%2C+and+Graham+Neubig%2E+2020%2E+Automatic+extraction+of+rules+governing+morphological+agreement%2E+In)
3. Danilo Croce, Alessandro Moschitti, and Roberto Basili. 2011. Structured lexical similarity via convolution kernels on dependency trees. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1034–1046, Edinburgh, Scotland, UK., July. Association for Computa… [find](https://scholar.google.com/scholar?q=Danilo+Croce%2C+Alessandro+Moschitti%2C+and+Roberto+Basili%2E+2011%2E+Structured+lexical+similarity+via+convolution+kernels+on+dependency+trees%2E+In+Proceedings+of+the+2011+Conference+on+Empirical+Methods+in+N)
4. Dmytro Kalpakchi and Johan Boye. 2020. UDon2: a library for manipulating Universal Dependencies trees. In Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020), pages 120–125, Barcelona, Spain (Online), December. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dmytro+Kalpakchi+and+Johan+Boye%2E+2020%2E+UDon2%3A+a+library+for+manipulating+Universal+Dependencies+trees%2E+In+Proceedings+of+the+Fourth+Workshop+on+Universal+Dependencies+%28UDW+2020%29%2C+pages+120%E2%80%93125%2C+Barcel)
5. Dmytro Kalpakchi and Johan Boye. 2021. Quinductor: a multilingual data-driven method for generating reading-comprehension questions using universal dependencies. arXiv preprint arXiv:2103.10121. — [arXiv:2103.10121](https://arxiv.org/abs/2103.10121) [find](https://scholar.google.com/scholar?q=Dmytro+Kalpakchi+and+Johan+Boye%2E+2021%2E+Quinductor%3A+a+multilingual+data%2Ddriven+method+for+generating+reading%2Dcomprehension+questions+using+universal+dependencies%2E+arXiv+preprint+arXiv%3A2103%2E10121%2E+%E2%80%94+%5Bar)
6. Alessandro Moschitti. 2006. Efficient convolution kernels for dependency and constituent syntactic trees. In European Conference on Machine Learning, pages 318–329. Springer. [find](https://scholar.google.com/scholar?q=Alessandro+Moschitti%2E+2006%2E+Efficient+convolution+kernels+for+dependency+and+constituent+syntactic+trees%2E+In+European+Conference+on+Machine+Learning%2C+pages+318%E2%80%93329%2E+Springer%2E)
7. Siyao Peng and Amir Zeldes. 2018. All roads lead to UD: Converting Stanford and Penn parses to English Universal Dependencies with multilayer annotations. In Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018), pages 167–177, Santa F… [find](https://scholar.google.com/scholar?q=Siyao+Peng+and+Amir+Zeldes%2E+2018%2E+All+roads+lead+to+UD%3A+Converting+Stanford+and+Penn+parses+to+English+Universal+Dependencies+with+multilayer+annotations%2E+In+Proceedings+of+the+Joint+Workshop+on+Lingu) [our card](https://md.mova.elamur.ai/free/en/docs/papers/peng-2018-all-roads-lead-ud-converting/)
8. Martin Popel, Zdeněk Žabokrtský, and Martin Vojtek. 2017. Udapi: Universal API for Universal Dependencies. In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017), pages 96–101, Gothenburg, Sweden, May. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Martin+Popel%2C+Zdene%CC%8Ck+Z%CC%8Cabokrtsky%CC%81%2C+and+Martin+Vojtek%2E+2017%2E+Udapi%3A+Universal+API+for+Universal+Dependencies%2E+In+Proceedings+of+the+NoDaLiDa+2017+Workshop+on+Universal+Dependencies+%28UDW+2017%29%2C+pages+9) [our card](https://md.mova.elamur.ai/free/en/docs/papers/popel-2017-udapi-universal-api-universal-dependencies/)
9. Adithya Pratapa, Antonios Anastasopoulos, Shruti Rijhwani, Aditi Chaudhary, David R Mortensen, Graham Neubig, and Yulia Tsvetkov. 2021. Evaluating the morphosyntactic well-formedness of generated texts. arXiv preprint arXiv:2103.16590. [find](https://scholar.google.com/scholar?q=Adithya+Pratapa%2C+Antonios+Anastasopoulos%2C+Shruti+Rijhwani%2C+Aditi+Chaudhary%2C+David+R+Mortensen%2C+Graham+Neubig%2C+and+Yulia+Tsvetkov%2E+2021%2E+Evaluating+the+morphosyntactic+well%2Dformedness+of+generated+text)
10. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online,… [find](https://scholar.google.com/scholar?q=Peng+Qi%2C+Yuhao+Zhang%2C+Yuhui+Zhang%2C+Jason+Bolton%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Stanza%3A+A+python+natural+language+processing+toolkit+for+many+human+languages%2E+In+Proceedings+of+the+58th+Annual+Meet) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2020-qi-stanza-python-natural-language-processing/)
11. Milan Straka, Jan Hajič, and Jana Straková. 2016. UDPipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4290–4297,… [find](https://scholar.google.com/scholar?q=Milan+Straka%2C+Jan+Hajic%CC%8C%2C+and+Jana+Strakova%CC%81%2E+2016%2E+UDPipe%3A+Trainable+pipeline+for+processing+CoNLL%2DU+files+performing+tokenization%2C+morphological+analysis%2C+POS+tagging+and+parsing%2E+In+Proceedings+of+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2016-straka-udpipe-trainable-pipeline-processing-conll/)