[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/ud-2017-straka-tokenizing-pos-tagging-lemmatizing-parsing/) [home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe # Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe **Authors:** [Milan Straka](https://md.mova.elamur.ai/free/en/docs/papers/authors/milan-straka/), [Jana Straková](https://md.mova.elamur.ai/free/en/docs/papers/authors/jana-strakova/) · **Year:** 2017 · **Venue:** Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies **Link:** [aclanthology.org/K17-3009](https://aclanthology.org/K17-3009/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) [**567** citations](https://openalex.org/works?filter=cites:W2741029840) openalex · 2026-10-18[**243** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/ud-2017-straka-tokenizing-pos-tagging-lemmatizing-parsing/) of the 140673 papers whose bibliographies we read[Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies](https://aclanthology.org/volumes/K17-3/)[licence](https://creativecommons.org/licenses/by/4.0/) - [Publisher's page](https://aclanthology.org/K17-3009/) - [PDF](https://aclanthology.org/K17-3009.pdf) - [DOI 10.18653/v1/k17-3009](https://doi.org/10.18653/v1/k17-3009) - [OpenAlex](https://openalex.org/W2741029840) **Authors (2)** 1. [Milan Straka](https://openalex.org/A5060081515) [ORCID](https://orcid.org/0000-0003-3295-5576) · h 22 · 3117 citations · 294 works 2. [Jana Straková](https://openalex.org/A5047678040) [ORCID](https://orcid.org/0000-0003-0075-2408) · h 12 · 1692 citations · 72 works Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart. ## What it is about The paper describes UDPipe 1.1 (the shared-task baseline) and UDPipe 1.2 (the participant system, which placed 8th of 33). Tokenisation and sentence splitting come from one character-level GRU; multi-word tokens are split by a dictionary plus suffix rules; the tagger uses a suffix-based morphological guesser with an averaged perceptron; lemmas are chosen as separate rules by a second tagger; and the parser is the transition-based Parsito with a search-based oracle and a default beam of 5. Tuning steps that helped include a larger tokenizer (dimension 24 to 64) for hard tokenisation, adding other treebanks of the same language to small training sets, and joint segmentation with parsing for three treebanks that lack punctuation. The search-based oracle raises the arc-standard static-oracle LAS from 68.27 to 68.87 on the authors' data. The full model has a median size of about 13 MB and runs at about 1,700 words per second on one CPU core. ## Ideas - I1. en/src/tok.rs: keep the exact spacing in MISC fields (SpacesBefore, SpacesAfter, TokenOffset, as this paper defines them) and measure the round-trip rate of raw text through tokenisation and back on EWT dev. - I2. en/src/parse.rs: train the arc-standard perceptron with a search-based oracle (the best gold-consistent state in the beam, not only the static move) and measure LAS on EWT dev through `en parse-eval`, twice. - I3. en/src/parse.rs: retrain on EWT plus a second English treebank whose licence allows training (check [Data and model licences](https://md.mova.elamur.ai/free/en/docs/licence/DATA/)), and plot dev LAS against the added size with `en parse-eval`, to see whether small treebanks gain as they did here. - I4. en/src/morph.rs: add suffix-rule splitting for contractions that the dictionary misses, applied only when a rule does not fire too often on the training data, and measure LAS on ATIS (currently 63.0, lowercase queries). ## For Mova The ideas touch the tokenizer `en/src/tok.rs`, the parser `en/src/parse.rs` and the lexicon in `en/src/morph.rs`, and they use `en parse-eval`, `en ud-eval` and the ATIS data. The chapters are `languages/en/07-punctuation-capitals-numbers` (contractions, spacing) and `languages/en/19-lexicon-mwe-register` (multi-word tokens). Measure first I2, because a change to the oracle is cheap to test and the paper reports a clear gain for it. Topics: [Dependency parsing](https://md.mova.elamur.ai/free/en/docs/papers/topics/dependency-parsing/) · [Universal Dependencies](https://md.mova.elamur.ai/free/en/docs/papers/topics/universal-dependencies/) · [Treebanks](https://md.mova.elamur.ai/free/en/docs/papers/topics/treebanks/) · [Part-of-speech tagging](https://md.mova.elamur.ai/free/en/docs/papers/topics/part-of-speech-tagging/) ## Cite The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/k17-3009](https://doi.org/10.18653/v1/k17-3009) ``` @inproceedings{straka2017tokenizing, title = {{Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe}}, author = {Straka, Milan and Straková, Jana}, booktitle = {Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies}, year = {2017}, doi = {10.18653/v1/k17-3009}, url = {https://doi.org/10.18653/v1/k17-3009} } ``` - [← previous Gapping Constructions in Universal Dependencies v2](https://md.mova.elamur.ai/free/en/docs/papers/ud-2017-schuster-gapping-constructions-universal-dependencies-v2/) - [next → A non-projective greedy dependency parser with bidirectional LSTMs](https://md.mova.elamur.ai/free/en/docs/papers/ud-2017-vilares-non-projective-greedy-dependency-parser/) - [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Glob-aly normalized transition-based neural networks. In Association for Computational Linguistic. http://arxiv.org/abs/1603.06042. [↗](http://arxiv.org/abs/1603.06042)