[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2012-dridan-tokenization-returning-long-solved-problem/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Tokenization: Returning to a Long Solved Problem – A Survey, Contrastive Experiment, Recommendations, and Toolkit

# Tokenization: Returning to a Long Solved Problem – A Survey, Contrastive Experiment, Recommendations, and Toolkit

**Authors:** [Rebecca Dridan](https://md.mova.elamur.ai/free/en/docs/papers/authors/rebecca-dridan/), [Stephan Oepen](https://md.mova.elamur.ai/free/en/docs/papers/authors/stephan-oepen/) · **Year:** 2012 · **Venue:** Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) **Link:** [aclanthology.org/P12-2074](https://aclanthology.org/P12-2074/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**26** citations](https://openalex.org/works?filter=cites:W2138181100) openalex · 2026-10-19[**18** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2012-dridan-tokenization-returning-long-solved-problem/) of the 140673 papers whose bibliographies we read[Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://aclanthology.org/volumes/P12-2/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/P12-2074/)
- [PDF](https://aclanthology.org/P12-2074.pdf)
- [OpenAlex](https://openalex.org/W2138181100)

**Authors (2)**

1. [Rebecca Dridan](https://openalex.org/A5043168719) [Semantic Scholar](https://www.semanticscholar.org/author/1833727) · h 8 · 235 citations · 25 works
2. [Stephan Oepen](https://openalex.org/A5058412010) [Semantic Scholar](https://www.semanticscholar.org/author/2949607) · h 36 · 3610 citations · 234 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

The authors ask how far Penn Treebank (PTB) style tokenization can be reproduced by rules, and why earlier tools differ from it. They compare three tools against the gold PTB tokens of 49,208 sentences (1,173,750 tokens): the PTB tokenizer.sed script, Stanford CoreNLP and the tokenizer of the Charniak and Johnson parser. The tools disagree on 3,264, 1,781 and 2,597 sentences, respectively. Their own rule-based toolkit keeps exact stand-off pointers into the raw text and is configurable per genre. With a small optional set of extra rules, the remaining differences fall to 603 sentences, just over 1%, and the authors say this removes two thirds of the errors of the best off-the-shelf tool.

## Ideas

- I1. `en/src/tok.rs`: add a PTB-compatible mode as a rule list (split "can't" into "ca" and "n't", handle possessive 's, keep "U.S." without a split point) and report the exact-match rate per rule. Measure on EWT test, where the current tokenizer gets 90.0%.
- I2. Stand-off offsets: check that every token of `en/src/tok.rs` keeps the byte span of the raw text, then test the paper's claim that each token links back to the input on the EWT test set.
- I3. Count, on EWT test, the sentences where the tokenizer and the gold tokens differ, and report the Levenshtein distance over tokens as the paper does, next to the sentence count. Use it as a second number beside the exact-match rate.

## For Mova

The code is `en/src/tok.rs`, with the command `en tok-eval`. The chapter is [07. Punctuation, capital letters, abbreviations and numbers](https://md.mova.elamur.ai/free/en/docs/languages/en/07-punctuation-capitals-numbers/). The PTB itself is licensed for research use; measure with it only as a test, never as a source of rules that ship. Measure first on EWT test: the sentence count and the Levenshtein distance for the current tokenizer.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{dridan2012tokenization,
  title = {{Tokenization: Returning to a Long Solved Problem – A Survey, Contrastive Experiment, Recommendations, and Toolkit}},
  author = {Dridan, Rebecca and Oepen, Stephan},
  booktitle = {Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2012},
  url = {https://aclanthology.org/P12-2074/}
}
```

- [← previous Automatic Detection and Correction of Errors in Dependency Treebanks](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2011-volokh-automatic-detection-correction-errors-dependency/)
- [next → Lemmatisation as a Tagging Task](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2012-gesmundo-lemmatisation-as-tagging-task/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

9 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Charniak, E., & Johnson, M. (2005). Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (pp. 173–180). Ann Arbor, USA. [find](https://scholar.google.com/scholar?q=Charniak%2C+E%2E%2C+%26+Johnson%2C+M%2E+%282005%29%2E+Coarse%2Dto%2Dfine+n%2Dbest+parsing+and+maxent+discriminative+reranking%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28pp%2E+1) [our card](https://md.mova.elamur.ai/free/en/docs/papers/narrative-2005-charniak-coarse-fine-n-best-parsing/)
2. Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., & Weischedel, R. (2006). Ontonotes. The 90% solution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (pp. 57–60). New York City, USA. [find](https://scholar.google.com/scholar?q=Hovy%2C+E%2E%2C+Marcus%2C+M%2E%2C+Palmer%2C+M%2E%2C+Ramshaw%2C+L%2E%2C+%26+Weischedel%2C+R%2E+%282006%29%2E+Ontonotes%2E+The+90%25+solution%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+North+American+Chapter+of+the+Ass)
3. Kaplan, R. M. (2005). A method for tokenizing text. Festschrift for Kimmo Koskenniemi on his 60th birthday. In A. Arppe, L. Carlson, K. Lindén, J. Piitulainen, M. Suominen, M. Vainio, H. Westerlund, & A. Yli-Jyrä (Eds.), Inquiries into words, constraints and contexts (pp. 55 – 64). Stanford, CA: CSL… [find](https://scholar.google.com/scholar?q=Kaplan%2C+R%2E+M%2E+%282005%29%2E+A+method+for+tokenizing+text%2E+Festschrift+for+Kimmo+Koskenniemi+on+his+60th+birthday%2E+In+A%2E+Arppe%2C+L%2E+Carlson%2C+K%2E+Lind%C3%A9n%2C+J%2E+Piitulainen%2C+M%2E+Suominen%2C+M%2E+Vainio%2C+H%2E+Westerlund%2C+%26)
4. Levenshtein, V. (1966). Binary codes capable of correcting deletions, insertions and reversals. Soviet Physice – Doklady, 10, 707–710. [find](https://scholar.google.com/scholar?q=Levenshtein%2C+V%2E+%281966%29%2E+Binary+codes+capable+of+correcting+deletions%2C+insertions+and+reversals%2E+Soviet+Physice+%E2%80%93+Doklady%2C+10%2C+707%E2%80%93710%2E)
5. Marcus, M. P., Santorini, B., & Marcinkiewicz, M. A. (1993). Building a large annotated corpus of English. The Penn Treebank. Computational Linguistics, 19, 313 – 330. [find](https://scholar.google.com/scholar?q=Marcus%2C+M%2E+P%2E%2C+Santorini%2C+B%2E%2C+%26+Marcinkiewicz%2C+M%2E+A%2E+%281993%29%2E+Building+a+large+annotated+corpus+of+English%2E+The+Penn+Treebank%2E+Computational+Linguistics%2C+19%2C+313+%E2%80%93+330%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
6. Øvrelid, L., Velldal, E., & Oepen, S. (2010). Syntactic scope resolution in uncertainty analysis. In Proceedings of the 23rd international conference on computational linguistics (pp. 1379 – 1387). Beijing, China. [find](https://scholar.google.com/scholar?q=%C3%98vrelid%2C+L%2E%2C+Velldal%2C+E%2E%2C+%26+Oepen%2C+S%2E+%282010%29%2E+Syntactic+scope+resolution+in+uncertainty+analysis%2E+In+Proceedings+of+the+23rd+international+conference+on+computational+linguistics+%28pp%2E+1379+%E2%80%93+1387%29%2E+Be)
7. Schäfer, U., Kiefer, B., Spurk, C., Steffen, J., & Wang, R. (2011). The ACL Anthology Searchbench. In Proceedings of the ACL-HLT 2011 system demonstrations (pp. 7–13). Portland, Oregon, USA. [find](https://scholar.google.com/scholar?q=Sch%C3%A4fer%2C+U%2E%2C+Kiefer%2C+B%2E%2C+Spurk%2C+C%2E%2C+Steffen%2C+J%2E%2C+%26+Wang%2C+R%2E+%282011%29%2E+The+ACL+Anthology+Searchbench%2E+In+Proceedings+of+the+ACL%2DHLT+2011+system+demonstrations+%28pp%2E+7%E2%80%9313%29%2E+Portland%2C+Oregon%2C+USA%2E)
8. Tateisi, Y., & Tsujii, J. (2006). GENIA annotation guidelines for tokenization and POS tagging (Technical Report \# TR-NLP-UT-2006-4). Tokyo, Japan: Tsujii Lab, University of Tokyo. [find](https://scholar.google.com/scholar?q=Tateisi%2C+Y%2E%2C+%26+Tsujii%2C+J%2E+%282006%29%2E+GENIA+annotation+guidelines+for+tokenization+and+POS+tagging+%28Technical+Report+%5C%23+TR%2DNLP%2DUT%2D2006%2D4%29%2E+Tokyo%2C+Japan%3A+Tsujii+Lab%2C+University+of+Tokyo%2E)
9. Waldron, B., Copestake, A., Schäfer, U., & Kiefer, B. (2006). Preprocessing and tokenisation standards in DELPH-IN tools. In Proceedings of the 5th International Conference on Language Resources and Evaluation (pp. 2263 – 2268). Genoa, Italy. [find](https://scholar.google.com/scholar?q=Waldron%2C+B%2E%2C+Copestake%2C+A%2E%2C+Sch%C3%A4fer%2C+U%2E%2C+%26+Kiefer%2C+B%2E+%282006%29%2E+Preprocessing+and+tokenisation+standards+in+DELPH%2DIN+tools%2E+In+Proceedings+of+the+5th+International+Conference+on+Language+Resources+and+E)