[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-1992-webster-tokenization-as-initial-phase-nlp/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Tokenization as the Initial Phase in NLP

# Tokenization as the Initial Phase in NLP

**Authors:** Jonathan J. Webster, [Chunyu Kit](https://md.mova.elamur.ai/free/en/docs/papers/authors/chunyu-kit/) · **Year:** 1992 · **Venue:** COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics **Link:** [aclanthology.org/C92-4173](https://aclanthology.org/C92-4173/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**378** citations](https://openalex.org/works?filter=cites:W2037450062) openalex · 2026-10-18[**19** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-1992-webster-tokenization-as-initial-phase-nlp/) of the 140673 papers whose bibliographies we read[COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics](https://aclanthology.org/volumes/C92-4/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/C92-4173/)
- [PDF](https://aclanthology.org/C92-4173.pdf)
- [DOI 10.3115/992424.992434](https://doi.org/10.3115/992424.992434)
- [OpenAlex](https://openalex.org/W2037450062)

**Authors (2)**

1. [Jonathan J. Webster](https://openalex.org/A5031246043) h 17 · 1890 citations · 124 works
2. [Chunyu Kit](https://openalex.org/A5069506463) [ORCID](https://orcid.org/0000-0002-6445-7400) · h 22 · 1937 citations · 107 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The paper argues that tokenization, the first step of NLP, is a real problem with its own definitions. It defines the notions of word and token from two points of view, that of the lexicographer and that of a working implementation. Automatic segmentation of Chinese words serves as the main illustration. For English, the authors develop practical methods to identify compound tokens, such as idioms, phrasal verbs and fixed expressions, so that each of them is read as one unit. The paper gives no evaluation table, so its value for Mova is in the definitions and in the list of compound types.

## Ideas

- I1. Keep UD tokens as they are, but mark compound units (phrasal verbs, fixed expressions such as *in spite of*) as spans from a list in the in-code dictionary, and pass the span to the parser as a feature; measure on EWT test the LAS of `fixed`, `compound:prt` and `flat` arcs with `en parse-eval`, with and without the span feature. Joining them into single tokens would break UD tokenization, so the span is the safe form of the paper's idea.
- I2. Write the definition of a token used in Mova as a short rule list in [19. The lexicon: vocabulary, multiword expressions and register](https://md.mova.elamur.ai/free/en/docs/languages/en/19-lexicon-mwe-register/), and check it against the EWT tokens that the paper's criteria would split or join.
- I3. A count probe: list the candidate compound tokens in EWT test that the dictionary covers, and report how many of them the tokenizer currently splits. This gives the size of the problem before any change.

## For Mova

The code is `en/src/tok.rs` and `en/src/dict.rs`, with the command `en tok-eval`. The chapter is [19. The lexicon: vocabulary, multiword expressions and register](https://md.mova.elamur.ai/free/en/docs/languages/en/19-lexicon-mwe-register/). The paper's method is for Chinese and for an early lexicon, so only the definitions and the idea of compound tokens carry over. Measure first on EWT test, with the count of joined tokens and the exact-match rate, and check that the result is the same in two runs.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/992424.992434](https://doi.org/10.3115/992424.992434)

```
@inproceedings{webster1992tokenization,
  title = {{Tokenization as the Initial Phase in NLP}},
  author = {Webster, Jonathan J. and Kit, Chunyu},
  booktitle = {COLING 1992 Volume 4: The 14th International Conference on Computational Linguistics},
  year = {1992},
  doi = {10.3115/992424.992434},
  url = {https://doi.org/10.3115/992424.992434}
}
```

- [← previous We Understand Elliptical Sentences, and Language Models should Too: A New Dataset for Studying Ellipsis and its Interaction with Thematic Fit](https://md.mova.elamur.ai/free/en/docs/papers/acl-2023-testa-we-understand-elliptical-sentences-language/)
- [next → Adaptive Sentence Boundary Disambiguation](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-1994-palmer-adaptive-sentence-boundary-disambiguation/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)