[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2005-nelken-arabic-diacritization-weighted-finite-state/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Semitic languages](https://md.mova.elamur.ai/free/en/docs/papers/sem/) / Arabic Diacritization Using Weighted Finite-State Transducers

# Arabic Diacritization Using Weighted Finite-State Transducers

**Authors:** Rani Nelken; [Stuart M. Shieber](https://md.mova.elamur.ai/free/en/docs/papers/authors/stuart-m-shieber/) · **Year:** 2005 · **Venue:** Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, pp. 79–86 **Link:** [aclanthology.org/W05-0711](https://aclanthology.org/W05-0711/) · library `aclbulk-2005-nelken-arabic-diacritization-weighted-finite-state` · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, before 2016) — summary only

[**90** citations](https://openalex.org/works?filter=cites:W2041349462) openalex-snapshot · 2026-10-14[**26** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2005-nelken-arabic-diacritization-weighted-finite-state/) of the 140673 papers whose bibliographies we read[Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages](https://aclanthology.org/volumes/W05-07/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/W05-0711/)
- [PDF](https://aclanthology.org/W05-0711.pdf)
- [DOI 10.3115/1621787.1621802](https://doi.org/10.3115/1621787.1621802)
- [OpenAlex](https://openalex.org/W2041349462)

**Authors (2)**

1. [Rani Nelken](https://openalex.org/A5079038949) h 11 · 400 citations · 32 works
2. [Stuart M. Shieber](https://openalex.org/A5053102850) [ORCID](https://orcid.org/0000-0002-7733-8195) · h 50 · 10398 citations · 191 works

Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart.

## What it is about

The paper restores short vowels and other diacritics (shadda, sukun, tanween, hamza forms) to undiacritized Modern Standard Arabic with a cascade of weighted finite-state transducers, modelling the loss of diacritics generatively and decoding with Viterbi. The cascade is: a word trigram language model with Katz backoff over diacritized words; a spelling transducer from words to letters; a diacritic-drop transducer, which also lists the alternative written forms of hamza without rules for when each occurs; a clitic-concatenation transducer that can join a fixed list of prefixes (b, l, k, w, f, Al, s) and pronoun suffixes to a word, with no knowledge of legal orders; and an unknown-word transducer. Unknown words are then diacritized by a 4-gram letter model composed with the drop transducer. Training and testing use the Arabic Treebank (Al-Hayat, 90/10 random split, about 14,000 test words). Without case endings, the unigram baseline gives WER 15.48% and DER 17.33%; the trigram model alone helps little (14.64/16.9); adding the clitic model gives 8.49/9.32; adding the letter model gives 7.33/6.35. With case endings the best is WER 23.61%, DER 12.79%. The text extraction of this PDF failed (font-encoded Arabic), so the card was written from the page images.

## Ideas

- I1. The whole method is deterministic once trained and needs very little morphology: count-based word n-grams, a clitic list with no ordering rules, and a letter n-gram model only for unknown words. For unpointed Hebrew this is a direct baseline for vowel restoration in `tools/hbo`: word trigram over pointed forms, a prefix/suffix transducer from the segmenter's tables, and a letter model as fallback — measured with the same two numbers (WER, DER).
- I2. The ablation shape is the lesson: the clitic model gives the largest single gain (WER 14.64 → 8.49), larger than the language model itself, because a clitic-split corpus has far fewer types. Measure the same step for Hebrew (pointing with and without the prefix split) before adding anything else.
- I3. Use the letter model selectively, only on words unknown to the word model; the authors found that better than running it on everything. A two-tier rule (lexicon first, letter model second, with the tier recorded) keeps each pointing explainable.
- I4. Report results with and without the final (case/mood) vowel, since the gap is large; for Hebrew the analogue is the word-final vowel and the paragogic endings.

## For Mova

I1–I3 apply to the unpointed mode of `tools/hbo` and to [1. Script and pointing](https://md.mova.elamur.ai/free/en/docs/languages/hbo/01-script/); I4 is a reporting rule for the same measurement. Gal's HMM (card `aclbulk-2002-gal-hmm-approach-vowel-restoration-arabic`) is the nearest relative and the baseline to compare against.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/1621787.1621802](https://doi.org/10.3115/1621787.1621802)

```
@inproceedings{nelken2005arabic,
  title = {{Arabic Diacritization Using Weighted Finite-State Transducers}},
  author = {Nelken, Rani and Shieber, Stuart M.},
  booktitle = {Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages},
  year = {2005},
  doi = {10.3115/1621787.1621802},
  url = {https://doi.org/10.3115/1621787.1621802}
}
```

- [← previous Memory-Based Morphological Analysis Generation and Part-of-Speech Tagging of Arabic](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-marsi-memory-based-morphological-analysis-generation/)
- [next → Parsing Arabic Dialects](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-chiang-parsing-arabic-dialects/)
- [↑ back to Semitic languages](https://md.mova.elamur.ai/free/en/docs/papers/sem/)