[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2008-roth-arabic-morphological-tagging-diacritization-lemmatization/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Semitic languages](https://md.mova.elamur.ai/free/en/docs/papers/sem/) / Arabic Morphological Tagging, Diacritization, and Lemmatization Using Lexeme Models and Feature Ranking

# Arabic Morphological Tagging, Diacritization, and Lemmatization Using Lexeme Models and Feature Ranking

**Authors:** [Ryan Roth](https://md.mova.elamur.ai/free/en/docs/papers/authors/ryan-roth/); [Owen Rambow](https://md.mova.elamur.ai/free/en/docs/papers/authors/owen-rambow/); [Nizar Habash](https://md.mova.elamur.ai/free/en/docs/papers/authors/nizar-habash/); [Mona Diab](https://md.mova.elamur.ai/free/en/docs/papers/authors/mona-diab/); [Cynthia Rudin](https://md.mova.elamur.ai/free/en/docs/papers/authors/cynthia-rudin/) · **Year:** 2008 · **Venue:** Proceedings of ACL-08: HLT, Short Papers **Link:** [aclanthology.org/P08-2030](https://aclanthology.org/P08-2030/) · library `aclbulk-2008-roth-arabic-morphological-tagging-diacritization-lemmatization` · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, before 2016) — summary only

[**155** citations](https://openalex.org/works?filter=cites:W2126784811) openalex · 2026-10-18[**62** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2008-roth-arabic-morphological-tagging-diacritization-lemmatization/) of the 140673 papers whose bibliographies we read[Proceedings of ACL-08: HLT, Short Papers](https://aclanthology.org/volumes/P08-2/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/P08-2030/)
- [PDF](https://aclanthology.org/P08-2030.pdf)
- [DOI 10.3115/1557690.1557721](https://doi.org/10.3115/1557690.1557721)
- [OpenAlex](https://openalex.org/W2126784811)

**Authors (5)**

1. [Ryan Roth](https://openalex.org/A5111429160) h 13 · 856 citations · 35 works
2. [Owen Rambow](https://openalex.org/A5021314411) [ORCID](https://orcid.org/0000-0003-2054-039X) · h 46 · 9290 citations · 420 works
3. [Nizar Habash](https://openalex.org/A5084517393) [ORCID](https://orcid.org/0000-0002-1831-3457) · h 57 · 12233 citations · 359 works
4. [Mona Diab](https://openalex.org/A5038581447) [ORCID](https://orcid.org/0000-0002-7696-1436) · h 48 · 11047 citations · 293 works
5. [Cynthia Rudin](https://openalex.org/A5040468715) [ORCID](https://orcid.org/0000-0003-4283-2780) · h 58 · 26033 citations · 492 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

MADA picks, for each Arabic word, one analysis from the candidates that the BAMA dictionary proposes. It scores each candidate by summing the weights of the features on which the candidate agrees with the model's prediction. The features are 14 morphological tags, each predicted by an SVM, two checks (spelling match, and whether the analysis is the dictionary default), a unigram model of diacritized forms and of lexemes, and a 4-gram lexeme model. The authors add lemmatization as a task and run seven tasks (tagging, diacritization, lemma choice, and full choice). The weights are tuned by downhill simplex on half of ATB3-DevTest (about 26K words), with the other half (about 25K words) held out for testing. Lexeme features cut error by 36–71% on the diacritic and lemma tasks, and by 3–11% on the morphological tasks. Tuning the weights reduces error further, by 8–20% with all features.

## Ideas

- I1. A lexeme n-gram as one vote: a 4-gram model over lemma sequences, which alone picks the right lexeme in 94.1% of dev-test words, is added as one more feature of the candidate score. In Mova the same role can be played by a lemma n-gram table over the segmenter's candidate analyses, built with `BTreeMap` so the result is deterministic.
- I2. Score candidates as a plain sum of named feature weights, and tune the weights on a split of the training data, one tuning objective per evaluation task. Measure with the two-fold `cv` check, not the final test, so that the choice of weights is never made on the held-out set.
- I3. Use the agreement-based score as an explanation. Each decision can be printed as the list of features that voted for the winner, which fits Mova's requirement that every step can say why. For unpointed Hebrew this is the diacritic decision of `tools/hbo` unpointed mode.

## For Mova

I1 goes into the candidate choice of `tools/hbo` (unpointed mode), where a lexeme table would vote on the segmenter's analyses. I2 is the test design for the same choice (`cv` split, no peeking at the release test). I3 is the explanation output of that choice. The Arabic data itself belongs to `treebanks/ar`, and the spelling-match idea is relevant to [1. Script and pointing](https://md.mova.elamur.ai/free/en/docs/languages/hbo/01-script/).

Topics: [Part-of-speech tagging](https://md.mova.elamur.ai/free/en/docs/papers/topics/part-of-speech-tagging/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/1557690.1557721](https://doi.org/10.3115/1557690.1557721)

```
@inproceedings{roth2008arabic,
  title = {{Arabic Morphological Tagging, Diacritization, and Lemmatization Using Lexeme Models and Feature Ranking}},
  author = {Roth, Ryan and Rambow, Owen and Habash, Nizar and Diab, Mona and Rudin, Cynthia},
  booktitle = {Proceedings of ACL-08: HLT, Short Papers},
  year = {2008},
  doi = {10.3115/1557690.1557721},
  url = {https://doi.org/10.3115/1557690.1557721}
}
```

- [← previous Construct State Modification in the Arabic Treebank](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2008-gabbard-construct-state-modification-arabic-treebank/)
- [next → The Karamel System and Semitic Languages: Structured Multi-Tiered Morphology](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2009-barthelemy-karamel-system-semitic-languages-structured/)
- [↑ back to Semitic languages](https://md.mova.elamur.ai/free/en/docs/papers/sem/)

## References

8 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tim Buckwalter. 2004. Buckwalter Arabic morphological analyzer version 2.0. [find](https://scholar.google.com/scholar?q=Tim+Buckwalter%2E+2004%2E+Buckwalter+Arabic+morphological+analyzer+version+2%2E0%2E)
2. Nizar Habash and Owen Rambow. 2005. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In ACL’05, Ann Arbor, MI, USA. [find](https://scholar.google.com/scholar?q=Nizar+Habash+and+Owen+Rambow%2E+2005%2E+Arabic+tokenization%2C+part%2Dof%2Dspeech+tagging+and+morphological+disambiguation+in+one+fell+swoop%2E+In+ACL%E2%80%9905%2C+Ann+Arbor%2C+MI%2C+USA%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-habash-arabic-tokenization-part-speech-tagging/)
3. Nizar Habash and Owen Rambow. 2007. Arabic diacritization through full morphological tagging. In NAACL HLT 2007 Companion Volume, Short Papers, Rochester, NY, USA. [find](https://scholar.google.com/scholar?q=Nizar+Habash+and+Owen+Rambow%2E+2007%2E+Arabic+diacritization+through+full+morphological+tagging%2E+In+NAACL+HLT+2007+Companion+Volume%2C+Short+Papers%2C+Rochester%2C+NY%2C+USA%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2007-habash-arabic-diacritization-through-full-morphological/)
4. Jan Hajič, Otakar Smrž, Tim Buckwalter, and Hubert Jin. 2005. Feature-based tagger of approximations of functional Arabic morphology. In Proceedings of the Workshop on Treebanks and Linguistic Theories (TLT), Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Jan+Hajic%CC%8C%2C+Otakar+Smrz%CC%8C%2C+Tim+Buckwalter%2C+and+Hubert+Jin%2E+2005%2E+Feature%2Dbased+tagger+of+approximations+of+functional+Arabic+morphology%2E+In+Proceedings+of+the+Workshop+on+Treebanks+and+Linguistic+Theor)
5. Jan Hajič. 2000. Morphological tagging: Data vs. dictionaries. In 1st Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL’00), Seattle, WA. [find](https://scholar.google.com/scholar?q=Jan+Hajic%CC%8C%2E+2000%2E+Morphological+tagging%3A+Data+vs%2E+dictionaries%2E+In+1st+Meeting+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28NAACL%E2%80%9900%29%2C+Seattle%2C+WA%2E)
6. J.A Nelder and R Mead. 1965. A simplex method for function minimization. In Computer Journal, pages 303–333. [find](https://scholar.google.com/scholar?q=J%2EA+Nelder+and+R+Mead%2E+1965%2E+A+simplex+method+for+function+minimization%2E+In+Computer+Journal%2C+pages+303%E2%80%93333%2E)
7. Andreas Stolcke. 2002. Srilm - an extensible language toolkit. In Proceedings of the International Conference on Spoken Language Processing (ICSLP). [find](https://scholar.google.com/scholar?q=Andreas+Stolcke%2E+2002%2E+Srilm+%2D+an+extensible+language+toolkit%2E+In+Proceedings+of+the+International+Conference+on+Spoken+Language+Processing+%28ICSLP%29%2E)
8. Imed Zitouni, Jeffrey S. Sorensen, and Ruhi Sarikaya. 2006. Maximum entropy based restoration of arabic diacritics. In Coling-ACL’06, pages 577–584, Sydney, Australia. [find](https://scholar.google.com/scholar?q=Imed+Zitouni%2C+Jeffrey+S%2E+Sorensen%2C+and+Ruhi+Sarikaya%2E+2006%2E+Maximum+entropy+based+restoration+of+arabic+diacritics%2E+In+Coling%2DACL%E2%80%9906%2C+pages+577%E2%80%93584%2C+Sydney%2C+Australia%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-zitouni-maximum-entropy-based-restoration-arabic/)