[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2007-habash-arabic-diacritization-through-full-morphological/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Semitic languages](https://md.mova.elamur.ai/free/en/docs/papers/sem/) / Arabic Diacritization through Full Morphological Tagging

# Arabic Diacritization through Full Morphological Tagging

**Authors:** [Nizar Habash](https://md.mova.elamur.ai/free/en/docs/papers/authors/nizar-habash/); [Owen Rambow](https://md.mova.elamur.ai/free/en/docs/papers/authors/owen-rambow/) · **Year:** 2007 · **Venue:** Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers (pp. 53–56) **Link:** [aclanthology.org/N07-2014](https://aclanthology.org/N07-2014/) · library `aclbulk-2007-habash-arabic-diacritization-through-full-morphological` · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, before 2016) — summary only

[**143** citations](https://openalex.org/works?filter=cites:W2109613320) openalex · 2026-10-18[**42** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2007-habash-arabic-diacritization-through-full-morphological/) of the 140673 papers whose bibliographies we read[Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers](https://aclanthology.org/volumes/N07-2/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/N07-2014/)
- [PDF](https://aclanthology.org/N07-2014.pdf)
- [DOI 10.3115/1614108.1614122](https://doi.org/10.3115/1614108.1614122)
- [OpenAlex](https://openalex.org/W2109613320)

**Authors (2)**

1. [Nizar Habash](https://openalex.org/A5084517393) [ORCID](https://orcid.org/0000-0002-1831-3457) · h 57 · 12233 citations · 359 works
2. [Owen Rambow](https://openalex.org/A5021314411) [ORCID](https://orcid.org/0000-0003-2054-039X) · h 46 · 9290 citations · 420 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The authors restore diacritics by choosing a whole diacritized analysis for each white-space token, not by labelling characters. The candidate analyses come from the Buckwalter analyzer; a set of taggers, one per morphological feature (including case and mood), score them; the best-scoring analysis is kept, and remaining ties are broken by n-gram models over fully diacritized word forms or over lexeme citation forms (trigrams over lemmas act as a rough word-sense step). The training data is ATB3-Train (288,000 words) with the devtest of 52,000 words used as the only held-out set, to compare with Zitouni et al. (2006) on their metric. The best system (tagger plus trigram lexeme model) gets a WER of 14.9% and a DER of 4.8% over all diacritics, and a WER of 5.5% and DER of 2.2% when the final letter is ignored. The lexeme model alone is weak; it helps only together with the tagger. The authors report that their system beats the Zitouni et al. figures in every column.

## Ideas

- I1. Choose among whole candidate pointings, not among single marks: for an unpointed Hebrew token, the candidates produced by `tools/hbo` (segmenter and parser output) are scored and the best one kept, so that one wrong vowel does not split the word into many errors. The paper's WER-against-DER gap shows why this matters.
- I2. Report the last letter apart: the paper's second column drops the final letter, where case and mood sit, and the gap is large (WER 14.9% against 5.5%). For Hebrew, add the same column for the final vowel and the matres, so that the finals of verbs and nouns are measured on their own in `tools/hbo`.
- I3. A deterministic tie-break by counting: the paper's first method counts how many predicted feature values each candidate matches; this is a rule with no learning and it is a fair baseline. For Mova it is the first version of the choice step in `tools/hbo` before any learned scorer, and its numbers are the reference the SLM has to beat.

## For Mova

I1 and I3 are design points for the unpointed mode of `tools/hbo`. I2 is a measure for the same tool, reported next to WER and DER. The lexeme trigram of the paper is a candidate for the lemma tables of `treebanks/hbo/ptnk`, to be tested on dev only; the tie-break idea also fits [1. Script and pointing](https://md.mova.elamur.ai/free/en/docs/languages/hbo/01-script/), where the rules for pointing are written.

Topics: [Part-of-speech tagging](https://md.mova.elamur.ai/free/en/docs/papers/topics/part-of-speech-tagging/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/1614108.1614122](https://doi.org/10.3115/1614108.1614122)

```
@inproceedings{habash2007arabic,
  title = {{Arabic Diacritization through Full Morphological Tagging}},
  author = {Habash, Nizar and Rambow, Owen},
  booktitle = {Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers},
  year = {2007},
  doi = {10.3115/1614108.1614122},
  url = {https://doi.org/10.3115/1614108.1614122}
}
```

- [← previous Maximum Entropy Based Restoration of Arabic Diacritics](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-zitouni-maximum-entropy-based-restoration-arabic/)
- [next → ElixirFM – Implementation of Functional Arabic Morphology](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2007-smrz-elixirfm-implementation-functional-arabic-morphology/)
- [↑ back to Semitic languages](https://md.mova.elamur.ai/free/en/docs/papers/sem/)

## References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Ananthakrishnan, S. Narayanan, and S. Bangalore. 2005. Automatic diacritization of arabic transcripts for asr. In Proceedings of ICON-05, Kanpur, India. [find](https://scholar.google.com/scholar?q=S%2E+Ananthakrishnan%2C+S%2E+Narayanan%2C+and+S%2E+Bangalore%2E+2005%2E+Automatic+diacritization+of+arabic+transcripts+for+asr%2E+In+Proceedings+of+ICON%2D05%2C+Kanpur%2C+India%2E)
2. T. Buckwalter. 2004. Buckwalter Arabic morphological analyzer version 2.0. [find](https://scholar.google.com/scholar?q=T%2E+Buckwalter%2E+2004%2E+Buckwalter+Arabic+morphological+analyzer+version+2%2E0%2E)
3. J. Giménez and L. Màrquez. 2004. Svmtool: A general pos tagger generator based on support vector machines. In Proceedings of LREC’04. [find](https://scholar.google.com/scholar?q=J%2E+Gime%CC%81nez+and+L%2E+Ma%CC%80rquez%2E+2004%2E+Svmtool%3A+A+general+pos+tagger+generator+based+on+support+vector+machines%2E+In+Proceedings+of+LREC%E2%80%9904%2E)
4. N. Habash and O. Rambow. 2005. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of (ACL’05). [find](https://scholar.google.com/scholar?q=N%2E+Habash+and+O%2E+Rambow%2E+2005%2E+Arabic+tokenization%2C+part%2Dof%2Dspeech+tagging+and+morphological+disambiguation+in+one+fell+swoop%2E+In+Proceedings+of+%28ACL%E2%80%9905%29%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-habash-arabic-tokenization-part-speech-tagging/)
5. N. Habash and O. Rambow. 2007. Arabic Diacritization through Full Morphological Tagging: A Detailed Discussion. Techncial Report, Center for Computational Learning Systems, Columbia University. [find](https://scholar.google.com/scholar?q=N%2E+Habash+and+O%2E+Rambow%2E+2007%2E+Arabic+Diacritization+through+Full+Morphological+Tagging%3A+A+Detailed+Discussion%2E+Techncial+Report%2C+Center+for+Computational+Learning+Systems%2C+Columbia+University%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2007-habash-arabic-diacritization-through-full-morphological/)
6. Jan Hajič, Otakar Smrž, Tim Buckwalter, and Hubert Jin. 2005. Feature-based tagger of approximations of functional Arabic morphology. In Proceedings of the Workshop on Treebanks and Linguistic Theories (TLT), Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Jan+Hajic%CC%8C%2C+Otakar+Smrz%CC%8C%2C+Tim+Buckwalter%2C+and+Hubert+Jin%2E+2005%2E+Feature%2Dbased+tagger+of+approximations+of+functional+Arabic+morphology%2E+In+Proceedings+of+the+Workshop+on+Treebanks+and+Linguistic+Theor)
7. Jan Hajič. 2000. Morphological tagging: Data vs. dictionaries. In Proceedings of (NAACL’00). [find](https://scholar.google.com/scholar?q=Jan+Hajic%CC%8C%2E+2000%2E+Morphological+tagging%3A+Data+vs%2E+dictionaries%2E+In+Proceedings+of+%28NAACL%E2%80%9900%29%2E)
8. Taku Kudo and Yuji Matsumoto. 2003. Fast methods for kernel-based text analysis. In Proceedings of (ACL’03). [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+Yuji+Matsumoto%2E+2003%2E+Fast+methods+for+kernel%2Dbased+text+analysis%2E+In+Proceedings+of+%28ACL%E2%80%9903%29%2E)
9. Mohamed Maamouri, Ann Bies, and Tim Buckwalter. 2004. The Penn Arabic Treebank: Building a large-scale annotated arabic corpus. In NEMLAR Conference on Arabic Language Resources and Tools, Cairo, Egypt. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Ann+Bies%2C+and+Tim+Buckwalter%2E+2004%2E+The+Penn+Arabic+Treebank%3A+Building+a+large%2Dscale+annotated+arabic+corpus%2E+In+NEMLAR+Conference+on+Arabic+Language+Resources+and+Tools%2C+Cairo%2C+Egyp)
10. Mohamed Maamouri, Ann Bies, and Seth Kulick. 2006. Diacritization: A challenge to arabic treebank annotation and parsing. In Proceedings of the Conference of the Machine Translation SIG of the British Computer Society. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Ann+Bies%2C+and+Seth+Kulick%2E+2006%2E+Diacritization%3A+A+challenge+to+arabic+treebank+annotation+and+parsing%2E+In+Proceedings+of+the+Conference+of+the+Machine+Translation+SIG+of+the+British)
11. Noah A. Smith, David A. Smith, and Roy W. Tromble. 2005. Context-based morphological disambiguation with random fields. In Proceedings of (EMNLP05). [find](https://scholar.google.com/scholar?q=Noah+A%2E+Smith%2C+David+A%2E+Smith%2C+and+Roy+W%2E+Tromble%2E+2005%2E+Context%2Dbased+morphological+disambiguation+with+random+fields%2E+In+Proceedings+of+%28EMNLP05%29%2E)
12. Andreas Stolcke. 2002. SRILM - an Extensible Language Modeling Toolkit. In Proceedings of the International Conference on Spoken Language Processing (ICSLP). [find](https://scholar.google.com/scholar?q=Andreas+Stolcke%2E+2002%2E+SRILM+%2D+an+Extensible+Language+Modeling+Toolkit%2E+In+Proceedings+of+the+International+Conference+on+Spoken+Language+Processing+%28ICSLP%29%2E)
13. Dimitra Vergyri and Katrin Kirchhoff. 2004. Automatic diacritization of arabic for acoustic modeling in speech recognition. In COLING 2004 Workshop on Computational Approaches to Arabic Script-based Languages, Geneva, Switzerland. [find](https://scholar.google.com/scholar?q=Dimitra+Vergyri+and+Katrin+Kirchhoff%2E+2004%2E+Automatic+diacritization+of+arabic+for+acoustic+modeling+in+speech+recognition%2E+In+COLING+2004+Workshop+on+Computational+Approaches+to+Arabic+Script%2Dbased+L)
14. Imed Zitouni, Jeffrey S. Sorensen, and Ruhi Sarikaya. 2006. Maximum entropy based restoration of arabic diacritics. In Proceedings of ACL’06. [find](https://scholar.google.com/scholar?q=Imed+Zitouni%2C+Jeffrey+S%2E+Sorensen%2C+and+Ruhi+Sarikaya%2E+2006%2E+Maximum+entropy+based+restoration+of+arabic+diacritics%2E+In+Proceedings+of+ACL%E2%80%9906%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-zitouni-maximum-entropy-based-restoration-arabic/)