[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2017-darwish-arabic-diacritization-stats-rules-hacks/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Semitic languages](https://md.mova.elamur.ai/free/en/docs/papers/sem/) / Arabic Diacritization: Stats, Rules, and Hacks

# Arabic Diacritization: Stats, Rules, and Hacks

**Authors:** [Kareem Darwish](https://md.mova.elamur.ai/free/en/docs/papers/authors/kareem-darwish/); [Hamdy Mubarak](https://md.mova.elamur.ai/free/en/docs/papers/authors/hamdy-mubarak/); [Ahmed Abdelali](https://md.mova.elamur.ai/free/en/docs/papers/authors/ahmed-abdelali/) · **Year:** 2017 · **Venue:** Proceedings of the Third Arabic Natural Language Processing Workshop **Link:** [aclanthology.org/W17-1302](https://aclanthology.org/W17-1302/) · library `aclbulk-2017-darwish-arabic-diacritization-stats-rules-hacks` · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology)

[**85** citations](https://openalex.org/works?filter=cites:W2740272870) openalex · 2026-10-19[**30** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2017-darwish-arabic-diacritization-stats-rules-hacks/) of the 140673 papers whose bibliographies we read[Proceedings of the Third Arabic Natural Language Processing Workshop](https://aclanthology.org/volumes/W17-13/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/W17-1302/)
- [PDF](https://aclanthology.org/W17-1302.pdf)
- [DOI 10.18653/v1/w17-1302](https://doi.org/10.18653/v1/w17-1302)
- [OpenAlex](https://openalex.org/W2740272870)

**Authors (3)**

1. [Kareem Darwish](https://openalex.org/A5111452036) [Semantic Scholar](https://www.semanticscholar.org/author/143758717) · h 38 · 4980 citations · 151 works
2. [Hamdy Mubarak](https://openalex.org/A5102924560) [ORCID](https://orcid.org/0000-0002-9051-6240) · [Semantic Scholar](https://www.semanticscholar.org/author/143779235) · h 30 · 3926 citations · 113 works
3. [Ahmed Abdelali](https://openalex.org/A5047182137) [ORCID](https://orcid.org/0000-0002-4160-8181) · [Semantic Scholar](https://www.semanticscholar.org/author/1683403) · h 25 · 2329 citations · 137 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

The diacritizer works in two cascaded steps. First it restores the vowels of the word core (the word without its case ending): a word-level bigram model with a Viterbi decoder, backed off to stem dictionaries, stem templates derived from roots (computed with Farasa), a transliteration-mined model for foreign names (a CRF trained on 500 names), and a dictionary of words that have one dominant diacritized form (at least 10 occurrences, more than 90% of them the same). Second it guesses the case ending with an SVM ranker over lexical, morphological and POS features, with filtering rules (a verb restricts the case to a few values, a dual suffix to one value, a DET prefix forbids tanween, a template restricts the endings seen in training). The corpus is a commercial diacritized corpus of 9.7 million tokens. On 70 WikiNews articles (about 18,300 words), the full system reaches 3.29% word error without case endings and 12.76% with them. MADAMIRA gets 6.73% and 19.02% on the same two measures, and Belinkov and Glass 14.87% on the core words and 30.50% with case endings. The system is written in Java, free for research, and needs 2.2 GB of memory.

## Ideas

- I1. A back-off ladder for unpointed words in `tools/hbo`: a fully pointed form first, then the stem and its vowels, then the root-and-pattern template, then a default. Each rung is a table in Rust, and the paper's table of gains per rung is the template for measuring each rung on a held-out Hebrew set.
- I2. Rules that filter case-like candidates before ranking: a construct noun cannot carry the article, a word with the article takes no construct ending, and a verb restricts the final vowel. For Hebrew the same logic restricts the final vowel of a noun in the construct state and forbids the article on construct heads. Put these as filters on candidate vowel patterns, with the count of how many candidates each rule removes.
- I3. A transliteration check for foreign names: a name written with English vowel letters implies its vowels in Arabic script. For Hebrew names with matres lectionis (vav, yod), a rule of this kind can guess the vowels. The paper measured it on a 200-name sample (79% correct), which is the bar a Hebrew version should meet before it is used.

## For Mova

I1 goes into the unpointed mode of `tools/hbo`, and the table of gains per rung becomes its measurement. I2 goes into [The Noun, the Adjective and the Numeral](https://md.mova.elamur.ai/free/en/docs/languages/hbo/04-noun/) (the construct state and the article) as candidate filters. I3 goes into [3. Phonology](https://md.mova.elamur.ai/free/en/docs/languages/he/03-phonology/) and the named-entity handling of `tools/hbo`, measured on a 200-name sample before use.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/w17-1302](https://doi.org/10.18653/v1/w17-1302)

```
@inproceedings{darwish2017arabic,
  title = {{Arabic Diacritization: Stats, Rules, and Hacks}},
  author = {Darwish, Kareem and Mubarak, Hamdy and Abdelali, Ahmed},
  booktitle = {Proceedings of the Third Arabic Natural Language Processing Workshop},
  year = {2017},
  doi = {10.18653/v1/w17-1302},
  url = {https://doi.org/10.18653/v1/w17-1302}
}
```

- [← previous Morphological Analysis for the Maltese Language: The challenges of a hybrid system](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2017-borg-morphological-analysis-maltese-language-challenges/)
- [next → Challenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2017-sajjad-challenging-language-dependent-segmentation-arabic/)
- [↑ back to Semitic languages](https://md.mova.elamur.ai/free/en/docs/papers/sem/)

## References

21 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions, pages 177–180. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+et+al%2E+2007%2E+Moses%3A+Open+source+toolkit+fo)
2. Gheith A Abandah, Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015. Automatic diacritization of arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18(2):183–197. [find](https://scholar.google.com/scholar?q=Gheith+A+Abandah%2C+Alex+Graves%2C+Balkees+Al%2DShagoor%2C+Alaa+Arabiyat%2C+Fuad+Jamour%2C+and+Majid+Al%2DTaee%2E+2015%2E+Automatic+diacritization+of+arabic+text+using+recurrent+neural+networks%2E+International+Journal+o)
3. Aqil M Azmi and Reham S Almajed. 2015. A survey of automatic arabic diacritization techniques. Natural Language Engineering, 21(03):477–495. [find](https://scholar.google.com/scholar?q=Aqil+M+Azmi+and+Reham+S+Almajed%2E+2015%2E+A+survey+of+automatic+arabic+diacritization+techniques%2E+Natural+Language+Engineering%2C+21%2803%29%3A477%E2%80%93495%2E)
4. Taku Kudo. 2005. Crf++: Yet another crf toolkit. Software available at [http://crfpp.sourceforge](http://crfpp.sourceforge). net. [↗](http://crfpp.sourceforge)
5. Mohamed Bebah, Chennoufi Amine, Mazroui Azzeddine, and Lakhouaja Abdelhak. 2014. Hybrid approaches for automatic vowelization of arabic texts. arXiv preprint arXiv:1410.2646. [↗](https://arxiv.org/abs/1410.2646)
6. Bebah Mohamed Ould Abdallahi Ould, Abdelouafi Meziane, Azzeddine Mazroui, and Abdelhak Lakhouaja. 2011. Alkhalil morphosys. In 7th International Computing Conference in Arabic, pages 66–73, Riyadh, Saudi Arabia. [find](https://scholar.google.com/scholar?q=Bebah+Mohamed+Ould+Abdallahi+Ould%2C+Abdelouafi+Meziane%2C+Azzeddine+Mazroui%2C+and+Abdelhak+Lakhouaja%2E+2011%2E+Alkhalil+morphosys%2E+In+7th+International+Computing+Conference+in+Arabic%2C+pages+66%E2%80%9373%2C+Riyadh%2C+Sa)
7. Yonatan Belinkov and James Glass. 2015. Arabic diacritization with recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2281– 2285, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+James+Glass%2E+2015%2E+Arabic+diacritization+with+recurrent+neural+networks%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+2281%E2%80%93+228) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2015-belinkov-arabic-diacritization-recurrent-neural-networks/)
8. Rani Nelken and Stuart M Shieber. 2005. Arabic diacritization using weighted finite-state transducers. In Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, pages 79– 86. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rani+Nelken+and+Stuart+M+Shieber%2E+2005%2E+Arabic+diacritization+using+weighted+finite%2Dstate+transducers%2E+In+Proceedings+of+the+ACL+Workshop+on+Computational+Approaches+to+Semitic+Languages%2C+pages+79%E2%80%93+86) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-nelken-arabic-diacritization-weighted-finite-state/)
9. Tim Buckwalter. 2004. Buckwalter arabic morphological analyzer version 2.0. ldc catalog number ldc2004l02, isbn 1-58563-324-0. [find](https://scholar.google.com/scholar?q=Tim+Buckwalter%2E+2004%2E+Buckwalter+arabic+morphological+analyzer+version+2%2E0%2E+ldc+catalog+number+ldc2004l02%2C+isbn+1%2D58563%2D324%2D0%2E)
10. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan M Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC-2014, Reykjavik, Iceland. [find](https://scholar.google.com/scholar?q=Arfath+Pasha%2C+Mohamed+Al%2DBadrashiny%2C+Mona+Diab%2C+Ahmed+El+Kholy%2C+Ramy+Eskander%2C+Nizar+Habash%2C+Manoj+Pooleery%2C+Owen+Rambow%2C+and+Ryan+M+Roth%2E+2014%2E+Madamira%3A+A+fast%2C+comprehensive+tool+for+morphological+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2014-pasha-madamira-fast-comprehensive-tool-morphological/)
11. Andy Cole, David Graff, and Kevin Walker. 2001. Arabic newswire part 1 corpus (1-58563-190-6). Linguistic Data Consortium (LDC). [find](https://scholar.google.com/scholar?q=Andy+Cole%2C+David+Graff%2C+and+Kevin+Walker%2E+2001%2E+Arabic+newswire+part+1+corpus+%281%2D58563%2D190%2D6%29%2E+Linguistic+Data+Consortium+%28LDC%29%2E)
12. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, may. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Kareem+Darwish+and+Hamdy+Mubarak%2E+2016%2E+Farasa%3A+A+new+fast+and+accurate+arabic+word+segmenter%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC+2016%29%2C+Pa)
13. Mohsen Rashwan, Mohammad Al-Badrashiny, Mohamed Attia, and Sherif Abdou. 2009. A hybrid system for automatic arabic diacritization. In The 2nd International Conference on Arabic Language Resources and Tools, pages 54–60. [find](https://scholar.google.com/scholar?q=Mohsen+Rashwan%2C+Mohammad+Al%2DBadrashiny%2C+Mohamed+Attia%2C+and+Sherif+Abdou%2E+2009%2E+A+hybrid+system+for+automatic+arabic+diacritization%2E+In+The+2nd+International+Conference+on+Arabic+Language+Resources+and)
14. Ali El-Kahky, Kareem Darwish, Ahmed Saad Aldein, Mohamed Abd El-Wahab, Ahmed Hefny, and Waleed Ammar. 2011. Improved transliteration mining using graph reinforcement. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1384–1393. [find](https://scholar.google.com/scholar?q=Ali+El%2DKahky%2C+Kareem+Darwish%2C+Ahmed+Saad+Aldein%2C+Mohamed+Abd+El%2DWahab%2C+Ahmed+Hefny%2C+and+Waleed+Ammar%2E+2011%2E+Improved+transliteration+mining+using+graph+reinforcement%2E+In+Proceedings+of+the+Conference+)
15. Mohsen Rashwan, Ahmad Al Sallab, M. Raafat, and Ahmed Rafea. 2015. Deep learning framework with confused sub-set resolution architecture for automatic arabic diacritization. In IEEE Transactions on Audio, Speech, and Language Processing, pages 505–516. [find](https://scholar.google.com/scholar?q=Mohsen+Rashwan%2C+Ahmad+Al+Sallab%2C+M%2E+Raafat%2C+and+Ahmed+Rafea%2E+2015%2E+Deep+learning+framework+with+confused+sub%2Dset+resolution+architecture+for+automatic+arabic+diacritization%2E+In+IEEE+Transactions+on+Au)
16. Ya’akov Gal. 2002. An hmm approach to vowel restoration in arabic and hebrew. In Proceedings of the ACL-02 workshop on Computational approaches to Semitic languages, pages 1–7. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ya%E2%80%99akov+Gal%2E+2002%2E+An+hmm+approach+to+vowel+restoration+in+arabic+and+hebrew%2E+In+Proceedings+of+the+ACL%2D02+workshop+on+Computational+approaches+to+Semitic+languages%2C+pages+1%E2%80%937%2E+Association+for+Computa) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2002-gal-hmm-approach-vowel-restoration-arabic/)
17. Dimitra Vergyri and Katrin Kirchhoff. 2004. Automatic diacritization of arabic for acoustic modeling in speech recognition. In Proceedings of the workshop on computational approaches to Arabic script-based languages, COLING’04, pages 66– 73, Geneva, Switzerland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dimitra+Vergyri+and+Katrin+Kirchhoff%2E+2004%2E+Automatic+diacritization+of+arabic+for+acoustic+modeling+in+speech+recognition%2E+In+Proceedings+of+the+workshop+on+computational+approaches+to+Arabic+script%2D)
18. Nizar Habash and Owen Rambow. 2007. Arabic diacritization through full morphological tagging. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 53–56. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1614108.1614122) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2007-habash-arabic-diacritization-through-full-morphological/)
19. Imed Zitouni, Jeffrey S Sorensen, and Ruhi Sarikaya. 2006. Maximum entropy based restoration of arabic diacritics. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 577–584. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Imed+Zitouni%2C+Jeffrey+S+Sorensen%2C+and+Ruhi+Sarikaya%2E+2006%2E+Maximum+entropy+based+restoration+of+arabic+diacritics%2E+In+Proceedings+of+the+21st+International+Conference+on+Computational+Linguistics+and+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-zitouni-maximum-entropy-based-restoration-arabic/)
20. Nizar Habash, Owen Rambow, and Ryan Roth. 2009. Mada+ tokan: A toolkit for arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization. In Proceedings of the 2nd International Conference on Arabic Language Resources and Tools (MEDAR), Cairo, Egypt, pages 102–109. [find](https://scholar.google.com/scholar?q=Nizar+Habash%2C+Owen+Rambow%2C+and+Ryan+Roth%2E+2009%2E+Mada%2B+tokan%3A+A+toolkit+for+arabic+tokenization%2C+diacritization%2C+morphological+disambiguation%2C+pos+tagging%2C+stemming+and+lemmatization%2E+In+Proceedings+of)
21. Thorsten Joachims. 2006. Training linear svms in linear time. In ACM SIGKDD-2006, pages 217–226. ACM. [find](https://scholar.google.com/scholar?q=Thorsten+Joachims%2E+2006%2E+Training+linear+svms+in+linear+time%2E+In+ACM+SIGKDD%2D2006%2C+pages+217%E2%80%93226%2E+ACM%2E)