[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/lingbulk-2018-mubarak-build-fast-accurate-lemmatization-arabic/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Semitic languages](https://md.mova.elamur.ai/free/en/docs/papers/sem/) / Build Fast and Accurate Lemmatization for Arabic

# Build Fast and Accurate Lemmatization for Arabic

**Authors:** [Hamdy Mubarak](https://md.mova.elamur.ai/free/en/docs/papers/authors/hamdy-mubarak/) · **Year:** 2018 · **Venue:** Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) **Link:** [aclanthology.org/L18-1181](https://aclanthology.org/L18-1181/) · library `lingbulk-2018-mubarak-build-fast-accurate-lemmatization-arabic` · **Licence:** [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (ELRA, LREC proceedings) — summary only

[**7** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/lingbulk-2018-mubarak-build-fast-accurate-lemmatization-arabic/) of the 140673 papers whose bibliographies we read[Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)](https://aclanthology.org/volumes/L18-1/)[licence](https://creativecommons.org/licenses/by-nc/4.0/)

- [Publisher's page](https://aclanthology.org/L18-1181/)
- [PDF](https://aclanthology.org/L18-1181.pdf)

**Authors (1)**

1. Mubarak, Hamdy

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

Arabic lemmatization differs from surface stemming: it must restore the citation form, which can require changing stem patterns, not only cutting affixes. The author builds a fast open-source Java lemmatizer. For each undiacritized word it takes the most frequent diacritized form from a 9.7-million-word corpus, about 73% MSA, and picks the Buckwalter analysis whose diacritized form matches best. Lemma order is fixed by corpus usage where analyses tie. A new test set of 70 WikiNews articles (18,300 words) is lemmatized by an expert and scored on undiacritized lemmas: 73.68% for the Farasa surface stemmer, 96.61% for MADAMIRA and 97.32% for the new system. The paper's text claims +7% relative over MADAMIRA, but the table gap is under one point. Speed: 7.4 million words in 2 minutes, against 2.5 hours for MADAMIRA. The test set is NC, so it is for measurement only.

## Ideas

- I1. For unpointed Hebrew, in `tools/hbo` unpointed mode: when a consonantal form has several analyses, prefer the one whose pointed form is most frequent in a pointed corpus (MACULA gives the pointed forms). It is table-based and needs no context, as the paper does. Where: [The lexicon](https://md.mova.elamur.ai/free/en/docs/languages/hbo/10-lexicon/).
- I2. Order lemmas by usage in the lexicon, not in the code: where two analyses share a form (the paper's car/walker case), the lemma with more corpus occurrences comes first, and the order is stored in `macula dict`.
- I3. Report lemma accuracy on undiacritized text, with an error table in the paper's style (POS error, named-entity segmentation error), as a test for `tools/hbo`. Build the set from our own open data; do not carry the NC set into a release.

## For Mova

I1 goes into `tools/hbo` (unpointed mode) and [The lexicon](https://md.mova.elamur.ai/free/en/docs/languages/hbo/10-lexicon/). I2 goes into `macula dict`. I3 is the test for `tools/hbo`.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{mubarak2018build,
  title = {{Build Fast and Accurate Lemmatization for Arabic}},
  author = {Mubarak, Hamdy},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  url = {https://aclanthology.org/L18-1181/}
}
```

- [← previous Parser combinators for Tigrinya and Oromo morphology](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2018-littell-parser-combinators-tigrinya-oromo-morphology/)
- [next → Rethinking templates: A syntactic analysis of verbal morphology in Emirati Arabic](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2019-kaabi-rethinking-templates-syntactic-analysis-verbal/)
- [↑ back to Semitic languages](https://md.mova.elamur.ai/free/en/docs/papers/sem/)

## References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdelali, A., Darwish, K., Durrani, N., and Mubarak, H. (2016). Farasa: A fast and furious segmenter for arabic. In HLT-NAACL Demos, pages 11–16. [find](https://scholar.google.com/scholar?q=Abdelali%2C+A%2E%2C+Darwish%2C+K%2E%2C+Durrani%2C+N%2E%2C+and+Mubarak%2C+H%2E+%282016%29%2E+Farasa%3A+A+fast+and+furious+segmenter+for+arabic%2E+In+HLT%2DNAACL+Demos%2C+pages+11%E2%80%9316%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2016-abdelali-farasa-fast-furious-segmenter-arabic/)
2. Beesley, K. (1996). Arabic finite-state morphological analysis and generation. In In COLING-96: Proceedings of the 16th international, pages 89–94. [find](https://scholar.google.com/scholar?q=Beesley%2C+K%2E+%281996%29%2E+Arabic+finite%2Dstate+morphological+analysis+and+generation%2E+In+In+COLING%2D96%3A+Proceedings+of+the+16th+international%2C+pages+89%E2%80%9394%2E)
3. Buckwalter, T. (2002). Arabic finite-state morphological analysis and generation. In [http://members.aol.com/ArabicLexicons/](http://members.aol.com/ArabicLexicons/). Figure 3: Summary of lemmatization algorithm Figure 4: Lemmatization online demo (part of Farasa Arabic NLP tools) [↗](http://members.aol.com/ArabicLexicons/)
4. Darwish, K. and Mubarak, H. (2016). Farasa: A new fast and accurate arabic word segmenter. In LREC. [find](https://scholar.google.com/scholar?q=Darwish%2C+K%2E+and+Mubarak%2C+H%2E+%282016%29%2E+Farasa%3A+A+new+fast+and+accurate+arabic+word+segmenter%2E+In+LREC%2E)
5. Darwish, K., Mubarak, H., and Abdelali, A. (2017). Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17. [find](https://scholar.google.com/scholar?q=Darwish%2C+K%2E%2C+Mubarak%2C+H%2E%2C+and+Abdelali%2C+A%2E+%282017%29%2E+Arabic+diacritization%3A+Stats%2C+rules%2C+and+hacks%2E+In+Proceedings+of+the+Third+Arabic+Natural+Language+Processing+Workshop%2C+pages+9%E2%80%9317%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2017-darwish-arabic-diacritization-stats-rules-hacks/)
6. Dichy, J. and Fargaly, A. (2003). Roots and patterns vs. stems plus grammar-lexis specifications: on what basis should a multilingual lexical database centred on arabic be built? In Proceedings of the MTSummit, New-Orleans. [find](https://scholar.google.com/scholar?q=Dichy%2C+J%2E+and+Fargaly%2C+A%2E+%282003%29%2E+Roots+and+patterns+vs%2E+stems+plus+grammar%2Dlexis+specifications%3A+on+what+basis+should+a+multilingual+lexical+database+centred+on+arabic+be+built%3F+In+Proceedings+of+the)
7. El-Shishtawy, T. and Al-Sammak, A. (2009). Arabic keyphrase extraction using linguistic knowledge and machine learning techniques. In Proceedings of the Second International Conference on Arabic Language Resources and Tools, The MEDAR Consortium. [find](https://scholar.google.com/scholar?q=El%2DShishtawy%2C+T%2E+and+Al%2DSammak%2C+A%2E+%282009%29%2E+Arabic+keyphrase+extraction+using+linguistic+knowledge+and+machine+learning+techniques%2E+In+Proceedings+of+the+Second+International+Conference+on+Arabic+Langu)
8. El-Shishtawy, T. and El-Ghannam, F. (2012). An accurate arabic root-based lemmatizer for information retrieval purposes. arXiv preprint arXiv:1203.3584. [↗](https://arxiv.org/abs/1203.3584) [our card](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2012-el-shishtawy-accurate-arabic-root-based-lemmatizer/)
9. Graff, D., Maamouri, M., Bouziri, B., Krouna, S., Kulick, S., and Buckwalter, T. (2009). Standard arabic morphological analyzer (sama) version 3.1. In Linguistic Data Consortium LDC2009E73. [find](https://scholar.google.com/scholar?q=Graff%2C+D%2E%2C+Maamouri%2C+M%2E%2C+Bouziri%2C+B%2E%2C+Krouna%2C+S%2E%2C+Kulick%2C+S%2E%2C+and+Buckwalter%2C+T%2E+%282009%29%2E+Standard+arabic+morphological+analyzer+%28sama%29+version+3%2E1%2E+In+Linguistic+Data+Consortium+LDC2009E73%2E)
10. Khoja, S. (1999). Stemming arabic text. In Computing Department, Lancaster University. [find](https://scholar.google.com/scholar?q=Khoja%2C+S%2E+%281999%29%2E+Stemming+arabic+text%2E+In+Computing+Department%2C+Lancaster+University%2E)
11. Pasha, A., Al-Badrashiny, M., Diab, M., El Kholy, A., Eskander, R., Habash, N., Pooleery, M., Rambow, O., and Roth, R. M. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. Proc. LREC. [find](https://scholar.google.com/scholar?q=Pasha%2C+A%2E%2C+Al%2DBadrashiny%2C+M%2E%2C+Diab%2C+M%2E%2C+El+Kholy%2C+A%2E%2C+Eskander%2C+R%2E%2C+Habash%2C+N%2E%2C+Pooleery%2C+M%2E%2C+Rambow%2C+O%2E%2C+and+Roth%2C+R%2E+M%2E+%282014%29%2E+Madamira%3A+A+fast%2C+comprehensive+tool+for+morphological+analysis+and+di) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2014-pasha-madamira-fast-comprehensive-tool-morphological/)
12. Plisson, J., Lavrac, N., and Mladenic, M. (2004). A rule based approach to word lemmatization. In researchgate.net. [find](https://scholar.google.com/scholar?q=Plisson%2C+J%2E%2C+Lavrac%2C+N%2E%2C+and+Mladenic%2C+M%2E+%282004%29%2E+A+rule+based+approach+to+word+lemmatization%2E+In+researchgate%2Enet%2E)