[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2009-dinur-unsupervised-concept-discovery-hebrew-simple/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/) / Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic

# Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic

**Authors:** Elad Dinur; Dmitry Davidov; [Ari Rappoport](https://md.mova.elamur.ai/free/en/docs/papers/authors/ari-rappoport/) · **Year:** 2009 · **Venue:** Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages **Link:** [aclanthology.org/W09-0805](https://aclanthology.org/W09-0805/) · library `aclbulk-2009-dinur-unsupervised-concept-discovery-hebrew-simple` · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, before 2016) — summary only

[**2** citations](https://openalex.org/works?filter=cites:W1977618669) openalex · 2026-10-19[**2** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2009-dinur-unsupervised-concept-discovery-hebrew-simple/) of the 140673 papers whose bibliographies we read[Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages](https://aclanthology.org/volumes/W09-08/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/W09-0805/)
- [PDF](https://aclanthology.org/W09-0805.pdf)
- [DOI 10.3115/1621774.1621782](https://doi.org/10.3115/1621774.1621782)
- [OpenAlex](https://openalex.org/W1977618669)

**Authors (3)**

1. [Elad Dinur](https://openalex.org/A5007657377) [Semantic Scholar](https://www.semanticscholar.org/author/2895721) · h 2 · 124 citations · 3 works
2. [Dmitry Davidov](https://openalex.org/A5034956621) [Semantic Scholar](https://www.semanticscholar.org/author/33406957) · h 11 · 1975 citations · 24 works
3. [Ari Rappoport](https://openalex.org/A5038467328) [Semantic Scholar](https://www.semanticscholar.org/author/145009917) · h 40 · 5677 citations · 147 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

Hebrew and Arabic attach function words (and, in, the, that) to the next word, so word-based pattern methods for concept discovery miss the patterns that use "and" or "or". The authors split off the closed set of prefixes without any annotation. A word is split only when a rank, which compares the frequency of the prefixed form with the frequency of the stem and a short-word factor, is below a threshold; a trigram model over morphemes, found by dynamic programming, picks the best split of each sentence. On a 24,000-word hand-segmented Hebrew sample the method reaches F 0.851 and accuracy 0.893, against 0.523 for a baseline and 0.658 for Morfessor. On a 130 MB corpus, segmentation raises the number of Hebrew concepts from 36 to 148 and their average size from 2.9 to 4.1, with one junk word; without it, about ten times as much data is needed for similar results.

## Ideas

- I1. A rank-and-threshold split for unseen prefixed forms in `tools/hbo`: keep the closed prefix list the segmenter already uses, and score each candidate split by the frequency ratio of the prefixed form to its stem, with a length factor for short words. Test it on the unseen-word path against a hand-segmented sample, and report F and accuracy.
- I2. Use the prefix "w" (and) as a coordination signal: a pattern "x w y" with the content words on both sides is a symmetric relation. Test whether the coordination reading helps the event chain or the coordinated-NP attachment in [Syntax: the clause and the sentence](https://md.mova.elamur.ai/free/en/docs/languages/hbo/08-syntax/).
- I3. Measure how much corpus Mova's lexicon needs with and without prefix splitting, on the same corpus size, by counting distinct lexemes. This is a cheap check of the segmenter's value for `world`.

## For Mova

I1 is a fallback for unseen forms in `tools/hbo` (the segmenter; see [13. Biblical Hebrew in Mova: the tools](https://md.mova.elamur.ai/free/en/docs/languages/hbo/13-mova-tools/)). I2 belongs to [Syntax: the clause and the sentence](https://md.mova.elamur.ai/free/en/docs/languages/hbo/08-syntax/) (coordination) and [Prepositions, prefixes, conjunctions and particles](https://md.mova.elamur.ai/free/en/docs/languages/he/06-prepositions-particles/) (the prefix "w"). I3 is a measurement for [12. Mova and Modern Hebrew: where it stands, what it would need](https://md.mova.elamur.ai/free/en/docs/languages/he/12-mova/), run on the corpus outside the repo. The method is unsupervised and needs only the prefix list, which fits Mova's rules. The paper notes that the "bbyt" form has two readings; keep it as a test case for `tools/hbo`.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/1621774.1621782](https://doi.org/10.3115/1621774.1621782)

```
@inproceedings{dinur2009unsupervised,
  title = {{Unsupervised Concept Discovery In Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic}},
  author = {Dinur, Elad and Davidov, Dmitry and Rappoport, Ari},
  booktitle = {Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages},
  year = {2009},
  doi = {10.3115/1621774.1621782},
  url = {https://doi.org/10.3115/1621774.1621782}
}
```

- [← previous Morphological Disambiguation of Hebrew: A Case Study in Classifier Combination](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2007-shacham-morphological-disambiguation-hebrew-case-study/)
- [next → Easy-First Dependency Parsing of Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2010-goldberg-easy-first-dependency-parsing-modern/)
- [↑ back to Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/)

## References

22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Meni Adler, Michael Elhadad, 2006. An Unsupervised Morpheme-Based HMM for Hebrew Morphological Disambiguation. ACL ’06. [find](https://scholar.google.com/scholar?q=Meni+Adler%2C+Michael+Elhadad%2C+2006%2E+An+Unsupervised+Morpheme%2DBased+HMM+for+Hebrew+Morphological+Disambiguation%2E+ACL+%E2%80%9906%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-adler-unsupervised-morpheme-based-hmm-hebrew/)
2. Roy Bar-Haim, Khalil Simaan, Yoad Winter, 2005. Choosing an Optimal Architecture for Segmentation and POS-Tagging of Modern Hebrew. ACL Workshop on Computational Approaches to Semitic Languages ’05. [find](https://scholar.google.com/scholar?q=Roy+Bar%2DHaim%2C+Khalil+Simaan%2C+Yoad+Winter%2C+2005%2E+Choosing+an+Optimal+Architecture+for+Segmentation+and+POS%2DTagging+of+Modern+Hebrew%2E+ACL+Workshop+on+Computational+Approaches+to+Semitic+Languages+%E2%80%9905%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-haim-choosing-optimal-architecture-segmentation-pos/)
3. Mathias Creutz, 2003. Unsupervised Segmentation of Words Using Prior Distributions of Morph Length and Frequency. ACL ’03. [find](https://scholar.google.com/scholar?q=Mathias+Creutz%2C+2003%2E+Unsupervised+Segmentation+of+Words+Using+Prior+Distributions+of+Morph+Length+and+Frequency%2E+ACL+%E2%80%9903%2E)
4. Mathias Creutz and Krista Lagus, 2005. Unsupervised Morpheme Segmentation and Morphology Induction from Text Corpora Using Morfessor 1.0. In Computer and Information Science, Report A81, Helsinki University of Technology. [find](https://scholar.google.com/scholar?q=Mathias+Creutz+and+Krista+Lagus%2C+2005%2E+Unsupervised+Morpheme+Segmentation+and+Morphology+Induction+from+Text+Corpora+Using+Morfessor+1%2E0%2E+In+Computer+and+Information+Science%2C+Report+A81%2C+Helsinki+Univ)
5. Sajib Dasgupta and Vincent Ng, 2007. High Performance, Language-Independent Morphological Segmentation. NAACL/HLT ’07. [find](https://scholar.google.com/scholar?q=Sajib+Dasgupta+and+Vincent+Ng%2C+2007%2E+High+Performance%2C+Language%2DIndependent+Morphological+Segmentation%2E+NAACL%2FHLT+%E2%80%9907%2E)
6. Dmitry Davidov, Ari Rappoport, 2006. Efficient Unsupervised Discovery of Word Categories Using Symmetric Patterns and High Frequency Words. ACL ’06. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Ari+Rappoport%2C+2006%2E+Efficient+Unsupervised+Discovery+of+Word+Categories+Using+Symmetric+Patterns+and+High+Frequency+Words%2E+ACL+%E2%80%9906%2E)
7. Dmitry Davidov, Ari Rappoport, Moshe Koppel, 2007. Fully Unsupervised Discovery of Concept-Specific Relationships by Web Mining. ACL ’07. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Ari+Rappoport%2C+Moshe+Koppel%2C+2007%2E+Fully+Unsupervised+Discovery+of+Concept%2DSpecific+Relationships+by+Web+Mining%2E+ACL+%E2%80%9907%2E)
8. Dmitry Davidov, Ari Rappoport, 2008a. Classification of Semantic Relationships between Nominals Using Pattern Clusters. ACL ’08. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Ari+Rappoport%2C+2008a%2E+Classification+of+Semantic+Relationships+between+Nominals+Using+Pattern+Clusters%2E+ACL+%E2%80%9908%2E)
9. Dmitry Davidov, Ari Rappoport, 2008b. Unsupervised Discovery of Generic Relationships Using Pattern Clusters and its Evaluation by Automatically Generated SAT Analogy Questions. ACL ’08. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Ari+Rappoport%2C+2008b%2E+Unsupervised+Discovery+of+Generic+Relationships+Using+Pattern+Clusters+and+its+Evaluation+by+Automatically+Generated+SAT+Analogy+Questions%2E+ACL+%E2%80%9908%2E)
10. Vera Demberg, 2007. A Language-Independent Unsupervised Model for Morphological Segmentation. ACL ’07. [find](https://scholar.google.com/scholar?q=Vera+Demberg%2C+2007%2E+A+Language%2DIndependent+Unsupervised+Model+for+Morphological+Segmentation%2E+ACL+%E2%80%9907%2E)
11. Beate Dorow, Dominic Widdows, Katarina Ling, Jean-Pierre Eckmann, Danilo Sergi, Elisha Moses, 2005. Using Curvature and Markov Clustering in Graphs for Lexical Acquisition and Word Sense Discrimination. MEANING ’05. [find](https://scholar.google.com/scholar?q=Beate+Dorow%2C+Dominic+Widdows%2C+Katarina+Ling%2C+Jean%2DPierre+Eckmann%2C+Danilo+Sergi%2C+Elisha+Moses%2C+2005%2E+Using+Curvature+and+Markov+Clustering+in+Graphs+for+Lexical+Acquisition+and+Word+Sense+Discriminatio)
12. Samarth Keshava, Emily Pitler, 2006. A Simpler, Intuitive Approach to Morpheme Induction. In Proceedings of 2nd Pascal Challenges Workshop, Venice, Italy. [find](https://scholar.google.com/scholar?q=Samarth+Keshava%2C+Emily+Pitler%2C+2006%2E+A+Simpler%2C+Intuitive+Approach+to+Morpheme+Induction%2E+In+Proceedings+of+2nd+Pascal+Challenges+Workshop%2C+Venice%2C+Italy%2E)
13. Young-Suk Lee, Kishore Papineni, Salim Roukos, Ossama Emam, Hany Hassan, 2003. Language Model Based Arabic Word Segmentation. ACL ’03. [find](https://scholar.google.com/scholar?q=Young%2DSuk+Lee%2C+Kishore+Papineni%2C+Salim+Roukos%2C+Ossama+Emam%2C+Hany+Hassan%2C+2003%2E+Language+Model+Based+Arabic+Word+Segmentation%2E+ACL+%E2%80%9903%2E)
14. Moshe Levinger, Uzzi Ornan, Alon Itai, 1995. Learning Morpho-Lexical Probabilities from an Untagged Corpus with an Application to Hebrew. Comput. Linguistics, 21:383:404. [find](https://scholar.google.com/scholar?q=Moshe+Levinger%2C+Uzzi+Ornan%2C+Alon+Itai%2C+1995%2E+Learning+Morpho%2DLexical+Probabilities+from+an+Untagged+Corpus+with+an+Application+to+Hebrew%2E+Comput%2E+Linguistics%2C+21%3A383%3A404%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/cl-1995-levinger-learning-morpho-lexical-probabilities-untagged/)
15. Dekang Lin, 1998. Automatic Retrieval and Clustering of Similar Words. COLING ’98. [find](https://scholar.google.com/scholar?q=Dekang+Lin%2C+1998%2E+Automatic+Retrieval+and+Clustering+of+Similar+Words%2E+COLING+%E2%80%9998%2E)
16. Uzzi Ornan, 2005. Hebrew in Latin Script. Lesonenu, LXIV:137:151 (in Hebrew). [find](https://scholar.google.com/scholar?q=Uzzi+Ornan%2C+2005%2E+Hebrew+in+Latin+Script%2E+Lesonenu%2C+LXIV%3A137%3A151+%28in+Hebrew%29%2E)
17. Patrick Pantel, Dekang Lin, 2002. Discovering Word Senses from Text. SIGKDD ’02. [find](https://scholar.google.com/scholar?q=Patrick+Pantel%2C+Dekang+Lin%2C+2002%2E+Discovering+Word+Senses+from+Text%2E+SIGKDD+%E2%80%9902%2E)
18. Patrick Pantel, Deepak Ravichandran, Eduard Hovy, 2004. Towards Terascale Knowledge Acquisition. COLING ’04. [find](https://scholar.google.com/scholar?q=Patrick+Pantel%2C+Deepak+Ravichandran%2C+Eduard+Hovy%2C+2004%2E+Towards+Terascale+Knowledge+Acquisition%2E+COLING+%E2%80%9904%2E)
19. Fernando Pereira, Naftali Tishby, Lillian Lee, 1993. Distributional Clustering of English Words. ACL ’93. [find](https://scholar.google.com/scholar?q=Fernando+Pereira%2C+Naftali+Tishby%2C+Lillian+Lee%2C+1993%2E+Distributional+Clustering+of+English+Words%2E+ACL+%E2%80%9993%2E)
20. Benjamin Snyder, Regina Bazilay, 2008. Unsupervised Multilingual Learning for Morphological Segmentation. ACL/HLT ’08. [find](https://scholar.google.com/scholar?q=Benjamin+Snyder%2C+Regina+Bazilay%2C+2008%2E+Unsupervised+Multilingual+Learning+for+Morphological+Segmentation%2E+ACL%2FHLT+%E2%80%9908%2E)
21. Andreas Stolcke, 2002. SRILM – an Extensible Language Modeling Toolkit. ICSLP, pages 901-904, Denver, Colorado. [find](https://scholar.google.com/scholar?q=Andreas+Stolcke%2C+2002%2E+SRILM+%E2%80%93+an+Extensible+Language+Modeling+Toolkit%2E+ICSLP%2C+pages+901%2D904%2C+Denver%2C+Colorado%2E)
22. Dominic Widdows, Beate Dorow, 2002. A Graph Model for Unsupervised Lexical Acquisition. COL-ING ’02. [find](https://scholar.google.com/scholar?q=Dominic+Widdows%2C+Beate+Dorow%2C+2002%2E+A+Graph+Model+for+Unsupervised+Lexical+Acquisition%2E+COL%2DING+%E2%80%9902%2E)