[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/eacl-2023-shmidman-do-pretrained-contextual-language-models/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Ancient Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/hbo/) / Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?

# Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?

**Authors:** [Avi Shmidman](https://md.mova.elamur.ai/free/en/docs/papers/authors/avi-shmidman/); [Cheyn Shmuel Shmidman](https://md.mova.elamur.ai/free/en/docs/papers/authors/cheyn-shmuel-shmidman/); [Dan Bareket](https://md.mova.elamur.ai/free/en/docs/papers/authors/dan-bareket/); [Moshe Koppel](https://md.mova.elamur.ai/free/en/docs/papers/authors/moshe-koppel/); [Reut Tsarfaty](https://md.mova.elamur.ai/free/en/docs/papers/authors/reut-tsarfaty/) · **Year:** 2023 · **Venue:** Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 849–864 **Link:** [aclanthology.org/2023.eacl-main.59](https://aclanthology.org/2023.eacl-main.59/) · library `eacl-2023-shmidman-do-pretrained-contextual-language-models` · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology)

[**3** citations](https://openalex.org/works?filter=cites:W4386566841) openalex · 2026-10-18[**2** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/eacl-2023-shmidman-do-pretrained-contextual-language-models/) of the 140673 papers whose bibliographies we read[Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics](https://aclanthology.org/volumes/2023.eacl-main/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2023.eacl-main.59/)
- [PDF](https://aclanthology.org/2023.eacl-main.59.pdf)
- [DOI 10.18653/v1/2023.eacl-main.59](https://doi.org/10.18653/v1/2023.eacl-main.59)
- [OpenAlex](https://openalex.org/W4386566841)

**Authors (5)**

1. [Avi Shmidman](https://openalex.org/A5013212409) h 5 · 122 citations · 53 works
2. [Cheyn Shmuel Shmidman](https://openalex.org/A5015862561) h 3 · 13 citations · 6 works
3. [Dan Bareket](https://openalex.org/A5005137956) [ORCID](https://orcid.org/0009-0008-2364-5676) · h 5 · 120 citations · 15 works
4. [Moshe Koppel](https://openalex.org/A5019908505) h 37 · 7124 citations · 150 works
5. [Reut Tsarfaty](https://openalex.org/A5063283689) [ORCID](https://orcid.org/0000-0001-6406-6568) · h 24 · 2886 citations · 217 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

Unpointed Hebrew is full of homographs whose analyses differ in segmentation (the prefix or suffix is or is not a word), in morphosyntax, or only in sense. The authors ask whether contextual embeddings (mBERT, HeBERT, AlephBERT) separate these analyses. They build 75 challenge sets from the 3,600 most frequent words, about 150K tagged sentences from news, Wikipedia, literature and social media, with 2–5 analyses per homograph. Each homograph gets its own MLP on the embedding of the target token (word-expert setup, 10-fold CV, macro-F1). AlephBERT is best and clearly beats the word2vec BiLSTM of their earlier study. It is strongest on segmentation and morphosyntax, weaker on pure sense, weaker with 4- or 5-way ambiguity and with 3-piece word-piece splits, and not helped by masking the target. With only 5 examples per analysis it stays close to full CV.

## Ideas

- I1. Report each homograph score split into three buckets (segmentation, morphosyntax, sense), as the paper does. Do the same for `tools/hbo` unpointed mode, so a segmentation gain cannot hide a sense loss.
- I2. Few-shot check as a test: for each homograph, train the `tools/hbo` disambiguation rules on 5 examples per analysis and report accuracy against the full-data rule set. It shows how far a rule table can go with little data.
- I3. Stratify the evaluation by the frequency ratio of the two analyses (balanced vs skewed, 1:1 to 50:1), the way the paper plots it, so that skewed homographs are scored separately.

## For Mova

This is a Modern Hebrew paper. I1 and I3 are measurement designs for `tools/hbo` (unpointed mode) and go into its test notes. I2 is a test for the unpointed segmenter tables (`lex`, `pre`) and the rules in [Nouns, adjectives, pronouns and numerals](https://md.mova.elamur.ai/free/en/docs/languages/he/04-nouns-adjectives-pronouns/) and [Prepositions, prefixes, conjunctions and particles](https://md.mova.elamur.ai/free/en/docs/languages/he/06-prepositions-particles/) once Mova reads Modern Hebrew (plan in [12. Mova and Modern Hebrew: where it stands, what it would need](https://md.mova.elamur.ai/free/en/docs/languages/he/12-mova/)). Nothing here changes `languages/hbo/`.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2023.eacl-main.59](https://doi.org/10.18653/v1/2023.eacl-main.59)

```
@inproceedings{shmidman2023do,
  title = {{Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?}},
  author = {Shmidman, Avi and Shmidman, Cheyn Shmuel and Bareket, Dan and Koppel, Moshe and Tsarfaty, Reut},
  booktitle = {Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.eacl-main.59},
  url = {https://doi.org/10.18653/v1/2023.eacl-main.59}
}
```

- [← previous Towards Named-Entity and Coreference Annotation of the Hebrew Bible](https://md.mova.elamur.ai/free/en/docs/papers/coref-2024-swanson-named-entity-coreference-annotation-hebrew/)
- [next → A Dataset for Metaphor Detection in Early Medieval Hebrew Poetry](https://md.mova.elamur.ai/free/en/docs/papers/eacl-2024-toker-dataset-metaphor-detection-early-medieval/)
- [↑ back to Ancient Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/hbo/)

## References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mohammed Alaeddine Abderrahim and Mohammed El-Amine Abderrahim. 2022. Arabic word sense disambiguation for information retrieval. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 21(4). [find](https://scholar.google.com/scholar?q=Mohammed+Alaeddine+Abderrahim+and+Mohammed+El%2DAmine+Abderrahim%2E+2022%2E+Arabic+word+sense+disambiguation+for+information+retrieval%2E+ACM+Trans%2E+Asian+Low%2DResour%2E+Lang%2E+Inf%2E+Process%2E%2C+21%284%29%2E)
2. Meni Adler and Michael Elhadad. 2006. An unsupervised morpheme-based hmm for hebrew morphological disambiguation. In COLING/ACL 2006 - 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, COLING/ACL 2006 - 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 665–672, United States. Association for Computational Linguistics (ACL). 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, COLING/ACL 2006 ; Conference date: 17-07-2006 Through 21-07-2006. [find](https://scholar.google.com/scholar?q=Meni+Adler+and+Michael+Elhadad%2E+2006%2E+An+unsupervised+morpheme%2Dbased+hmm+for+hebrew+morphological+disambiguation%2E+In+COLING%2FACL+2006+%2D+21st+International+Conference+on+Computational+Linguistics+and+44) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-adler-unsupervised-morpheme-based-hmm-hebrew/)
3. Eneko Agirre and Philip Edmonds, editors. 2006. Word Sense Disambiguation: Algorithms and Applications, volume 33 of Text, Speech and Language Technology. Springer. [find](https://scholar.google.com/scholar?q=Eneko+Agirre+and+Philip+Edmonds%2C+editors%2E+2006%2E+Word+Sense+Disambiguation%3A+Algorithms+and+Applications%2C+volume+33+of+Text%2C+Speech+and+Language+Technology%2E+Springer%2E)
4. Yonatan Belinkov. 2021. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, pages 1–13. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2E+2021%2E+Probing+Classifiers%3A+Promises%2C+Shortcomings%2C+and+Advances%2E+Computational+Linguistics%2C+pages+1%E2%80%9313%2E)
5. Avihay Chriqui and Inbal Yahav. 2021. Hebert hebemo: a hebrew bert model and a tool for polarity analysis and emotion recognition. [find](https://scholar.google.com/scholar?q=Avihay+Chriqui+and+Inbal+Yahav%2E+2021%2E+Hebert+hebemo%3A+a+hebrew+bert+model+and+a+tool+for+polarity+analysis+and+emotion+recognition%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2021-chriqui-hebert-hebemo-hebrew-bert-model/)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E)
7. Ali Elkahky, Kellie Webster, Daniel Andor, and Emily Pitler. 2018. A challenge set and methods for noun-verb ambiguity. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2562–2572, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ali+Elkahky%2C+Kellie+Webster%2C+Daniel+Andor%2C+and+Emily+Pitler%2E+2018%2E+A+challenge+set+and+methods+for+noun%2Dverb+ambiguity%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+P)
8. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2E+2020%2E+What+BERT+is+not%3A+Lessons+from+a+new+suite+of+psycholinguistic+diagnostics+for+language+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A34%E2%80%9348%2E)
9. Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models’ local decision boundaries via contrast sets. [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Yoav+Artzi%2C+Victoria+Basmova%2C+Jonathan+Berant%2C+Ben+Bogin%2C+Sihao+Chen%2C+Pradeep+Dasigi%2C+Dheeru+Dua%2C+Yanai+Elazar%2C+Ananth+Gottumukkala%2C+Nitish+Gupta%2C+Hanna+Hajishirzi%2C+Gabriel+Ilharco%2C+Dani) [our card](https://md.mova.elamur.ai/free/en/docs/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
10. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. CoRR, abs/1901.05287. [↗](https://arxiv.org/abs/1901.05287)
11. Nizar Habash and Owen Rambow. 2005. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nizar+Habash+and+Owen+Rambow%2E+2005%2E+Arabic+tokenization%2C+part%2Dof%2Dspeech+tagging+and+morphological+disambiguation+in+one+fell+swoop%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+Association+for+Com) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-habash-arabic-tokenization-part-speech-tagging/)
12. Josef Klafka and Allyson Ettinger. 2020. Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4801–4811, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Josef+Klafka+and+Allyson+Ettinger%2E+2020%2E+Spying+on+your+neighbors%3A+Fine%2Dgrained+probing+of+contextual+embeddings+for+information+about+surrounding+words%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+t)
13. Stav Klein and Reut Tsarfaty. 2020. Getting the [find](https://scholar.google.com/scholar?q=Stav+Klein+and+Reut+Tsarfaty%2E+2020%2E+Getting+the)
14. Laroussi Merhben, Zouaghi Anis, and Mounir Zrigui. 2010. Ambiguous arabic words disambiguation. pages 157–164. [find](https://scholar.google.com/scholar?q=Laroussi+Merhben%2C+Zouaghi+Anis%2C+and+Mounir+Zrigui%2E+2010%2E+Ambiguous+arabic+words+disambiguation%2E+pages+157%E2%80%93164%2E)
15. Laroussi Merhbene, Anis Zouaghi, and Mounir Zrigui. 2013. A semi-supervised method for Arabic word sense disambiguation using a weighted directed graph. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1027–1031, Nagoya, Japan. Asian Federation of Natural Language Processing. [find](https://scholar.google.com/scholar?q=Laroussi+Merhbene%2C+Anis+Zouaghi%2C+and+Mounir+Zrigui%2E+2013%2E+A+semi%2Dsupervised+method+for+Arabic+word+sense+disambiguation+using+a+weighted+directed+graph%2E+In+Proceedings+of+the+Sixth+International+Joint)
16. Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM Comput. Surv., 41. [find](https://scholar.google.com/scholar?q=Roberto+Navigli%2E+2009%2E+Word+sense+disambiguation%3A+A+survey%2E+ACM+Comput%2E+Surv%2E%2C+41%2E)
17. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert:a hebrew large pre-trained language model to start-off your hebrew nlp application with. [find](https://scholar.google.com/scholar?q=Amit+Seker%2C+Elron+Bandel%2C+Dan+Bareket%2C+Idan+Brusilovsky%2C+Refael+Shaked+Greenfeld%2C+and+Reut+Tsarfaty%2E+2021%2E+Alephbert%3Aa+hebrew+large+pre%2Dtrained+language+model+to+start%2Doff+your+hebrew+nlp+application+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2021-seker-alephbert-hebrew-large-pre-trained/)
18. Danny Shacham and Shuly Wintner. 2007. Morphological disambiguation of Hebrew: A case study in classifier combination. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 439– 447, Prague, Czech Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Danny+Shacham+and+Shuly+Wintner%2E+2007%2E+Morphological+disambiguation+of+Hebrew%3A+A+case+study+in+classifier+combination%2E+In+Proceedings+of+the+2007+Joint+Conference+on+Empirical+Methods+in+Natural+Langu) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2007-shacham-morphological-disambiguation-hebrew-case-study/)
19. Rushin Shah, Paramveer S. Dhillon, Mark Liberman, Dean Foster, Mohamed Maamouri, and Lyle Ungar. 2010. A new approach to lexical disambiguation of Arabic text. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 725–735, Cambridge, MA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rushin+Shah%2C+Paramveer+S%2E+Dhillon%2C+Mark+Liberman%2C+Dean+Foster%2C+Mohamed+Maamouri%2C+and+Lyle+Ungar%2E+2010%2E+A+new+approach+to+lexical+disambiguation+of+Arabic+text%2E+In+Proceedings+of+the+2010+Conference+on)
20. Avi Shmidman, Joshua Guedalia, Shaltiel Shmidman, Moshe Koppel, and Reut Tsarfaty. 2020. A novel challenge set for Hebrew morphological disambiguation and diacritics restoration. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3316–3326, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Avi+Shmidman%2C+Joshua+Guedalia%2C+Shaltiel+Shmidman%2C+Moshe+Koppel%2C+and+Reut+Tsarfaty%2E+2020%2E+A+novel+challenge+set+for+Hebrew+morphological+disambiguation+and+diacritics+restoration%2E+In+Findings+of+the+As) [our card](https://md.mova.elamur.ai/free/en/docs/papers/findings-2020-shmidman-novel-challenge-set-hebrew-morphological/)
21. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Patrick+Xia%2C+Berlin+Chen%2C+Alex+Wang%2C+Adam+Poliak%2C+R+Thomas+McCoy%2C+Najoung+Kim%2C+Benjamin+Van+Durme%2C+Samuel+R%2E+Bowman%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019%2E+What+do+you+learn+from+context%3F+)
22. Reut Tsarfaty, Dan Bareket, Stav Klein, and Amit Seker. 2020. From SPMRL to NMRL: What did we learn (and unlearn) in a decade of parsing morphologically-rich languages (MRLs)? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7396–7408, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Reut+Tsarfaty%2C+Dan+Bareket%2C+Stav+Klein%2C+and+Amit+Seker%2E+2020%2E+From+SPMRL+to+NMRL%3A+What+did+we+learn+%28and+unlearn%29+in+a+decade+of+parsing+morphologically%2Drich+languages+%28MRLs%29%3F+In+Proceedings+of+the+58)
23. Reut Tsarfaty, Amit Seker, Shoval Sadde, and Stav Klein. 2019. What’s wrong with hebrew nlp? and how to make it right. [find](https://scholar.google.com/scholar?q=Reut+Tsarfaty%2C+Amit+Seker%2C+Shoval+Sadde%2C+and+Stav+Klein%2E+2019%2E+What%E2%80%99s+wrong+with+hebrew+nlp%3F+and+how+to+make+it+right%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2019-tsarfaty-what-s-wrong-hebrew-nlp/)
24. Shuly Wintner. 2014. Morphological processing of semitic languages. In NLP of Semitic Languages. [find](https://scholar.google.com/scholar?q=Shuly+Wintner%2E+2014%2E+Morphological+processing+of+semitic+languages%2E+In+NLP+of+Semitic+Languages%2E)
25. Yadollah Yaghoobzadeh, Katharina Kann, T. J. Hazen, Eneko Agirre, and Hinrich Schütze. 2019. Probing for semantic classes: Diagnosing the meaning content of word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5740–5753, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yadollah+Yaghoobzadeh%2C+Katharina+Kann%2C+T%2E+J%2E+Hazen%2C+Eneko+Agirre%2C+and+Hinrich+Sch%C3%BCtze%2E+2019%2E+Probing+for+semantic+classes%3A+Diagnosing+the+meaning+content+of+word+embeddings%2E+In+Proceedings+of+the+57th)
26. Shlomo Yona and Shuly Wintner. 2005. A finite-state morphological grammar of Hebrew. In Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, pages 9–16, Ann Arbor, Michigan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shlomo+Yona+and+Shuly+Wintner%2E+2005%2E+A+finite%2Dstate+morphological+grammar+of+Hebrew%2E+In+Proceedings+of+the+ACL+Workshop+on+Computational+Approaches+to+Semitic+Languages%2C+pages+9%E2%80%9316%2C+Ann+Arbor%2C+Michiga) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-yona-finite-state-morphological-grammar-hebrew/)
27. Mengjie Zhao, Philipp Dufter, Yadollah Yaghoobzadeh, and Hinrich Schütze. 2020. Quantifying the contextualization of word representations with semantic class probing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1219–1234, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mengjie+Zhao%2C+Philipp+Dufter%2C+Yadollah+Yaghoobzadeh%2C+and+Hinrich+Sch%C3%BCtze%2E+2020%2E+Quantifying+the+contextualization+of+word+representations+with+semantic+class+probing%2E+In+Findings+of+the+Association+fo)