[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/hbo-2022-gueta-large-pre-trained-models-extra/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Ancient Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/hbo/) / Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All

# Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All

**Authors:** [Eylon Gueta](https://md.mova.elamur.ai/free/en/docs/papers/authors/eylon-gueta/), [Avi Shmidman](https://md.mova.elamur.ai/free/en/docs/papers/authors/avi-shmidman/), [Shaltiel Shmidman](https://md.mova.elamur.ai/free/en/docs/papers/authors/shaltiel-shmidman/), [Cheyn Shmuel Shmidman](https://md.mova.elamur.ai/free/en/docs/papers/authors/cheyn-shmuel-shmidman/), [Joshua Guedalia](https://md.mova.elamur.ai/free/en/docs/papers/authors/joshua-guedalia/), [Moshe Koppel](https://md.mova.elamur.ai/free/en/docs/papers/authors/moshe-koppel/), [Dan Bareket](https://md.mova.elamur.ai/free/en/docs/papers/authors/dan-bareket/), [Amit Seker](https://md.mova.elamur.ai/free/en/docs/papers/authors/amit-seker/), [Reut Tsarfaty](https://md.mova.elamur.ai/free/en/docs/papers/authors/reut-tsarfaty/) · **Year:** 2022 · **Venue:** arXiv 2211.15199 **Link:** [arxiv.org/abs/2211.15199](https://arxiv.org/abs/2211.15199) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**5** citations](https://openalex.org/works?filter=cites:W4320169836) openalex · 2026-10-18[**23** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/hbo-2022-gueta-large-pre-trained-models-extra/) of the 140673 papers whose bibliographies we read[arXiv:2211.15199](https://openalex.org/S4306400194)[licence](http://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://arxiv.org/abs/2211.15199)
- [PDF](https://arxiv.org/pdf/2211.15199)
- [arXiv 2211.15199](https://arxiv.org/abs/2211.15199)
- [OpenAlex](https://openalex.org/W4320169836)

**Authors (9)**

1. Eylon Gueta
2. [Avi Shmidman](https://openalex.org/A5013212409) h 5 · 122 citations · 53 works
3. [Shaltiel Shmidman](https://openalex.org/A5041361841) h 3 · 30 citations · 21 works
4. [Cheyn Shmuel Shmidman](https://openalex.org/A5015862561) h 3 · 13 citations · 6 works
5. [Joshua Guedalia](https://openalex.org/A5059119329) [ORCID](https://orcid.org/0000-0002-3458-7896) · h 14 · 492 citations · 48 works
6. [Moshe Koppel](https://openalex.org/A5019908505) h 37 · 7124 citations · 150 works
7. [Dan Bareket](https://openalex.org/A5005137956) [ORCID](https://orcid.org/0009-0008-2364-5676) · h 5 · 120 citations · 15 works
8. [Amit Seker](https://openalex.org/A5049561362) h 7 · 185 citations · 13 works
9. [Reut Tsarfaty](https://openalex.org/A5063283689) [ORCID](https://orcid.org/0000-0001-6406-6568) · h 24 · 2886 citations · 217 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The authors train AlephBERTGimmel, a Modern Hebrew BERT with a 128K word-piece vocabulary (earlier models had 2.4K–52K), on the same ~2 billion words as AlephBERT. They also merge the UD, SPMRL and NEMO annotations of the same Hebrew texts into one consistent benchmark (UD-NEMO). The large-vocabulary model reaches the best results on segmentation, POS, morphological features, named entities and sentiment. Their analysis shows that the more pieces a word is split into, the worse every model does on it, so a vocabulary that keeps more words whole helps most, especially on semantic tasks.

## Ideas

- I1. For Mova's own Hebrew analysis, look words up whole first (a full-form lexicon from the Torah trees and the dictionary) and only then split off prefixes and suffixes; report accuracy by number of segments per word, the paper's diagnostic.
- I2. Use the UD-NEMO test set (if its licence allows tests) to measure a Modern Hebrew parser's segmentation, and the morpheme-level "mset" score as the metric.

## For Mova

The model is a neural Modern Hebrew model and cannot be part of Mova at answer time; it could at most serve as a teacher or a baseline. I1 bears on the word segmentation described in [13. Biblical Hebrew in Mova: the tools](https://md.mova.elamur.ai/free/en/docs/languages/hbo/13-mova-tools/) §4 (tokens) and on the Modern Hebrew manual (`languages/he/`); I2 on the measurements in `12-biblical-mishnaic-modern.md` ("why a Modern Hebrew parser misreads the Bible").

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 2211.15199](https://arxiv.org/abs/2211.15199)

```
@misc{gueta2022large,
  title = {{Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All}},
  author = {Gueta, Eylon and Shmidman, Avi and Shmidman, Shaltiel and Shmidman, Cheyn Shmuel and Guedalia, Joshua and Koppel, Moshe and Bareket, Dan and Seker, Amit and Tsarfaty, Reut},
  year = {2022},
  eprint = {2211.15199},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.15199}
}
```

- [← previous Negating the infinitive in Biblical Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/hbo-2022-ehrensvard-negating-infinitive-biblical-hebrew/)
- [next → The System of Masoretic Accentuation: Some Introductory Issues](https://md.mova.elamur.ai/free/en/docs/papers/hbo-2022-hoop-system-masoretic-accentuation-some-introductory/)
- [↑ back to Ancient Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/hbo/)

## References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Antoine Nzeyimana and Andre Niyongabo Rubungo. 2022. KinyaBERT: a morphology-aware Kinyarwanda language model. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5347 5363, Dublin, Ireland. Association for Compu tational Linguistics. [find](https://scholar.google.com/scholar?q=Antoine+Nzeyimana+and+Andre+Niyongabo+Rubungo%2E+2022%2E+KinyaBERT%3A+a+morphology%2Daware+Kinyarwanda+language+model%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistic)
2. Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from modern hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242 2252. [find](https://scholar.google.com/scholar?q=Adam+Amram%2C+Anat+Ben+David%2C+and+Reut+Tsarfaty%2E+2018%2E+Representations+and+architectures+in+neural+sentiment+analysis+for+morphologically+rich+languages%3A+A+case+study+from+modern+hebrew%2E+In+Proceedings+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/coling-2018-amram-representations-architectures-neural-sentiment-analysis/)
3. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
4. Dan Bareket and Reut Tsarfaty. 2021. Neural modeling for named entities and morphology (nemo^2). Transactions of the Association for Computational Linguistics, 9:909–928. [find](https://scholar.google.com/scholar?q=Dan+Bareket+and+Reut+Tsarfaty%2E+2021%2E+Neural+modeling+for+named+entities+and+morphology+%28nemo%5E2%29%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+9%3A909%E2%80%93928%2E)
5. Shoval Sade, Amit Seker, and Reut Tsarfaty. 2018. The hebrew universal dependency treebank: Past present and future. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 133 143. [find](https://scholar.google.com/scholar?q=Shoval+Sade%2C+Amit+Seker%2C+and+Reut+Tsarfaty%2E+2018%2E+The+hebrew+universal+dependency+treebank%3A+Past+present+and+future%2E+In+Proceedings+of+the+Second+Workshop+on+Universal+Dependencies+%28UDW+2018%29%2C+pages+1) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2018-sade-hebrew-universal-dependency-treebank-past/)
6. Idan Brusilovsky and Reut Tsarfaty. 2022. Neural token segmentation for high token-internal complexity. arXiv preprint arXiv:2203.10845. [↗](https://arxiv.org/abs/2203.10845)
7. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Woliński, Alina Wróblewska, and Eric Villemonte de la Clergerie. 2013. Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 146–182, Seattle, Washington, USA. Association for Computa tional Linguistics. [find](https://scholar.google.com/scholar?q=Djam%C3%A9+Seddah%2C+Reut+Tsarfaty%2C+Sandra+K%C3%BCbler%2C+Marie+Candito%2C+Jinho+D%2E+Choi%2C+Rich%C3%A1rd+Farkas%2C+Jennifer+Foster%2C+Iakes+Goenaga%2C+Koldo+Gojenola+Galletebeitia%2C+Yoav+Goldberg%2C+Spence+Green%2C+Nizar+Habash%2C+Marco)
8. Avihay Chriqui and Inbal Yahav. 2021. Hebert & hebemo: a hebrew bert model and a tool for polarity analysis and emotion recognition. [find](https://scholar.google.com/scholar?q=Avihay+Chriqui+and+Inbal+Yahav%2E+2021%2E+Hebert+%26+hebemo%3A+a+hebrew+bert+model+and+a+tool+for+polarity+analysis+and+emotion+recognition%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2021-chriqui-hebert-hebemo-hebrew-bert-model/)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
10. Omri Keren, Tal Avinari, Reut Tsarfaty, and Omer Levy. 2022. Breaking character: Are subwords good enough for mrls after all? [find](https://scholar.google.com/scholar?q=Omri+Keren%2C+Tal+Avinari%2C+Reut+Tsarfaty%2C+and+Omer+Levy%2E+2022%2E+Breaking+character%3A+Are+subwords+good+enough+for+mrls+after+all%3F)
11. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert: A hebrew large pre-trained language model to start-off your hebrew NLP application with. CoRR, abs/2104.04052. [↗](https://arxiv.org/abs/2104.04052) [our card](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2021-seker-alephbert-hebrew-large-pre-trained/)
12. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State of the-art natural language processing. CoRR, abs/1910.03771. [↗](https://arxiv.org/abs/1910.03771)