[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/arxivrecent-2023-shmidman-dictabert-state-art-bert-suite/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/) / DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew

# DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew

**Authors:** [Shaltiel Shmidman](https://md.mova.elamur.ai/free/en/docs/papers/authors/shaltiel-shmidman/); [Avi Shmidman](https://md.mova.elamur.ai/free/en/docs/papers/authors/avi-shmidman/); [Moshe Koppel](https://md.mova.elamur.ai/free/en/docs/papers/authors/moshe-koppel/) · **Year:** 2023 · **Venue:** arXiv:2308.16687 **Link:** [arxiv.org/abs/2308.16687](https://arxiv.org/abs/2308.16687) · library `arxivrecent-2023-shmidman-dictabert-state-art-bert-suite` · **Licence:** [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) — summary only, never carried into a release

[**2** citations](https://openalex.org/works?filter=cites:W4386384732) openalex · 2026-10-18[**20** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/arxivrecent-2023-shmidman-dictabert-state-art-bert-suite/) of the 140673 papers whose bibliographies we read[arXiv:2308.16687](https://openalex.org/S4306400194)[licence](http://creativecommons.org/licenses/by-sa/4.0/)

- [Publisher's page](https://arxiv.org/abs/2308.16687)
- [PDF](https://arxiv.org/pdf/2308.16687)
- [arXiv 2308.16687](https://arxiv.org/abs/2308.16687)
- [OpenAlex](https://openalex.org/W4386384732)

**Authors (3)**

1. [Shaltiel Shmidman](https://openalex.org/A5041361841) h 3 · 30 citations · 21 works
2. [Avi Shmidman](https://openalex.org/A5013212409) h 5 · 122 citations · 53 works
3. [Moshe Koppel](https://openalex.org/A5019908505) h 37 · 7124 citations · 150 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The paper presents DictaBERT, a BERT-architecture model for Modern Hebrew, pre-trained with masked language modelling only (next-sentence prediction removed) on about 3.8B tokens: 70% cleaned HeDC4 web text and 30% news, blogs, subtitles and novels. Three fine-tuned heads are released with it: DictaBERT-seg (prefix segmentation, 52K sentences), DictaBERT-morph (five classifiers for POS, proclitic functions, features, suffix presence and suffix features) and DictaBERT-heq (question answering on HeQ). Against mBERT, AlephBERT, AlephBERT-Gimmel, HeRo and mT5-Base it scores best on most tasks: segmentation 98.16 (AlephBERT-Gimmel 98.09), NER 87.01 (HeRo 86.79), sentiment 89.79 (HeRo 89.56), and QA F1 72.9, close to mT5-XL at 73.5 with about ten times the parameters. The paper is a model release; its evaluation tables are the reference numbers.

## Ideas

- I1. Constrain prefix segmentation by a table of legal proclitic sets keyed on the word's initial letters. The model applies only the valid function sets for a word's first letters at inference. A deterministic `pre` table in the segmenter can enforce the same rule; measure how many illegal splits it removes on a held-out set.
- I2. Keep the morphology as separate outputs: POS, proclitic functions (several may stack), core features, suffix presence and suffix features. This is a usable layout for mapping Modern Hebrew tags to UD features, one slot per decision.
- I3. Reproduce the multi-task segmentation and POS setup of Seker et al. (2021) on the same data before any claim about a Mova segmenter, and report it against the paper's Table 1 figures.

## For Mova

I1 goes into [Prepositions, prefixes, conjunctions and particles](https://md.mova.elamur.ai/free/en/docs/languages/he/06-prepositions-particles/) (the legal prefix sets) and into the planned Modern Hebrew segmenter in [12. Mova and Modern Hebrew: where it stands, what it would need](https://md.mova.elamur.ai/free/en/docs/languages/he/12-mova/), following the table pattern of `tools/hbo`. I2 goes into [11. Modern Hebrew in Universal Dependencies](https://md.mova.elamur.ai/free/en/docs/languages/he/11-ud-analysis/). I3 is a test for [12. Mova and Modern Hebrew: where it stands, what it would need](https://md.mova.elamur.ai/free/en/docs/languages/he/12-mova/). The model itself is neural and is only a teacher or a reference; nothing of it is a dependency.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 2308.16687](https://arxiv.org/abs/2308.16687)

```
@misc{shmidman2023dictabert,
  title = {{DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew}},
  author = {Shmidman, Shaltiel and Shmidman, Avi and Koppel, Moshe},
  year = {2023},
  eprint = {2308.16687},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.16687}
}
```

- [← previous Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2026-samo-modelling-morphology-verbal-paradigms-case/)
- [next → The Knesset Corpus: An Annotated Corpus of Hebrew Parliamentary Proceedings](https://md.mova.elamur.ai/free/en/docs/papers/arxivrecent-2024-goldin-knesset-corpus-annotated-corpus-hebrew/)
- [↑ back to Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/)

## References

14 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from Modern Hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242–2252, Santa The base model DictaBERT is available at [https://huggingface.co/dicta-il/dictabert](https://huggingface.co/dicta-il/dictabert) The fine-tuned segmentation model DictaBERT-seg is available at https://huggingface.co/dicta-il/ dictabert-seg The fine-tuned morphology model DictaBERT-morph is available at https://huggingface.co/dicta-il/ dictabert-morph The fine-tuned QA model DictaBERT-heq is available at https://huggingface.co/dicta-il/ dictabert-heq. Fe, New Mexico, USA. Association for Computational Linguistics. [↗](https://huggingface.co/dicta-il/dictabert) [our card](https://md.mova.elamur.ai/free/en/docs/papers/coling-2018-amram-representations-architectures-neural-sentiment-analysis/)
2. Dan Bareket and Reut Tsarfaty. 2021. Neural Modeling for Named Entities and Morphology (NEMO2). Transactions of the Association for Computational Linguistics, 9:909–928. [find](https://scholar.google.com/scholar?q=Dan+Bareket+and+Reut+Tsarfaty%2E+2021%2E+Neural+Modeling+for+Named+Entities+and+Morphology+%28NEMO2%29%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+9%3A909%E2%80%93928%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/tacl-2021-bareket-neural-modeling-named-entities-morphology/)
3. Avihay Chriqui and Inbal Yahav. 2022. HeBERT and HebEMO: A hebrew BERT model and a tool for polarity analysis and emotion recognition. INFORMS Journal on Data Science, 1(1):81–95. [↗](https://doi.org/10.1287/ijds.2022.0016) [our card](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2021-chriqui-hebert-hebemo-hebrew-bert-model/)
4. Amir DN Cohen, Hilla Merhav Fine, Yoav Goldberg, and Reut Tsarfaty. 2023. Heq: a large and diverse hebrew reading comprehension benchmark. [find](https://scholar.google.com/scholar?q=Amir+DN+Cohen%2C+Hilla+Merhav+Fine%2C+Yoav+Goldberg%2C+and+Reut+Tsarfaty%2E+2023%2E+Heq%3A+a+large+and+diverse+hebrew+reading+comprehension+benchmark%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/findings-2023-cohen-heq-large-diverse-hebrew-reading/)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E)
6. Matan Eyal, Hila Noga, Roee Aharoni, Idan Szpektor, and Reut Tsarfaty. 2022. Multilingual sequence-to-sequence models for hebrew nlp. [find](https://scholar.google.com/scholar?q=Matan+Eyal%2C+Hila+Noga%2C+Roee+Aharoni%2C+Idan+Szpektor%2C+and+Reut+Tsarfaty%2E+2022%2E+Multilingual+sequence%2Dto%2Dsequence+models+for+hebrew+nlp%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/findings-2023-eyal-multilingual-sequence-sequence-models-hebrew/)
7. Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2023. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. [find](https://scholar.google.com/scholar?q=Eylon+Gueta%2C+Avi+Shmidman%2C+Shaltiel+Shmidman%2C+Cheyn+Shmuel+Shmidman%2C+Joshua+Guedalia%2C+Moshe+Koppel%2C+Dan+Bareket%2C+Amit+Seker%2C+and+Reut+Tsarfaty%2E+2023%2E+Large+pre%2Dtrained+models+with+extra%2Dlarge+vocabula) [our card](https://md.mova.elamur.ai/free/en/docs/papers/hbo-2022-gueta-large-pre-trained-models-extra/)
8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E)
9. Shoval Sade, Amit Seker, and Reut Tsarfaty. 2018. The Hebrew Universal Dependency treebank: Past present and future. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 133–143, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shoval+Sade%2C+Amit+Seker%2C+and+Reut+Tsarfaty%2E+2018%2E+The+Hebrew+Universal+Dependency+treebank%3A+Past+present+and+future%2E+In+Proceedings+of+the+Second+Workshop+on+Universal+Dependencies+%28UDW+2018%29%2C+pages+1) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2018-sade-hebrew-universal-dependency-treebank-past/)
10. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert:a hebrew large pre-trained language model to start-off your hebrew nlp application with. [find](https://scholar.google.com/scholar?q=Amit+Seker%2C+Elron+Bandel%2C+Dan+Bareket%2C+Idan+Brusilovsky%2C+Refael+Shaked+Greenfeld%2C+and+Reut+Tsarfaty%2E+2021%2E+Alephbert%3Aa+hebrew+large+pre%2Dtrained+language+model+to+start%2Doff+your+hebrew+nlp+application+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2021-seker-alephbert-hebrew-large-pre-trained/)
11. Vitaly Shalumov and Harel Haskey. 2023. Hero: Roberta and longformer hebrew language models. arXiv:2304.11077. [↗](https://arxiv.org/abs/2304.11077) [our card](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2023-shalumov-hero-roberta-longformer-hebrew-language/)
12. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast wordpiece tokenization. [find](https://scholar.google.com/scholar?q=Xinying+Song%2C+Alex+Salcianu%2C+Yang+Song%2C+Dave+Dopson%2C+and+Denny+Zhou%2E+2021%2E+Fast+wordpiece+tokenization%2E)
13. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Noah+Constant%2C+Adam+Roberts%2C+Mihir+Kale%2C+Rami+Al%2DRfou%2C+Aditya+Siddhant%2C+Aditya+Barua%2C+and+Colin+Raffel%2E+2021%2E+mT5%3A+A+massively+multilingual+pre%2Dtrained+text%2Dto%2Dtext+transformer%2E+In+Procee)
14. Amir Zeldes, Nick Howell, Noam Ordan, and Yifat Ben Moshe. 2022. A second wave of ud hebrew treebanking and cross-domain parsing. [find](https://scholar.google.com/scholar?q=Amir+Zeldes%2C+Nick+Howell%2C+Noam+Ordan%2C+and+Yifat+Ben+Moshe%2E+2022%2E+A+second+wave+of+ud+hebrew+treebanking+and+cross%2Ddomain+parsing%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2022-zeldes-second-wave-ud-hebrew-treebanking/)