[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/langstream-2023-shmidman-introducing-dictalm-large-generative-language/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/) / Introducing DictaLM -- A Large Generative Language Model for Modern Hebrew

# Introducing DictaLM -- A Large Generative Language Model for Modern Hebrew

**Authors:** [Shaltiel Shmidman](https://md.mova.elamur.ai/free/en/docs/papers/authors/shaltiel-shmidman/); [Avi Shmidman](https://md.mova.elamur.ai/free/en/docs/papers/authors/avi-shmidman/); Amir David Nissan Cohen; [Moshe Koppel](https://md.mova.elamur.ai/free/en/docs/papers/authors/moshe-koppel/) · **Year:** 2023 · **Venue:** arXiv:2309.14568 **Link:** [arxiv.org/abs/2309.14568](https://arxiv.org/abs/2309.14568) · library `langstream-2023-shmidman-introducing-dictalm-large-generative-language` · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**0** citations](https://openalex.org/works?filter=cites:W4387113842) openalex · 2026-10-18[**1** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/langstream-2023-shmidman-introducing-dictalm-large-generative-language/) of the 140673 papers whose bibliographies we read[arXiv:2309.14568](https://openalex.org/S4306400194)[licence](http://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://arxiv.org/abs/2309.14568)
- [PDF](https://arxiv.org/pdf/2309.14568)
- [arXiv 2309.14568](https://arxiv.org/abs/2309.14568)
- [OpenAlex](https://openalex.org/W4387113842)

**Authors (4)**

1. [Shaltiel Shmidman](https://openalex.org/A5041361841) h 3 · 30 citations · 21 works
2. [Avi Shmidman](https://openalex.org/A5013212409) h 5 · 122 citations · 53 works
3. [Amir David Nissan Cohen](https://openalex.org/A5101249520) h 0 · 0 citations · 4 works
4. [Moshe Koppel](https://openalex.org/A5019908505) h 37 · 7124 citations · 150 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The authors release DictaLM, a 7B-parameter generative model for Modern Hebrew, trained on about 7.5 billion tokens. Most of the data is a cleaned HeDC4 (80%), the rest is news, blogs, subtitles and novels. The cleaning removes about 15% of HeDC4 with histograms, gibberish detectors and a perplexity filter from a Hebrew BERT. The corpus keeps only English and Hebrew words, and every other language becomes one `<foreign>` token. A byte-pair tokenizer with 56,000 entries gives about 1.3 tokens per word. The base model is then instruction-tuned on Hebrew QA data and translated MPT-Instruct. A second base model, DictaLM-Rab, is trained with half of each batch drawn from 1.2B tokens of Rabbinic Hebrew. The paper gives no benchmark scores; the authors call the release an alpha.

## Ideas

- I1. Map every non-target-language span to one `<foreign>` marker before parsing. `tools/langseg` already cuts a mixed text into language segments, so its non-Hebrew, non-English segments could be replaced by a marker, and the parser would see a clean sentence with a known gap.
- I2. A register sampler for [11. Mishnaic Hebrew (hbo-x-mishnaic)](https://md.mova.elamur.ai/free/en/docs/languages/hbo/11-mishnaic/): draw a fixed share (for example 50%) of Mishnaic or Rabbinic sentences into a training or test mix, with oversampling, and measure the effect on a held-out check. Run it twice, as the determinism rule asks, before claiming any gain.
- I3. A cheap corpus filter: drop sentences a small statistical model finds very surprising (a character or word n-gram model in Rust, in place of the paper's BERT). Measure how much text goes, and check a sample of the removed sentences.

## For Mova

I1 goes into `tools/langseg` and the `prag` input stage. I2 goes into [11. Mishnaic Hebrew (hbo-x-mishnaic)](https://md.mova.elamur.ai/free/en/docs/languages/hbo/11-mishnaic/) as a test-set design. I3 is a corpus step in the Hebrew data pipeline, not a parser change.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 2309.14568](https://arxiv.org/abs/2309.14568)

```
@misc{shmidman2023introducing,
  title = {{Introducing DictaLM -- A Large Generative Language Model for Modern Hebrew}},
  author = {Shmidman, Shaltiel and Shmidman, Avi and Cohen, Amir David Nissan and Koppel, Moshe},
  year = {2023},
  eprint = {2309.14568},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.14568}
}
```

- [← previous HeRo: RoBERTa and Longformer Hebrew Language Models](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2023-shalumov-hero-roberta-longformer-hebrew-language/)
- [next → Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2024-goldin-knesset-dictabert-hebrew-language-model/)
- [↑ back to Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/)

## References

11 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from Modern Hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242–2252, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [https://github.com/NVIDIA/NeMo](https://github.com/NVIDIA/NeMo) https://github.com/NVIDIA/ TransformerEngine 8 https://www.sefaria.org.il/ 9 https://library.dicta.org.il/ [↗](https://github.com/NVIDIA/NeMo) [our card](https://md.mova.elamur.ai/free/en/docs/papers/coling-2018-amram-representations-architectures-neural-sentiment-analysis/)
2. Dan Bareket and Reut Tsarfaty. 2021. Neural Modeling for Named Entities and Morphology (NEMO2). Transactions of the Association for Computational Linguistics, 9:909–928. [find](https://scholar.google.com/scholar?q=Dan+Bareket+and+Reut+Tsarfaty%2E+2021%2E+Neural+Modeling+for+Named+Entities+and+Morphology+%28NEMO2%29%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+9%3A909%E2%80%93928%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/tacl-2021-bareket-neural-modeling-named-entities-morphology/)
3. Amir DN Cohen, Hilla Merhav Fine, Yoav Goldberg, and Reut Tsarfaty. 2023. Heq: a large and diverse hebrew reading comprehension benchmark. [find](https://scholar.google.com/scholar?q=Amir+DN+Cohen%2C+Hilla+Merhav+Fine%2C+Yoav+Goldberg%2C+and+Reut+Tsarfaty%2E+2023%2E+Heq%3A+a+large+and+diverse+hebrew+reading+comprehension+benchmark%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/findings-2023-cohen-heq-large-diverse-hebrew-reading/)
4. Dan Hendrycks and Kevin Gimpel. 2023. Gaussian error linear units (gelus). [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+2023%2E+Gaussian+error+linear+units+%28gelus%29%2E)
5. Omri Keren and Omer Levy. 2021. Parashoot: A hebrew question answering dataset. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 106–112. [find](https://scholar.google.com/scholar?q=Omri+Keren+and+Omer+Levy%2E+2021%2E+Parashoot%3A+A+hebrew+question+answering+dataset%2E+In+Proceedings+of+the+3rd+Workshop+on+Machine+Reading+for+Question+Answering%2C+pages+106%E2%80%93112%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2021-keren-parashoot-hebrew-question-answering-dataset/)
6. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. CoRR, abs/1508.07909. [↗](https://arxiv.org/abs/1508.07909)
7. Vitaly Shalumov and Harel Haskey. 2023. Hero: Roberta and longformer hebrew language models. arXiv:2304.11077. [↗](https://arxiv.org/abs/2304.11077) [our card](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2023-shalumov-hero-roberta-longformer-hebrew-language/)
8. Noam Shazeer. 2020. Glu variants improve transformer. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2020%2E+Glu+variants+improve+transformer%2E)
9. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2022. Roformer: Enhanced transformer with rotary position embedding. [find](https://scholar.google.com/scholar?q=Jianlin+Su%2C+Yu+Lu%2C+Shengfeng+Pan%2C+Ahmed+Murtadha%2C+Bo+Wen%2C+and+Yunfeng+Liu%2E+2022%2E+Roformer%3A+Enhanced+transformer+with+rotary+position+embedding%2E)
10. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762. [↗](https://arxiv.org/abs/1706.03762)
11. Charles Welch, Rada Mihalcea, and Jonathan K. Kummerfeld. 2020. Improving low compute language modeling with in-domain embedding initialisation. [find](https://scholar.google.com/scholar?q=Charles+Welch%2C+Rada+Mihalcea%2C+and+Jonathan+K%2E+Kummerfeld%2E+2020%2E+Improving+low+compute+language+modeling+with+in%2Ddomain+embedding+initialisation%2E)