[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2026-stoupak-bereshit-ancient-hebrew-model-based/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Ancient Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/hbo/) / BEReshiT: an Ancient Hebrew Model based on DictaBERT

# BEReshiT: an Ancient Hebrew Model based on DictaBERT

**Authors:** [Iglika Nikolova-Stoupak](https://md.mova.elamur.ai/free/en/docs/papers/authors/iglika-nikolova-stoupak/); [Maxime Amblard](https://md.mova.elamur.ai/free/en/docs/papers/authors/maxime-amblard/); [Frédérique Rey](https://md.mova.elamur.ai/free/en/docs/papers/authors/frederique-rey/) · **Year:** 2026 · **Venue:** LT4HALA 2026 @ LREC 2026 **Link:** [aclanthology.org/2026.lt4hala-1.7](https://aclanthology.org/2026.lt4hala-1.7/) · library `aclbulk-2026-stoupak-bereshit-ancient-hebrew-model-based` · **Licence:** [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — summary only

[**0** citations](https://openalex.org/works?filter=cites:W7166570842) openalex · 2026-10-18[Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026](https://aclanthology.org/volumes/2026.lt4hala-1/)[licence](https://creativecommons.org/licenses/by-nc/4.0/)

- [Publisher's page](https://aclanthology.org/2026.lt4hala-1.7/)
- [PDF](https://aclanthology.org/2026.lt4hala-1.7.pdf)
- [DOI 10.63317/4j3oje5q7bcd](https://doi.org/10.63317/4j3oje5q7bcd)
- [OpenAlex](https://openalex.org/W7166570842)

**Authors (3)**

1. Iglika Nikolova-Stoupak
2. [Maxime Amblard](https://openalex.org/A5066715241) [ORCID](https://orcid.org/0000-0002-2924-475X) · h 7 · 174 citations · 210 works
3. [Frédérique Rey](https://openalex.org/A5115404782) h 1 · 5 citations · 29 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The authors fine-tune DictaBERT, a BERT model for Modern Hebrew, on the Hebrew Bible (about 373,000 words) and on clean Dead Sea Scrolls lines (about 68,000 words), to make an Ancient Hebrew model, BEReshiT, and a morphology submodel, BEReshiT-morph. They train twelve versions that differ in how many layers are frozen; the best freezes nine and trains three. Evaluation is a masked-word cloze on a 200-verse set: half is spread over genres, half is built around five phenomena absent or rare in Modern Hebrew (wayyiqtol chains, infinitive absolute with a finite verb, verb-subject-object order, suffixed possessives, particles). BEReshiT beats DictaBERT by 0.02 to 0.09 in accuracy once BEREL and GPT-4o, which may have seen the biblical text, are set aside. On the DSS-only items their accuracy drops to 0.18 (BEREL) and 0.09 (GPT). The morphology submodel, with prefixes split off first, reaches mean macro-F1 0.76 and 0.97 for POS.

## Ideas

- I1. A phenomenon-stratified test for `tools/hbo`: use the paper's design (verses grouped by wayyiqtol chains, VSO, suffixed possessives, particles, infinitive absolute with finite verb) as a held-out set, and report the parser's and segmenter's scores per phenomenon as well as overall.
- I2. A contamination rule: the authors drop BEREL and GPT from the headline comparison because they have seen the text. For Mova, the segmenter tables learned from MACULA must not be scored on the verses they were learned from; hold verses out before any `tools/hbo` number is written down.
- I3. Prefix-first morphology: they split prefixes before tagging, and report POS macro-F1 on the same split. Mova can report per-feature macro-F1 for POS and prefix tags on one split, so the numbers can be compared with theirs without a neural model.

## For Mova

I1 and I3 are measures for `tools/hbo` (its test suite and its report); the particles and possessive items feed [Pronouns, particles and prepositions](https://md.mova.elamur.ai/free/en/docs/languages/hbo/05-pronouns-particles/), and the wayyiqtol item feeds [The verb](https://md.mova.elamur.ai/free/en/docs/languages/hbo/06-verb/). I2 is a rule for how `tools/hbo` tables are built and tested, recorded in [13. Biblical Hebrew in Mova: the tools](https://md.mova.elamur.ai/free/en/docs/languages/hbo/13-mova-tools/). The neural model itself is not a Mova target; only the data design and the measures are.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.63317/4j3oje5q7bcd](https://doi.org/10.63317/4j3oje5q7bcd)

```
@inproceedings{nikolovastoupak2026bereshit,
  title = {{BEReshiT: an Ancient Hebrew Model based on DictaBERT}},
  author = {Nikolova-Stoupak, Iglika and Amblard, Maxime and Rey, Frédérique},
  booktitle = {Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026},
  year = {2026},
  doi = {10.63317/4j3oje5q7bcd},
  url = {https://doi.org/10.63317/4j3oje5q7bcd}
}
```

- [← previous Domain-Aware Error Correction for Citation NER in Medieval Hebrew Responsa](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2026-liebeskind-domain-aware-error-correction-citation/)
- [next → Modeling the Dalet Clitic in Historical Hebrew Texts: A New Prefix-Segmented BERT Model and Stylistic Analysis](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2026-tal-modeling-dalet-clitic-historical-hebrew/)
- [↑ back to Ancient Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/hbo/)

## References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 1955–2009. Discoveries in the Judean Desert. Clarendon Press, Oxford. [https://www.qumran-digital.org/](https://www.qumran-digital.org/) [↗](https://www.qumran-digital.org/)
2. David Bamman and Patrick J. Burns. 2020. Latin bert: A contextual language model for classical philology. [find](https://scholar.google.com/scholar?q=David+Bamman+and+Patrick+J%2E+Burns%2E+2020%2E+Latin+bert%3A+A+contextual+language+model+for+classical+philology%2E)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171–4186. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
4. Simon Gabay, Pedro Ortiz Suarez, Alexandre Bartz, Alix Chagué, Rachel Bawden, Philippe Gambette, and Benoît Sagot. 2022. From freem to d’alembert: a large corpus and a language model for early modern french. [find](https://scholar.google.com/scholar?q=Simon+Gabay%2C+Pedro+Ortiz+Suarez%2C+Alexandre+Bartz%2C+Alix+Chagu%C3%A9%2C+Rachel+Bawden%2C+Philippe+Gambette%2C+and+Beno%C3%AEt+Sagot%2E+2022%2E+From+freem+to+d%E2%80%99alembert%3A+a+large+corpus+and+a+language+model+for+early+modern+)
5. Ayaka Harigai, Oshitaka Toyama, Mitsutoshi Nagano, Mirei Abe, Masahiro Kawabata, Li Li, Jin Yamamura, and Kei Takase. 2025. Response accuracy of gpt-4 across languages: insights from diagnostic radiology questions. Japanese Journal of Radiology. [find](https://scholar.google.com/scholar?q=Ayaka+Harigai%2C+Oshitaka+Toyama%2C+Mitsutoshi+Nagano%2C+Mirei+Abe%2C+Masahiro+Kawabata%2C+Li+Li%2C+Jin+Yamamura%2C+and+Kei+Takase%2E+2025%2E+Response+accuracy+of+gpt%2D4+across+languages%3A+insights+from+diagnostic+radiol)
6. Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2021. A survey on recent approaches for natural language processing in low-resource scenarios. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2545–2568. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Michael+A%2E+Hedderich%2C+Lukas+Lange%2C+Heike+Adel%2C+Jannik+Str%C3%B6tgen%2C+and+Dietrich+Klakow%2E+2021%2E+A+survey+on+recent+approaches+for+natural+language+processing+in+low%2Dresource+scenarios%2E+In+Proceedings+of+th)
7. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Christopher+D%2E+Manning%2E+2019%2E+A+structural+probe+for+finding+syntax+in+word+representations%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+)
8. Aaron D. Hornkohl. 2014. Ancient Hebrew Periodization and the Language of the Book of Jeremiah: The Case for a Sixth-Century Date of Composition, volume 74 of Studies in Semitic Languages and Linguistics. Brill, Leiden. [find](https://scholar.google.com/scholar?q=Aaron+D%2E+Hornkohl%2E+2014%2E+Ancient+Hebrew+Periodization+and+the+Language+of+the+Book+of+Jeremiah%3A+The+Case+for+a+Sixth%2DCentury+Date+of+Composition%2C+volume+74+of+Studies+in+Semitic+Languages+and+Linguist)
9. Avi Hurwitz. 2014. A Concise Lexicon of Late Biblical Hebrew: Linguistic Innovations in the Writings of the Second Temple Period. Vetus Testamentum Supplements. Brill, Leiden. [find](https://scholar.google.com/scholar?q=Avi+Hurwitz%2E+2014%2E+A+Concise+Lexicon+of+Late+Biblical+Hebrew%3A+Linguistic+Innovations+in+the+Writings+of+the+Second+Temple+Period%2E+Vetus+Testamentum+Supplements%2E+Brill%2C+Leiden%2E)
10. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ganesh+Jawahar%2C+Beno%C3%AEt+Sagot%2C+and+Djam%C3%A9+Seddah%2E+2019%2E+What+does+BERT+learn+about+the+structure+of+language%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
11. Pratik M. Joshi, Sebastin Santy, Ameya Godbole, Kalika Bali, and Chandra Sekhar Mukherjee. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pratik+M%2E+Joshi%2C+Sebastin+Santy%2C+Ameya+Godbole%2C+Kalika+Bali%2C+and+Chandra+Sekhar+Mukherjee%2E+2020%2E+The+state+and+fate+of+linguistic+diversity+and+inclusion+in+the+nlp+world%2E+In+Proceedings+of+the+58th+A)
12. Geoffrey Khan, Shmuel Bolozky, Steven E. Fassberg, Gary A. Rendsburg, Aaron D. Rubin, Ora R. Schwarzwald, and Tamar Zewi, editors. 2013. Encyclopedia of Hebrew Language and Linguistics, 1 edition, volume 1-4. Brill, Leiden. [find](https://scholar.google.com/scholar?q=Geoffrey+Khan%2C+Shmuel+Bolozky%2C+Steven+E%2E+Fassberg%2C+Gary+A%2E+Rendsburg%2C+Aaron+D%2E+Rubin%2C+Ora+R%2E+Schwarzwald%2C+and+Tamar+Zewi%2C+editors%2E+2013%2E+Encyclopedia+of+Hebrew+Language+and+Linguistics%2C+1+edition%2C+vol)
13. Piroska Lendvai, Uwe Reichel, Anna Jouravel, Achim Rabus, and Elena Renje. 2023. Domain-adapting BERT for attributing manuscript, century and region in pre-Modern Slavic texts. In Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change, pages 15–21, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Piroska+Lendvai%2C+Uwe+Reichel%2C+Anna+Jouravel%2C+Achim+Rabus%2C+and+Elena+Renje%2E+2023%2E+Domain%2Dadapting+BERT+for+attributing+manuscript%2C+century+and+region+in+pre%2DModern+Slavic+texts%2E+In+Proceedings+of+the+4)
14. Josh McGiff and Nikola S. Nikolov. 2025. Overcoming data scarcity in generative language modelling for low-resource languages: A systematic review. arXiv preprint. [find](https://scholar.google.com/scholar?q=Josh+McGiff+and+Nikola+S%2E+Nikolov%2E+2025%2E+Overcoming+data+scarcity+in+generative+language+modelling+for+low%2Dresource+languages%3A+A+systematic+review%2E+arXiv+preprint%2E)
15. Ricardo Muñoz Sánchez, David Alfter, Simon Dobnik, Maria Irena Szawerna, and Elena Volodina. 2024. Jingle BERT, jingle BERT, frozen all the way: Freezing layers to identify CEFR levels of second language learners using BERT. In Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning, pages 137–152, Rennes, France. LiU Electronic Press. [find](https://scholar.google.com/scholar?q=Ricardo+Mu%C3%B1oz+S%C3%A1nchez%2C+David+Alfter%2C+Simon+Dobnik%2C+Maria+Irena+Szawerna%2C+and+Elena+Volodina%2E+2024%2E+Jingle+BERT%2C+jingle+BERT%2C+frozen+all+the+way%3A+Freezing+layers+to+identify+CEFR+levels+of+second+langu)
16. Iglika Nikolova-Stoupak, Maxime Amblard, Sophie Robert-Hayek, and Frédérique Rey. 2025. A classifier of word-level variants in witnesses of biblical Hebrew manuscripts. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21313–21329, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Iglika+Nikolova%2DStoupak%2C+Maxime+Amblard%2C+Sophie+Robert%2DHayek%2C+and+Fr%C3%A9d%C3%A9rique+Rey%2E+2025%2E+A+classifier+of+word%2Dlevel+variants+in+witnesses+of+biblical+Hebrew+manuscripts%2E+In+Findings+of+the+Association+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/findings-2025-stoupak-classifier-word-level-variants-witnesses/)
17. OpenAI. 2024. Gpt-4o system [https://openai.com/index/gpt-4o-system-card/](https://openai.com/index/gpt-4o-system-card/). [↗](https://openai.com/index/gpt-4o-system-card/)
18. Robert Rezetko and Ian Young. 2014. Historical Linguistics and Biblical Hebrew: Steps toward an Integrated Approach, volume 9 of Ancient Near East Monographs. SBL Press, Atlanta. [find](https://scholar.google.com/scholar?q=Robert+Rezetko+and+Ian+Young%2E+2014%2E+Historical+Linguistics+and+Biblical+Hebrew%3A+Steps+toward+an+Integrated+Approach%2C+volume+9+of+Ancient+Near+East+Monographs%2E+SBL+Press%2C+Atlanta%2E)
19. Stefan Schweter, Luisa März, Katharina Schmid, and Erion Çano. 2022. hmbert: Historical multilingual language models for named entity recognition. [find](https://scholar.google.com/scholar?q=Stefan+Schweter%2C+Luisa+M%C3%A4rz%2C+Katharina+Schmid%2C+and+Erion+%C3%87ano%2E+2022%2E+hmbert%3A+Historical+multilingual+language+models+for+named+entity+recognition%2E)
20. Avi Shmidman, Joshua Guedalia, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Eli Handel, and Moshe Koppel. 2022. Introducing berel: Bert embeddings for rabbinic-encoded language. [find](https://scholar.google.com/scholar?q=Avi+Shmidman%2C+Joshua+Guedalia%2C+Shaltiel+Shmidman%2C+Cheyn+Shmuel+Shmidman%2C+Eli+Handel%2C+and+Moshe+Koppel%2E+2022%2E+Introducing+berel%3A+Bert+embeddings+for+rabbinic%2Dencoded+language%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/hbo-2022-shmidman-introducing-berel-bert-embeddings-rabbinic/)
21. Avi Shmidman, Ometz Shmidman, Hillel Gershuni, and Moshe Koppel. 2024a. MsBERT: A new model for the reconstruction of lacunae in Hebrew manuscripts. In Proceedings of the 1st Workshop on Machine Learning for Ancient Languages (ML4AL 2024), pages 13–18, Hybrid in Bangkok, Thailand and online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2024.ml4al-1.2) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2024-shmidman-msbert-new-model-reconstruction-lacunae/)
22. Shaltiel Shmidman, Avi Shmidman, Amir DN Cohen, and Moshe Koppel. 2024b. Adapting llms to hebrew: Unveiling dictalm 2.0 with enhanced vocabulary and instruction capabilities. [find](https://scholar.google.com/scholar?q=Shaltiel+Shmidman%2C+Avi+Shmidman%2C+Amir+DN+Cohen%2C+and+Moshe+Koppel%2E+2024b%2E+Adapting+llms+to+hebrew%3A+Unveiling+dictalm+2%2E0+with+enhanced+vocabulary+and+instruction+capabilities%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/arxivrecent-2024-shmidman-adapting-llms-hebrew-unveiling-dictalm/)
23. Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. Dictabert: A state-of-the-art BERT suite for modern Hebrew. [find](https://scholar.google.com/scholar?q=Shaltiel+Shmidman%2C+Avi+Shmidman%2C+and+Moshe+Koppel%2E+2023%2E+Dictabert%3A+A+state%2Dof%2Dthe%2Dart+BERT+suite+for+modern+Hebrew%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/arxivrecent-2023-shmidman-dictabert-state-art-bert-suite/)
24. Elad Shvartz, Leah Attal, Omri Zur, Zaki Nujeidat, Gilad Plopsky, and Daniel Bahir. 2025. Bilingual comparison of the performance of gpt-4o and gpt-4 on multilingual medical questions. European Journal of Radiology. [find](https://scholar.google.com/scholar?q=Elad+Shvartz%2C+Leah+Attal%2C+Omri+Zur%2C+Zaki+Nujeidat%2C+Gilad+Plopsky%2C+and+Daniel+Bahir%2E+2025%2E+Bilingual+comparison+of+the+performance+of+gpt%2D4o+and+gpt%2D4+on+multilingual+medical+questions%2E+European+Journa)
25. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019%2E+Bert+rediscovers+the+classical+nlp+pipeline%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4593%E2%80%934)
26. Wenhao Zhu, Hongyu Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models. In Findings of the Association for Computational Linguistics: NAACL 2024. card. [find](https://scholar.google.com/scholar?q=Wenhao+Zhu%2C+Hongyu+Liu%2C+Qingxiu+Dong%2C+Jingjing+Xu%2C+Shujian+Huang%2C+Lingpeng+Kong%2C+Jiajun+Chen%2C+and+Lei+Li%2E+2024%2E+Multilingual+machine+translation+with+large+language+models%2E+In+Findings+of+the+Associat)