[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/langstream-2024-goldin-knesset-dictabert-hebrew-language-model/) [home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/) / Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings # Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings **Authors:** [Gili Goldin](https://md.mova.elamur.ai/free/en/docs/papers/authors/gili-goldin/); Shuly Wintner · **Year:** 2024 · **Venue:** arXiv:2407.20581 **Link:** [arxiv.org/abs/2407.20581](https://arxiv.org/abs/2407.20581) · library `langstream-2024-goldin-knesset-dictabert-hebrew-language-model` · **Licence:** [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) — summary only, never carried into a release [**0** citations](https://openalex.org/works?filter=cites:W4401203007) openalex · 2026-10-18[**1** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/langstream-2024-goldin-knesset-dictabert-hebrew-language-model/) of the 140673 papers whose bibliographies we read[arXiv:2407.20581](https://openalex.org/S4306400194)[licence](http://creativecommons.org/licenses/by-sa/4.0/) - [Publisher's page](https://arxiv.org/abs/2407.20581) - [PDF](https://arxiv.org/pdf/2407.20581) - [arXiv 2407.20581](https://arxiv.org/abs/2407.20581) - [OpenAlex](https://openalex.org/W4401203007) **Authors (4)** 1. [Gili Goldin](https://openalex.org/A5062160732) [Semantic Scholar](https://www.semanticscholar.org/author/51511114) · h 1 · 26 citations · 8 works 2. Shuly Wintner Department of Computer Science 3. University of Haifa 4. Israel) Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart. ## What it is about The authors continue masked-language-model training of DictaBERT on the Knesset Corpus, about 384 million tokens of plenary and committee protocols. They split the data 80/10/10 and use 256-token chunks with 15% masking. Training runs two epochs at learning rate 1e-4 in fp16. On a test set of about 3.2 million sentences the perplexity falls from 22.87 (base DictaBERT) to 6.60. Top-1 masked-token accuracy rises from 48.02% to 52.55%, top-2 from 58.60% to 63.07%, and top-5 from 68.98% to 73.59%. The authors note that the corpus carries the political and social biases of parliamentary debate. ## Ideas - I1. A register cloze test for `treebanks/he/iahltknesset`: hide one word per sentence, ask a candidate model which word fits, and report top-1, top-2 and top-5 agreement. This measures how well the register's lexicon is covered, and it is a check on [8. Lexicon and register](https://md.mova.elamur.ai/free/en/docs/languages/he/08-lexicon-register/). - I2. Domain adaptation measured by perplexity, as a small statistical model: a word n-gram model trained on the Knesset text, compared against a general Hebrew model on the same held-out sentences. The gain should be reported with the held-out split, not the training split. - I3. Record the bias caveat in the data card: any lexicon or frequency list taken from parliamentary text should carry a label saying it reflects that register and its politics. ## For Mova I1 is a test in `treebanks/he/iahltknesset`, with its result written into [8. Lexicon and register](https://md.mova.elamur.ai/free/en/docs/languages/he/08-lexicon-register/). I2 is an n-gram experiment for the `he` lexicon. I3 is a note in the `treebanks/he/` data notes. ## Cite The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 2407.20581](https://arxiv.org/abs/2407.20581) ``` @misc{goldin2024knesset, title = {{Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings}}, author = {Goldin, Gili and Science, Shuly Wintner Department of Computer and Haifa, University of and Israel)}, year = {2024}, eprint = {2407.20581}, archivePrefix = {arXiv}, url = {https://arxiv.org/abs/2407.20581} } ``` - [← previous Introducing DictaLM -- A Large Generative Language Model for Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2023-shmidman-introducing-dictalm-large-generative-language/) - [next → A Language-agnostic Model of Child Language Acquisition](https://md.mova.elamur.ai/free/en/docs/papers/langstream-2024-mahon-language-agnostic-model-child-language/) - [↑ back to Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/) ## References 4 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423) 2. Gili Goldin, Nick Howell, Noam Ordan, Ella Rabinovich, and Shuly Wintner. 2024. The Knesset Corpus: An annotated corpus of Hebrew parliamentary proceedings. ArXiv, abs/2405.18115. [↗](https://arxiv.org/abs/2405.18115) [our card](https://md.mova.elamur.ai/free/en/docs/papers/arxivrecent-2024-goldin-knesset-corpus-annotated-corpus-hebrew/) 3. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2017. Mixed precision training. CoRR, abs/1710.03740. [↗](https://arxiv.org/abs/1710.03740) 4. Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. DictaBERT: A state-of-the-art BERT suite for Modern Hebrew. This figure "committee_ttr.png" is available in "png" format from: [http://arxiv.org/ps/2407.20581v1](http://arxiv.org/ps/2407.20581v1) [↗](http://arxiv.org/ps/2407.20581v1) [our card](https://md.mova.elamur.ai/free/en/docs/papers/arxivrecent-2023-shmidman-dictabert-state-art-bert-suite/)