[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2021-rubinstein-nlp-dh-pipeline-transfer-learning/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/) / NLP in the DH pipeline: Transfer-learning to a Chronolect

# NLP in the DH pipeline: Transfer-learning to a Chronolect

**Authors:** [Aynat Rubinstein](https://md.mova.elamur.ai/free/en/docs/papers/authors/aynat-rubinstein/); [Avi Shmidman](https://md.mova.elamur.ai/free/en/docs/papers/authors/avi-shmidman/) · **Year:** 2021 · **Venue:** Proceedings of the Workshop on Natural Language Processing for Digital Humanities (NLP4DH), pages 106–110, co-located with ICON 2021 **Link:** [aclanthology.org/2021.nlp4dh-1.12](https://aclanthology.org/2021.nlp4dh-1.12/) · library `aclbulk-2021-rubinstein-nlp-dh-pipeline-transfer-learning` · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[Proceedings of the Workshop on Natural Language Processing for Digital Humanities](https://aclanthology.org/volumes/2021.nlp4dh-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2021.nlp4dh-1.12/)
- [PDF](https://aclanthology.org/2021.nlp4dh-1.12.pdf)

**Authors (2)**

1. Aynat Rubinstein
2. Avi Shmidman

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

The authors build a diacritizer (vowel marks) for Emergent Modern Hebrew, the Hebrew of the early twentieth century, using a hand-vocalised goldset of 144,446 words from the Jerusalem Corpus of Emergent Modern Hebrew (129,453 words of literature, 14,993 of ephemera). They use 120K words for training and 11K for testing. They compare an LSTM trained from scratch at four sizes (50K to 120K words) with fine-tuning Dicta's LSTM for Modern Hebrew, and also with fine-tuning a model for Rabbinic Hebrew. Fine-tuning from Modern Hebrew is better at every size; unconstrained word accuracy rises from 78.30% with no target-chronolect training to 81.98% at 120K, while training from scratch reaches 77.29%. A wordlist that restricts each word to known valid vocalisations adds 6 to 11 points in every setting (87.38% for the best fine-tuned model). Transfer helps only within the genre of the training data: on the ephemera test set, fine-tuning lowers accuracy from 80.66% to 74.01%.

## Ideas

- I1. Constrain by a wordlist: in unpointed mode, restrict each token's candidates to forms in Mova's lexicon, then score them. The paper's largest single gain (6 to 11 points) comes from this lookup step, not from a bigger model. It fits the unpointed mode of `tools/hbo`, with the `lex` table as the list.
- I2. A learning curve as a planning tool: each 20K to 25K extra words gave about half a point to one and a half points. Use the same stepwise test before annotating any new Hebrew or Aramaic variety (for example `languages/tmr/` or `languages/jpa/`) to decide how much gold to build first.
- I3. A held-out genre test: the transfer failed on the genre not seen in training. Keep a held-out genre in every test of a model trained on Biblical or Mishnaic text, so that a score is not read as success on a new genre.

## For Mova

I1 goes into `tools/hbo` (unpointed mode) and is documented in [The lexicon](https://md.mova.elamur.ai/free/en/docs/languages/hbo/10-lexicon/). I2 is a planning note for `languages/tmr/` and `languages/jpa/`. I3 goes into [11. Mishnaic Hebrew (hbo-x-mishnaic)](https://md.mova.elamur.ai/free/en/docs/languages/hbo/11-mishnaic/) and [12. Biblical, Mishnaic and Modern Hebrew compared](https://md.mova.elamur.ai/free/en/docs/languages/hbo/12-biblical-mishnaic-modern/).

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{rubinstein2021nlp,
  title = {{NLP in the DH pipeline: Transfer-learning to a Chronolect}},
  author = {Rubinstein, Aynat and Shmidman, Avi},
  booktitle = {Proceedings of the Workshop on Natural Language Processing for Digital Humanities},
  year = {2021},
  url = {https://aclanthology.org/2021.nlp4dh-1.12/}
}
```

- [← previous Semantically Motivated Hebrew Verb-Noun Multi-Word Expressions Identification](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2016-liebeskind-semantically-motivated-hebrew-verb-noun/)
- [next → Hebrew Psychological Lexicons](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2021-shapira-hebrew-psychological-lexicons/)
- [↑ back to Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/)

## References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alistair Baron and Paul Rayson. 2008. VARD2: a tool for dealing with spelling variation in historical corpora. Postgraduate Conference in Corpus Linguistics ; Conference date: 22-05-2008. [find](https://scholar.google.com/scholar?q=Alistair+Baron+and+Paul+Rayson%2E+2008%2E+VARD2%3A+a+tool+for+dealing+with+spelling+variation+in+historical+corpora%2E+Postgraduate+Conference+in+Corpus+Linguistics+%3B+Conference+date%3A+22%2D05%2D2008%2E)
2. Ze’ev Ben-Hayyim. 1992. The struggle for a language. The Academy of the Hebrew Language, Jerusalem. In Hebrew. [find](https://scholar.google.com/scholar?q=Ze%E2%80%99ev+Ben%2DHayyim%2E+1992%2E+The+struggle+for+a+language%2E+The+Academy+of+the+Hebrew+Language%2C+Jerusalem%2E+In+Hebrew%2E)
3. Haim Blanc. 1968. The Israeli koine as an emergent national standard. In Joshua A. Fishman, editor, Language problems of developing nations, pages 237–251. Wiley, New York. [find](https://scholar.google.com/scholar?q=Haim+Blanc%2E+1968%2E+The+Israeli+koine+as+an+emergent+national+standard%2E+In+Joshua+A%2E+Fishman%2C+editor%2C+Language+problems+of+developing+nations%2C+pages+237%E2%80%93251%2E+Wiley%2C+New+York%2E)
4. Yaacov Choueka and Yoni Neeman. 1995. Nakdan-text,(an in-context text-vocalizer for modern hebrew). In BISFAI-95, The Fifth Bar Ilan Symposium for Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Yaacov+Choueka+and+Yoni+Neeman%2E+1995%2E+Nakdan%2Dtext%2C%28an+in%2Dcontext+text%2Dvocalizer+for+modern+hebrew%29%2E+In+BISFAI%2D95%2C+The+Fifth+Bar+Ilan+Symposium+for+Artificial+Intelligence%2E)
5. Edit Doron, Malka Rappaport Hovav, Yael Reshef, and Moshe Taube. 2019a. Introduction. In Edit Doron, Malka Rappaport Hovav, Yael Reshef, and Moshe Taube, editors, Language Contact, Continuity and Change in the Genesis of Modern Hebrew, pages 1–31. John Benjamins, Amsterdam. [find](https://scholar.google.com/scholar?q=Edit+Doron%2C+Malka+Rappaport+Hovav%2C+Yael+Reshef%2C+and+Moshe+Taube%2E+2019a%2E+Introduction%2E+In+Edit+Doron%2C+Malka+Rappaport+Hovav%2C+Yael+Reshef%2C+and+Moshe+Taube%2C+editors%2C+Language+Contact%2C+Continuity+and+Chan)
6. Edit Doron, Malka Rappaport Hovav, Yael Reshef, and Moshe Taube, editors. 2019b. Language Contact, Continuity and Change in the Genesis of Modern Hebrew. John Benjamins, Amsterdam. [find](https://scholar.google.com/scholar?q=Edit+Doron%2C+Malka+Rappaport+Hovav%2C+Yael+Reshef%2C+and+Moshe+Taube%2C+editors%2E+2019b%2E+Language+Contact%2C+Continuity+and+Change+in+the+Genesis+of+Modern+Hebrew%2E+John+Benjamins%2C+Amsterdam%2E)
7. Ya’akov Gal. 2002. An hmm approach to vowel restoration in arabic and hebrew. In Proceedings of the ACL-02 workshop on Computational approaches to semitic languages, pages 1–7. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ya%E2%80%99akov+Gal%2E+2002%2E+An+hmm+approach+to+vowel+restoration+in+arabic+and+hebrew%2E+In+Proceedings+of+the+ACL%2D02+workshop+on+Computational+approaches+to+semitic+languages%2C+pages+1%E2%80%937%2E+Association+for+Computa) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2002-gal-hmm-approach-vowel-restoration-arabic/)
8. Eva Pettersson and Joakim Nivre. 2015. Improving verb phrase extraction from historical text by use of verb valency frames. In Proceedings of the 20th Nordic Conference of Computational Linguistics (NODALIDA 2015), pages 153– 161, Vilnius, Lithuania. Linköping University Electronic Press, Sweden. [find](https://scholar.google.com/scholar?q=Eva+Pettersson+and+Joakim+Nivre%2E+2015%2E+Improving+verb+phrase+extraction+from+historical+text+by+use+of+verb+valency+frames%2E+In+Proceedings+of+the+20th+Nordic+Conference+of+Computational+Linguistics+%28N)
9. Chaim Rabin. 1999. What was the revival of the Hebrew language? Linguistic Studies, pages 359–376. In Hebrew. [find](https://scholar.google.com/scholar?q=Chaim+Rabin%2E+1999%2E+What+was+the+revival+of+the+Hebrew+language%3F+Linguistic+Studies%2C+pages+359%E2%80%93376%2E+In+Hebrew%2E)
10. Yael Reshef. 2013. Revival of Hebrew: Grammatical structure and lexicon. In Geoffrey Khan, editor, Encyclopedia of Hebrew Language and Linguistics, volume 3, pages 397– 405. Brill, Leiden. [↗](https://doi.org/10.1163/2212-4241_ehll_ehll_com_00000702)
11. Yael Reshef. 2015. Hebrew in the Mandate period. The Academy of the Hebrew Language, Jerusalem. In Hebrew. [find](https://scholar.google.com/scholar?q=Yael+Reshef%2E+2015%2E+Hebrew+in+the+Mandate+period%2E+The+Academy+of+the+Hebrew+Language%2C+Jerusalem%2E+In+Hebrew%2E)
12. Yael Reshef. 2016. Written Hebrew of the revival generation as a distinct phase in the evolution of Modern Hebrew. Journal of Semitic Studies, 61(1):187–213. [↗](https://doi.org/10.1093/jss/fgv036)
13. Haiim Rosén. 1956. Ha-‘ivrit šelanu. Am Oved, Tel Aviv. In Hebrew. [find](https://scholar.google.com/scholar?q=Haiim+Rose%CC%81n%2E+1956%2E+Ha%2D%E2%80%98ivrit+s%CC%8Celanu%2E+Am+Oved%2C+Tel+Aviv%2E+In+Hebrew%2E)
14. Aynat Rubinstein. 2019. Historical corpora meet the digital humanities: the Jerusalem Corpus of Emergent Modern Hebrew. Language Resources and Evaluation, 53(4):807– 835. [↗](https://doi.org/10.1007/s10579-019-09458-4)
15. Gerold Schneider. 2020. Spelling normalisation of Late Modern English. In Merja Kytö and Erik Smitterberg, editors, Late Modern English: Novel encounters, pages 244—-268. John Benjamins, Amsterdam. [↗](https://doi.org/10.1075/slcs.214.11sch)
16. Avi Shmidman, Shaltiel Shmidman, Moshe Koppel, and Yoav Goldberg. 2020. Nakdan: Professional Hebrew diacritizer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 197–203, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Avi+Shmidman%2C+Shaltiel+Shmidman%2C+Moshe+Koppel%2C+and+Yoav+Goldberg%2E+2020%2E+Nakdan%3A+Professional+Hebrew+diacritizer%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguist) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2020-shmidman-nakdan-professional-hebrew-diacritizer/)
17. Reut Tsarfaty, Amit Seker, Shoval Sadde, and Stav Klein. 2019. What’s wrong with Hebrew NLP? and how to make it right. CoRR, abs/1908.05453. [↗](https://arxiv.org/abs/1908.05453) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2019-tsarfaty-what-s-wrong-hebrew-nlp/)
18. Shuly Wintner. 1998. Towards a linguistically motivated computational grammar for Hebrew. In Computational Approaches to Semitic Languages. [find](https://scholar.google.com/scholar?q=Shuly+Wintner%2E+1998%2E+Towards+a+linguistically+motivated+computational+grammar+for+Hebrew%2E+In+Computational+Approaches+to+Semitic+Languages%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-1998-wintner-linguistically-motivated-computational-grammar-hebrew/)
19. Elazar Gershuni and Yuval Pinter. 2021. Restoring hebrew diacritics without a dictionary. [find](https://scholar.google.com/scholar?q=Elazar+Gershuni+and+Yuval+Pinter%2E+2021%2E+Restoring+hebrew+diacritics+without+a+dictionary%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/findings-2022-gershuni-restoring-hebrew-diacritics-without-dictionary/)
20. Viktor Golinets. 2013. Masora, Tiberian. In Geoffrey Khan, editor, Encyclopedia of Hebrew Language and Linguistics. Brill, Leiden. [find](https://scholar.google.com/scholar?q=Viktor+Golinets%2E+2013%2E+Masora%2C+Tiberian%2E+In+Geoffrey+Khan%2C+editor%2C+Encyclopedia+of+Hebrew+Language+and+Linguistics%2E+Brill%2C+Leiden%2E)
21. Jirka Hana, Anna Feldman, and Katsiaryna Aharodnik. 2011. A low-budget tagger for old Czech. In Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 10–18, Portland, OR, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jirka+Hana%2C+Anna+Feldman%2C+and+Katsiaryna+Aharodnik%2E+2011%2E+A+low%2Dbudget+tagger+for+old+Czech%2E+In+Proceedings+of+the+5th+ACL%2DHLT+Workshop+on+Language+Technology+for+Cultural+Heritage%2C+Social+Sciences%2C+a)