[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2016-bollmann-improving-historical-spelling-normalization-bi/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Improving historical spelling normalization with bi-directional LSTMs and multi-task learning

# Improving historical spelling normalization with bi-directional LSTMs and multi-task learning

**Authors:** [Marcel Bollmann](https://md.mova.elamur.ai/free/en/docs/papers/authors/marcel-bollmann/) and [Anders Søgaard](https://md.mova.elamur.ai/free/en/docs/papers/authors/anders-sogaard/) · **Year:** 2016 · **Venue:** Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers **Link:** [aclanthology.org/C16-1013](https://aclanthology.org/C16-1013/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**4** citations](https://openalex.org/works?filter=cites:W2539746012) openalex · 2026-10-18[**15** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2016-bollmann-improving-historical-spelling-normalization-bi/) of the 140673 papers whose bibliographies we read[Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers](https://aclanthology.org/volumes/C16-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/C16-1013/)
- [PDF](https://aclanthology.org/C16-1013.pdf)
- [OpenAlex](https://openalex.org/W2539746012)

**Authors (2)**

1. [Marcel Bollmann](https://openalex.org/A5005438563) [ORCID](https://orcid.org/0000-0003-2598-8150) · [Semantic Scholar](https://www.semanticscholar.org/author/34887843) · h 11 · 337 citations · 42 works
2. [Anders Søgaard](https://openalex.org/A5018138946) [ORCID](https://orcid.org/0000-0001-5250-4276) · [Semantic Scholar](https://www.semanticscholar.org/author/1700187) · h 43 · 8552 citations · 440 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The paper frames historical spelling normalisation as character-level sequence labelling and uses a bi-directional LSTM for it. The model needs no lexical resource and is trained on small sets of about 2,000 to 11,000 tokens of Early New High German. It beats the Norma tool and a CRF tagger on almost all texts. Adding auxiliary normalisation data through multi-task learning raises the average accuracy by 0.7 points, while the same extra data lowers the accuracy of Norma by 0.4 points and of the CRF by 2.0 points. The authors conclude that neural models work with small data when the architecture is right.

## Ideas

- I1. `en/src/morph.rs`: the transparent analogue of a character tagger is a scored set of edit operations per character position, learned from pairs of archaic and modern forms. Measure word accuracy on tools/gutenberg lessons against the `archaic` rules, twice.
- I2. Auxiliary data as a test: add the modern lemma pairs from `data/lemmas.tsv` as extra training pairs for the rule learner. The paper's CRF result suggests this can lower accuracy, so measure it with `en folds` before any use.
- I3. Per-text report: give word accuracy for each lesson and not only the average, and a gate that shows red when any lesson falls below the identity baseline. The gate must be seen to fail on a deliberately broken rule set.

## For Mova

The chapter is [09. Early Modern and archaic forms](https://md.mova.elamur.ai/free/en/docs/languages/en/09-early-modern-and-archaic/). The code is `en/src/morph.rs` and the `en morph-eval` command. The paper's data is German and the English figures must come from our own lessons. The neural model itself is not for Mova; only the evaluation and the count-based analogue are.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{bollmann2016improving,
  title = {{Improving historical spelling normalization with bi-directional LSTMs and multi-task learning}},
  author = {Bollmann, Marcel and Søgaard, Anders},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  url = {https://aclanthology.org/C16-1013/}
}
```

- [← previous Evaluating Inter-Annotator Agreement on Historical Spelling Normalization](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2016-bollmann-evaluating-inter-annotator-agreement-historical/)
- [next → Verb Phrase Ellipsis Resolution Using Discriminative and Margin-Infused Algorithms](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2016-dean-verb-phrase-ellipsis-resolution-discriminative/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

23 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marilisa Amoia and Jose Manuel Martinez. 2013. Using comparable collections of historical texts for building a diachronic dictionary for spelling normalization. In Proceedings of the 7th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 84–89, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=Marilisa+Amoia+and+Jose+Manuel+Martinez%2E+2013%2E+Using+comparable+collections+of+historical+texts+for+building+a+diachronic+dictionary+for+spelling+normalization%2E+In+Proceedings+of+the+7th+Workshop+on+L)
2. Mayce Al Azawi, Muhammad Zeshan Afzal, and Thomas M. Breuel. 2013. Normalizing historical orthography for OCR historical documents using LSTM. In Proceedings of the 2nd International Workshop on Historical Document Imaging and Processing, pages 80–85. ACM. [find](https://scholar.google.com/scholar?q=Mayce+Al+Azawi%2C+Muhammad+Zeshan+Afzal%2C+and+Thomas+M%2E+Breuel%2E+2013%2E+Normalizing+historical+orthography+for+OCR+historical+documents+using+LSTM%2E+In+Proceedings+of+the+2nd+International+Workshop+on+Histo)
3. R. Harald Baayen, Richard Piepenbrock, and Léon Gulikers. 1995. The CELEX lexical database (Release 2) (CD-ROM). Linguistic Data Consortium, University of Pennsylvania, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=R%2E+Harald+Baayen%2C+Richard+Piepenbrock%2C+and+Le%CC%81on+Gulikers%2E+1995%2E+The+CELEX+lexical+database+%28Release+2%29+%28CD%2DROM%29%2E+Linguistic+Data+Consortium%2C+University+of+Pennsylvania%2C+Philadelphia%2C+PA%2E)
4. Alistair Baron and Paul Rayson. 2008. VARD 2: A tool for dealing with spelling variation in historical corpora. In Proceedings of the Postgraduate Conference in Corpus Linguistics. [find](https://scholar.google.com/scholar?q=Alistair+Baron+and+Paul+Rayson%2E+2008%2E+VARD+2%3A+A+tool+for+dealing+with+spelling+variation+in+historical+corpora%2E+In+Proceedings+of+the+Postgraduate+Conference+in+Corpus+Linguistics%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/enprag-2008-baron-vard2-tool-dealing-spelling-variation/)
5. Marcel Bollmann. 2012. (Semi-)automatic normalization of historical texts using distance measures and the Norma tool. In Proceedings of the Second Workshop on Annotation of Corpora for Research in the Humanities (ACRH-2), Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Marcel+Bollmann%2E+2012%2E+%28Semi%2D%29automatic+normalization+of+historical+texts+using+distance+measures+and+the+Norma+tool%2E+In+Proceedings+of+the+Second+Workshop+on+Annotation+of+Corpora+for+Research+in+the)
6. Rich Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the 10th International Conference on Machine Learning (ICML), pages 41–48. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+1993%2E+Multitask+learning%3A+A+knowledge%2Dbased+source+of+inductive+bias%2E+In+Proceedings+of+the+10th+International+Conference+on+Machine+Learning+%28ICML%29%2C+pages+41%E2%80%9348%2E)
7. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. In Proceedings of the Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8), pages 103–111, Doha, Qatar. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Dzmitry+Bahdanau%2C+and+Yoshua+Bengio%2E+2014%2E+On+the+properties+of+neural+machine+translation%3A+Encoder%E2%80%93decoder+approaches%2E+In+Proceedings+of+the+Eighth+Workshop+on+Sy)
8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+The+Journal+of+Machine+Learning+Research%2C+12%3A2)
9. Stefanie Dipper and Simone Schultz-Balluff. 2013. The Anselm corpus: Methods and perspectives of a parallel aligned corpus. In Proceedings of the NODALIDA Workshop on Computational Historical Linguistics. [find](https://scholar.google.com/scholar?q=Stefanie+Dipper+and+Simone+Schultz%2DBalluff%2E+2013%2E+The+Anselm+corpus%3A+Methods+and+perspectives+of+a+parallel+aligned+corpus%2E+In+Proceedings+of+the+NODALIDA+Workshop+on+Computational+Historical+Linguist)
10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%93+1780%2E)
11. Bryan Jurish. 2010. More than words: using token context to improve canonicalization of historical German. Journal for Language Technology and Computational Linguistics, 25(1):23–39. [find](https://scholar.google.com/scholar?q=Bryan+Jurish%2E+2010%2E+More+than+words%3A+using+token+context+to+improve+canonicalization+of+historical+German%2E+Journal+for+Language+Technology+and+Computational+Linguistics%2C+25%281%29%3A23%E2%80%9339%2E)
12. Sigrid Klerke, Yoav Goldberg, and Anders Søgaard. 2016. Improving sentence compression by learning to predict gaze. In Proceedings of NAACL-HLT 2016, pages 1528–1533, San Diego, CA. [find](https://scholar.google.com/scholar?q=Sigrid+Klerke%2C+Yoav+Goldberg%2C+and+Anders+S%C3%B8gaard%2E+2016%2E+Improving+sentence+compression+by+learning+to+predict+gaze%2E+In+Proceedings+of+NAACL%2DHLT+2016%2C+pages+1528%E2%80%931533%2C+San+Diego%2C+CA%2E)
13. Julia Krasselt, Marcel Bollmann, Stefanie Dipper, and Florian Petran. 2015. Guidelines for normalizing historical German texts. Bochumer Linguistische Arbeitsberichte, 15. [find](https://scholar.google.com/scholar?q=Julia+Krasselt%2C+Marcel+Bollmann%2C+Stefanie+Dipper%2C+and+Florian+Petran%2E+2015%2E+Guidelines+for+normalizing+historical+German+texts%2E+Bochumer+Linguistische+Arbeitsberichte%2C+15%2E)
14. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707–710. [find](https://scholar.google.com/scholar?q=Vladimir+I%2E+Levenshtein%2E+1966%2E+Binary+codes+capable+of+correcting+deletions%2C+insertions%2C+and+reversals%2E+Soviet+Physics+Doklady%2C+10%288%29%3A707%E2%80%93710%2E)
15. Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. arXiv:1511.06114v4. [↗](https://arxiv.org/abs/1511.06114)
16. Naoaki Okazaki. 2007. CRFsuite: a fast implementation of conditional random fields (CRFs). [http://www.chokkan.org/software/crfsuite/](http://www.chokkan.org/software/crfsuite/). [↗](http://www.chokkan.org/software/crfsuite/)
17. Eva Pettersson, Beáta Megyesi, and Jörg Tiedemann. 2013. An SMT approach to automatic annotation of historical text. In Proceedings of the NODALIDA Workshop on Computational Historical Linguistics, Oslo, Norway. [find](https://scholar.google.com/scholar?q=Eva+Pettersson%2C+Bea%CC%81ta+Megyesi%2C+and+Jo%CC%88rg+Tiedemann%2E+2013%2E+An+SMT+approach+to+automatic+annotation+of+historical+text%2E+In+Proceedings+of+the+NODALIDA+Workshop+on+Computational+Historical+Linguistics%2C+)
18. Michael Piotrowski. 2012. Natural Language Processing for Historical Texts. Number 17 in Synthesis Lectures on Human Language Technologies. Morgan & Claypool, San Rafael, CA. [find](https://scholar.google.com/scholar?q=Michael+Piotrowski%2E+2012%2E+Natural+Language+Processing+for+Historical+Texts%2E+Number+17+in+Synthesis+Lectures+on+Human+Language+Technologies%2E+Morgan+%26+Claypool%2C+San+Rafael%2C+CA%2E)
19. Jordi Porta, José-Luis Sancho, and Javier Gómez. 2013. Edit transducers for spelling variation in Old Spanish. In Proceedings of the NODALIDA Workshop on Computational Historical Linguistics, Oslo, Norway. [find](https://scholar.google.com/scholar?q=Jordi+Porta%2C+Jose%CC%81%2DLuis+Sancho%2C+and+Javier+Go%CC%81mez%2E+2013%2E+Edit+transducers+for+spelling+variation+in+Old+Spanish%2E+In+Proceedings+of+the+NODALIDA+Workshop+on+Computational+Historical+Linguistics%2C+Oslo%2C+)
20. Yves Scherrer and Tomaž Erjavec. 2013. Modernizing historical Slovene words with character-based SMT. In Proceedings of the 4th Biennial Workshop on Balto-Slavic Natural Language Processing, Sofia, Bulgaria. [find](https://scholar.google.com/scholar?q=Yves+Scherrer+and+Tomaz%CC%8C+Erjavec%2E+2013%2E+Modernizing+historical+Slovene+words+with+character%2Dbased+SMT%2E+In+Proceedings+of+the+4th+Biennial+Workshop+on+Balto%2DSlavic+Natural+Language+Processing%2C+Sofia%2C+B)
21. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS 2014), number 27, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS+2014%29%2C+number+27%2C+pages+3104%E2%80%933112%2E)
22. Felipe Sánchez-Martı́nez, Isabel Martı́nez-Sempere, Xavier Ivars-Ribes, and Rafael C. Carrasco. 2013. An open diachronic corpus of historical Spanish: annotation criteria and automatic modernisation of spelling. arXiv:1306.3692v1, 06. [↗](https://arxiv.org/abs/1306.3692)
23. Martijn Wieling, Jelena Prokić, and John Nerbonne. 2009. Evaluating the pairwise string alignment of pronunciations. In Proceedings of the EACL 2009 Workshop on Language Technology and Resources for Cultural Heritage, Social Sciences, Humanities, and Education (LaTeCH – SHELT&R 2009), pages 26–34, Athens, Greece. [find](https://scholar.google.com/scholar?q=Martijn+Wieling%2C+Jelena+Prokic%CC%81%2C+and+John+Nerbonne%2E+2009%2E+Evaluating+the+pairwise+string+alignment+of+pronunciations%2E+In+Proceedings+of+the+EACL+2009+Workshop+on+Language+Technology+and+Resources+for+)