[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/lingbulk-2014-black-evaluating-lemmatization-models-machine-assisted/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Aramaic](https://md.mova.elamur.ai/free/en/docs/papers/arc/) / Evaluating Lemmatization Models for Machine-Assisted Corpus-Dictionary Linkage

# Evaluating Lemmatization Models for Machine-Assisted Corpus-Dictionary Linkage

**Authors:** Kevin Black; [Eric Ringger](https://md.mova.elamur.ai/free/en/docs/papers/authors/eric-ringger/); [Paul Felt](https://md.mova.elamur.ai/free/en/docs/papers/authors/paul-felt/); [Kevin Seppi](https://md.mova.elamur.ai/free/en/docs/papers/authors/kevin-seppi/); [Kristian Heal](https://md.mova.elamur.ai/free/en/docs/papers/authors/kristian-heal/); [Deryle Lonsdale](https://md.mova.elamur.ai/free/en/docs/papers/authors/deryle-lonsdale/) · **Year:** 2014 · **Venue:** Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14) **Link:** [aclanthology.org/L14-1142](https://aclanthology.org/L14-1142/) · library `lingbulk-2014-black-evaluating-lemmatization-models-machine-assisted` · **Licence:** [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (ELRA, LREC proceedings) — summary only

[**0** citations](https://openalex.org/works?filter=cites:W2251440390) openalex · 2026-10-19[Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)](https://aclanthology.org/volumes/L14-1/)[licence](https://creativecommons.org/licenses/by-nc/4.0/)

- [Publisher's page](https://aclanthology.org/L14-1142/)
- [PDF](https://aclanthology.org/L14-1142.pdf)
- [DOI 10.63317/3xt99g4swge5](https://doi.org/10.63317/3xt99g4swge5)
- [OpenAlex](https://openalex.org/W2251440390)

**Authors (6)**

1. Kevin Black [Semantic Scholar](https://www.semanticscholar.org/author/2069483822)
2. [Eric Ringger](https://openalex.org/A5090791372) [Semantic Scholar](https://www.semanticscholar.org/author/1772625) · h 21 · 1822 citations · 103 works
3. [Paul Felt](https://openalex.org/A5060115458) [Semantic Scholar](https://www.semanticscholar.org/author/1873296) · h 7 · 106 citations · 20 works
4. [Kevin Seppi](https://openalex.org/A5029074659) [ORCID](https://orcid.org/0000-0002-4932-2700) · [Semantic Scholar](https://www.semanticscholar.org/author/31819161) · h 23 · 1923 citations · 137 works
5. [Kristian Heal](https://openalex.org/A5061488216) [ORCID](https://orcid.org/0000-0002-7710-5252) · [Semantic Scholar](https://www.semanticscholar.org/author/39770620) · h 6 · 104 citations · 62 works
6. [Deryle Lonsdale](https://openalex.org/A5017104888) [ORCID](https://orcid.org/0000-0001-7784-6061) · [Semantic Scholar](https://www.semanticscholar.org/author/1730612) · h 20 · 1542 citations · 139 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

The task is to link each word token of a Syriac corpus to the right dictionary headword (corpus-dictionary linkage), here reduced to stem-to-baseform linkage, since segmentation already reaches about 98.9% accuracy. The authors adapt the DirecTL+ discriminative string transducer to this task and compare it with Morfette (a maximum-entropy edit-script classifier) and with a baseline that memorises the most frequent baseform per stem. The data is the Peshitta New Testament (109,640 tokens, 3,038 baseform types). With local sentence context, DirecTL+ reaches 96.92% overall accuracy on the blind test, against 96.06% for Morfette with local context, but it trains about 25 times slower (100 minutes against 3.7). The baseline is at 98.2% on known stems and 26% on unknown ones; the discriminative models are at about 60% on unknown stems. The error analysis splits stems into known or unknown and stable or ambiguous cases.

## Ideas

- I1. Report lemmatizer accuracy in the same six error classes (known stable unambiguous, known latent ambiguous, known stable ambiguous, known dynamic ambiguous, unknown unambiguous, unknown ambiguous). For `tools/hbo` and `macula dict`, this shows whether a gain comes from known stems or from unknown ones, which a single overall figure hides.
- I2. A hybrid lemmatizer: a lookup table of stem-to-baseform pairs seen in the dictionary or in `macula dict` first, a rule-based or statistical guesser only for unknown stems. The paper shows the table is near-perfect on seen stems and the guesser is needed for the rest.
- I3. Train time as a selection criterion: a model that is 1% better but 25 times slower to train loses in an iteratively refined assist loop. Mova's choice of lemmatizer should be scored on accuracy and on the cost of each update.

## For Mova

I1 is a test table for `tools/hbo` and for the Hebrew lemma work in [The lexicon](https://md.mova.elamur.ai/free/en/docs/languages/hbo/10-lexicon/). I2 belongs to [The lexicon of Biblical Aramaic](https://md.mova.elamur.ai/free/en/docs/languages/arc/lexicon/) (the Aramaic lemma table) and to `macula dict` for Hebrew. I3 is a decision rule for any future lemmatizer comparison.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.63317/3xt99g4swge5](https://doi.org/10.63317/3xt99g4swge5)

```
@inproceedings{black2014evaluating,
  title = {{Evaluating Lemmatization Models for Machine-Assisted Corpus-Dictionary Linkage}},
  author = {Black, Kevin and Ringger, Eric and Felt, Paul and Seppi, Kevin and Heal, Kristian and Lonsdale, Deryle},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/3xt99g4swge5},
  url = {https://doi.org/10.63317/3xt99g4swge5}
}
```

- [← previous First Results in a Study Evaluating Pre-annotation and Correction Propagation for Machine-Assisted Syriac Morphological Analysis](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2012-felt-first-results-study-evaluating-pre/)
- [next → The morphosyntax of aspect stacking in Northeastern Neo-Aramaic](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2016-kalin-morphosyntax-aspect-stacking-northeastern-neo/)
- [↑ back to Aramaic](https://md.mova.elamur.ai/free/en/docs/papers/arc/)

## References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Carmen, M., Felt, P., Haertel, R., Lonsdale, D., McClanahan, P., Merkling, O., Ringger, E., and Seppi, K. (2010). Tag dictionaries accelerate manual annotation. In Proceedings of LREC-10, Valletta, Malta. ELRA. [↗](https://doi.org/10.63317/2wvufaog66zv)
2. Chrupała, G., Dinu, G., and van Genabith, J. (2008). Learning morphology with Morfette. In Proceedings of LREC-08, Marrakech, Morocco. ELRA. [↗](https://doi.org/10.63317/46dhmje42wcr) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2008-chrupala-learning-morphology-morfette/)
3. Collins, M. (2002). Discriminative training methods for Hidden Markov Models: Theory and experiments with perceptron algorithms. In Proceedings of the EMNLP-02, pages 1–8, Stroudsburg, PA. ACL. [↗](https://doi.org/10.3115/1118693.1118694) [our card](https://md.mova.elamur.ai/free/en/docs/papers/collins-2002-discriminative-training-methods-hidden-markov/)
4. Felt, P., Ringger, E. K., Seppi, K., Heal, K. S., Haertel, R. A., and Lonsdale, D. (2013). Evaluating machine-assisted annotation in under-resourced settings. Language Resources and Evaluation. [↗](https://doi.org/10.1007/s10579-013-9258-8)
5. Habash, N. and Rambow, O. (2005). Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of ACL-05, pages 573–580, Stroudsburg, PA. ACL. [↗](https://doi.org/10.3115/1219840.1219911) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-habash-arabic-tokenization-part-speech-tagging/)
6. Jiampojamarn, S., Kondrak, G., and Sherif, T. (2007). KLA Correct 61 ± 40 39 ± 42 40 ± 42 41 ± 42 UU Correct 344 ± 20 79 ± 14 208 ± 14 205 ± 15 KSA Acc. N/A 90.70 ± 0.98 90.17 ± 0.93 93.48 ± 0.86 UA Acc. N/A 27.77 ± 31.03 36.11 ± 28.70 27.77 ± 31.03 KSA Correct 1297 ± 55 1176 ± 57 1170 ± 57 1212 ± 56 UA Correct 1.3 ± 1.1 0.40 ± 0.70 0.50 ± 0.71 0.40 ± 0.70 Applying many-to-many alignments and hidden Markov models to letter-to-phoneme conversion. In Proceeding of NAACL-07, pages 372–379, Rochester, NY. ACL. [find](https://scholar.google.com/scholar?q=Jiampojamarn%2C+S%2E%2C+Kondrak%2C+G%2E%2C+and+Sherif%2C+T%2E+%282007%29%2E+KLA+Correct+61+%C2%B1+40+39+%C2%B1+42+40+%C2%B1+42+41+%C2%B1+42+UU+Correct+344+%C2%B1+20+79+%C2%B1+14+208+%C2%B1+14+205+%C2%B1+15+KSA+Acc%2E+N%2FA+90%2E70+%C2%B1+0%2E98+90%2E17+%C2%B1+0%2E93+93%2E48+%C2%B1+0%2E86+UA+A)
7. Jiampojamarn, S., Cherry, C., and Kondrak, G. (2008). Joint processing and discriminative training for letter-to-phoneme conversion. In Proceedings of ACL-08: HLT, pages 905–913, Columbus, OH. ACL. [find](https://scholar.google.com/scholar?q=Jiampojamarn%2C+S%2E%2C+Cherry%2C+C%2E%2C+and+Kondrak%2C+G%2E+%282008%29%2E+Joint+processing+and+discriminative+training+for+letter%2Dto%2Dphoneme+conversion%2E+In+Proceedings+of+ACL%2D08%3A+HLT%2C+pages+905%E2%80%93913%2C+Columbus%2C+OH%2E+ACL%2E)
8. Jiampojamarn, S., Bhargava, A., Dou, Q., Dwyer, K., and Kondrak, G. (2009). DirecTL: a language independent approach to transliteration. In Proceedings of NEWS-09, pages 28–31, Suntec, Singapore. ACL. [↗](https://doi.org/10.3115/1699705.1699712)
9. Jiampojamarn, S., Cherry, C., and Kondrak, G. (2010). Integrating joint n-gram features into a discriminative training framework. In Proceedings of NAACL-10, pages 697–700, Los Angeles, CA. ACL. [find](https://scholar.google.com/scholar?q=Jiampojamarn%2C+S%2E%2C+Cherry%2C+C%2E%2C+and+Kondrak%2C+G%2E+%282010%29%2E+Integrating+joint+n%2Dgram+features+into+a+discriminative+training+framework%2E+In+Proceedings+of+NAACL%2D10%2C+pages+697%E2%80%93700%2C+Los+Angeles%2C+CA%2E+ACL%2E)
10. Kilgarriff, A. and Rundell, M. (2002). Lexical profiling software and its lexicographic applications - a case study. In Proceedings of EURALEX-02, pages 807–818, København, Denmark. Center for Sprogteknologi. [find](https://scholar.google.com/scholar?q=Kilgarriff%2C+A%2E+and+Rundell%2C+M%2E+%282002%29%2E+Lexical+profiling+software+and+its+lexicographic+applications+%2D+a+case+study%2E+In+Proceedings+of+EURALEX%2D02%2C+pages+807%E2%80%93818%2C+K%C3%B8benhavn%2C+Denmark%2E+Center+for+Sprogte)
11. Kilgarriff, A. (2005). Linking dictionary and corpus. In Proceedings of ASIALEX-05, Singapore. [find](https://scholar.google.com/scholar?q=Kilgarriff%2C+A%2E+%282005%29%2E+Linking+dictionary+and+corpus%2E+In+Proceedings+of+ASIALEX%2D05%2C+Singapore%2E)
12. Kiraz, G. A. (1994). Automatic concordance generation of Syriac texts. Orientalia Christiana Analecta, 247:461– 475. [find](https://scholar.google.com/scholar?q=Kiraz%2C+G%2E+A%2E+%281994%29%2E+Automatic+concordance+generation+of+Syriac+texts%2E+Orientalia+Christiana+Analecta%2C+247%3A461%E2%80%93+475%2E)
13. Kiraz, G. A. (2000). Multitiered nonlinear morphology using multitape finite automata: A case study on Syriac and Arabic. Computational Linguistics, 26(1):77–105. [↗](https://doi.org/10.1162/089120100561647) [our card](https://md.mova.elamur.ai/free/en/docs/papers/cl-2000-kiraz-multitiered-nonlinear-morphology-multitape-finite/)
14. Lindgren, L. (2011). dkrMorph: A Syriac morphological analyzer. Bachelor’s thesis, Uppsala University, Department of Informatics and Media. [find](https://scholar.google.com/scholar?q=Lindgren%2C+L%2E+%282011%29%2E+dkrMorph%3A+A+Syriac+morphological+analyzer%2E+Bachelor%E2%80%99s+thesis%2C+Uppsala+University%2C+Department+of+Informatics+and+Media%2E)
15. McClanahan, P., Busby, G., Haertel, R., Heal, K., Lonsdale, D., Seppi, K., and Ringger, E. (2010). A probabilistic morphological analyzer for Syriac. In Proceedings of EMNLP-10, pages 810–820, Cambridge, MA. ACL. [find](https://scholar.google.com/scholar?q=McClanahan%2C+P%2E%2C+Busby%2C+G%2E%2C+Haertel%2C+R%2E%2C+Heal%2C+K%2E%2C+Lonsdale%2C+D%2E%2C+Seppi%2C+K%2E%2C+and+Ringger%2C+E%2E+%282010%29%2E+A+probabilistic+morphological+analyzer+for+Syriac%2E+In+Proceedings+of+EMNLP%2D10%2C+pages+810%E2%80%93820%2C+Cambrid) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2010-mcclanahan-probabilistic-morphological-analyzer-syriac/)
16. Payne Smith, J. (1903). A compendious Syriac dictionary founded upon the Thesaurus Syriacus of R. Payne Smith. [find](https://scholar.google.com/scholar?q=Payne+Smith%2C+J%2E+%281903%29%2E+A+compendious+Syriac+dictionary+founded+upon+the+Thesaurus+Syriacus+of+R%2E+Payne+Smith%2E)
17. Ringger, E., Carmen, M., Haertel, R., Seppi, K., Lonsdale, D., McClanahan, P., Carroll, J., and Ellison, N. (2008). Assessing the costs of machine-assisted corpus annotation through a user study. In Proceedings of LREC-08, pages 3318–3324, Marrakech, Morocco. ELRA. [↗](https://doi.org/10.63317/3afovxbya4dv)