[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/lingbulk-2012-felt-first-results-study-evaluating-pre/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Aramaic](https://md.mova.elamur.ai/free/en/docs/papers/arc/) / First Results in a Study Evaluating Pre-annotation and Correction Propagation for Machine-Assisted Syriac Morphological Analysis

# First Results in a Study Evaluating Pre-annotation and Correction Propagation for Machine-Assisted Syriac Morphological Analysis

**Authors:** [Paul Felt](https://md.mova.elamur.ai/free/en/docs/papers/authors/paul-felt/); [Eric Ringger](https://md.mova.elamur.ai/free/en/docs/papers/authors/eric-ringger/); [Kevin Seppi](https://md.mova.elamur.ai/free/en/docs/papers/authors/kevin-seppi/); [Kristian Heal](https://md.mova.elamur.ai/free/en/docs/papers/authors/kristian-heal/); [Robbie Haertel](https://md.mova.elamur.ai/free/en/docs/papers/authors/robbie-haertel/); [Deryle Lonsdale](https://md.mova.elamur.ai/free/en/docs/papers/authors/deryle-lonsdale/) · **Year:** 2012 · **Venue:** Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12) **Link:** [aclanthology.org/L12-1281](https://aclanthology.org/L12-1281/) · library `lingbulk-2012-felt-first-results-study-evaluating-pre` · **Licence:** [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (ELRA, LREC proceedings) — summary only

[**4** citations](https://openalex.org/works?filter=cites:W2251975282) openalex-snapshot · 2026-10-14[**1** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/lingbulk-2012-felt-first-results-study-evaluating-pre/) of the 140673 papers whose bibliographies we read[Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)](https://aclanthology.org/volumes/L12-1/)[licence](https://creativecommons.org/licenses/by-nc/4.0/)

- [Publisher's page](https://aclanthology.org/L12-1281/)
- [PDF](https://aclanthology.org/L12-1281.pdf)
- [DOI 10.63317/2gr6g4m4soxd](https://doi.org/10.63317/2gr6g4m4soxd)
- [OpenAlex](https://openalex.org/W2251975282)

**Authors (6)**

1. [Paul Felt](https://openalex.org/A5060115458) [Semantic Scholar](https://www.semanticscholar.org/author/1873296) · h 7 · 106 citations · 20 works
2. [Eric Ringger](https://openalex.org/A5090791372) [Semantic Scholar](https://www.semanticscholar.org/author/1772625) · h 21 · 1800 citations · 103 works
3. [Kevin Seppi](https://openalex.org/A5029074659) [ORCID](https://orcid.org/0000-0002-4932-2700) · [Semantic Scholar](https://www.semanticscholar.org/author/31819161) · h 23 · 1886 citations · 137 works
4. [Kristian Heal](https://openalex.org/A5061488216) [ORCID](https://orcid.org/0000-0002-7710-5252) · [Semantic Scholar](https://www.semanticscholar.org/author/39770620) · h 6 · 104 citations · 62 works
5. [Robbie Haertel](https://openalex.org/A5017536910) [Semantic Scholar](https://www.semanticscholar.org/author/1963625) · h 9 · 275 citations · 31 works
6. [Deryle Lonsdale](https://openalex.org/A5017104888) [ORCID](https://orcid.org/0000-0001-7784-6061) · [Semantic Scholar](https://www.semanticscholar.org/author/1730612) · h 19 · 1512 citations · 121 works

Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart.

## What it is about

The authors ask when machine pre-annotation (the annotator corrects a machine guess instead of starting from nothing) and correction propagation (a correction re-runs the model on the rest of the same sentence) make Syriac morphological annotation faster and more accurate. They use the Syromorph analyzer, trained on subsets of the Peshitta New Testament (about 100,000 tokens), to produce pre-annotations at fixed accuracy levels from 25% to 100%. Nine Syriac experts annotated 30 sentences from the Acts of Judas Thomas in the CCASH web tool, with the conditions assigned by a rotating matrix. The preliminary result: pre-annotations improve accuracy at 60% model accuracy and up, and speed at 80% and up (by about 10 to 20 seconds per word); the extra effect of correction propagation is unclear. The analysis rests on 1,368 word-level data points.

## Ideas

- I1. A threshold rule for the assist layer: offer pre-annotations only when the analyzer's measured accuracy on held-out data is above a set level (about 60% for accuracy and 80% for speed, as a starting point). For `tools/hbo`, this means measuring the segmenter and parser on a held-out set before the output is shown as a pre-fill.
- I2. Correction propagation as a constrained re-parse: after a user fixes one decision (a segment, a POS, a head), re-run the deterministic chain with that decision pinned and show only the decisions that changed. In `gparse` terms: a re-parse with fixed arcs, not a retrained model.
- I3. The assignment design (each annotator meets each condition about equally often, and the sentence order is permuted so that model quality does not look like it rises over time) is a template for any future Mova study with human annotators.

## For Mova

I1 belongs to `tools/hbo` (a held-out check of the segmenter before any pre-fill). I2 is a feature of `gparse` on CoNLL-U output. I3 is a method note for a study document, not a rule. Only the threshold figures, not the paper's data, go into a test.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.63317/2gr6g4m4soxd](https://doi.org/10.63317/2gr6g4m4soxd)

```
@inproceedings{felt2012first,
  title = {{First Results in a Study Evaluating Pre-annotation and Correction Propagation for Machine-Assisted Syriac Morphological Analysis}},
  author = {Felt, Paul and Ringger, Eric and Seppi, Kevin and Heal, Kristian and Haertel, Robbie and Lonsdale, Deryle},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  doi = {10.63317/2gr6g4m4soxd},
  url = {https://doi.org/10.63317/2gr6g4m4soxd}
}
```

- [← previous CCASH: A Web Application Framework for Efficient, Distributed Language Resource Development](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2010-felt-ccash-web-application-framework-efficient/)
- [next → Evaluating Lemmatization Models for Machine-Assisted Corpus-Dictionary Linkage](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2014-black-evaluating-lemmatization-models-machine-assisted/)
- [↑ back to Aramaic](https://md.mova.elamur.ai/free/en/docs/papers/arc/)

## References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Baldridge, J., & Osborne, M. (2004). Active Learning and the Total Cost of Annotation. In Proceedings of EMNLP, 2004, pp. 9-16. [find](https://scholar.google.com/scholar?q=Baldridge%2C+J%2E%2C+%26+Osborne%2C+M%2E+%282004%29%2E+Active+Learning+and+the+Total+Cost+of+Annotation%2E+In+Proceedings+of+EMNLP%2C+2004%2C+pp%2E+9%2D16%2E)
2. Brants, T. & Plaehn, O. (2000). Interactive Corpus Annotation. In Proceedings of LREC 2000. Athens, Greece. [↗](https://doi.org/10.63317/4k3a26gvvdqa)
3. Chiou, F., Chiang, D., & Palmer, M. (2001). Facilitating Treebank Annotation with a Statistical Parser. In Proceedings of the Human Language Technology (HLT) Conference. San Diego: ACL. [find](https://scholar.google.com/scholar?q=Chiou%2C+F%2E%2C+Chiang%2C+D%2E%2C+%26+Palmer%2C+M%2E+%282001%29%2E+Facilitating+Treebank+Annotation+with+a+Statistical+Parser%2E+In+Proceedings+of+the+Human+Language+Technology+%28HLT%29+Conference%2E+San+Diego%3A+ACL%2E)
4. Felt, P., Merkling, O., Carmen, M., Ringger, E., Lemmon, W., Seppi, K. and Haertel, R. (2010). CCASH: A Web Application Framework for Efficient, Distributed Language Resource Development. In Proceedings of LREC 2010. Malta. [↗](https://doi.org/10.63317/36n37t2792qj) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2010-felt-ccash-web-application-framework-efficient/)
5. Ganchev, K., Pereira, F., & Mandel, M. (2007). Semi-automated Named Entity Annotation. In Proceedings of the Linguistic Annotation Workshop. Prague: ACL, pp. 53-56. [↗](https://doi.org/10.3115/1642059.1642068)
6. Haertel, R., Ringger, E., Seppi, K., Carroll, J., and McClanahan, P. (2008). Assessing the Costs of Sampling Methods in Active Learning for Annotation. In Proceedings of the 46 th Annual Meeting of the ACL-HLT. Columbus, OH: ACL, pp. 65-68. [↗](https://doi.org/10.3115/1557690.1557708)
7. Kiraz, G. (1994). Automatic concordance generation of Syriac texts. R. Lavenant, editor. Rome, Italy: VI Symposium Syriacum 1992, Orientalia Christiana Analecta, pp. 461-475. [find](https://scholar.google.com/scholar?q=Kiraz%2C+G%2E+%281994%29%2E+Automatic+concordance+generation+of+Syriac+texts%2E+R%2E+Lavenant%2C+editor%2E+Rome%2C+Italy%3A+VI+Symposium+Syriacum+1992%2C+Orientalia+Christiana+Analecta%2C+pp%2E+461%2D475%2E)
8. Klebanov, B., and Beigman, E. (2009). From Annotator Agreement to Noise Models. Computational Linguistics 35(4), pp. 495-503. [↗](https://doi.org/10.1162/coli.2009.35.4.35402)
9. Kristjannson, T., Culotta, A., Viola, P., and McCallum, A. M. (2004). Interactive Information Extraction with Constrained Conditional Random Fields. In Nineteenth Conference of the AAAI. San Jose, CA. [find](https://scholar.google.com/scholar?q=Kristjannson%2C+T%2E%2C+Culotta%2C+A%2E%2C+Viola%2C+P%2E%2C+and+McCallum%2C+A%2E+M%2E+%282004%29%2E+Interactive+Information+Extraction+with+Constrained+Conditional+Random+Fields%2E+In+Nineteenth+Conference+of+the+AAAI%2E+San+Jose%2C+CA%2E)
10. Marcus, M., Santorini, B., & Marcinkiewicz, M. (1993). Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics 19(2), pp. 313-330. [↗](https://doi.org/10.21236/ada273556) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
11. McClanahan, P., Busby, G., Haertel, R., Heal, K., Lonsdale, D. Seppi, K. and Ringger, E. (2010). A Probabilistic Morphological Analyzer for Syriac. In Proceedings of EMNLP 2010, pp. 810-820. [find](https://scholar.google.com/scholar?q=McClanahan%2C+P%2E%2C+Busby%2C+G%2E%2C+Haertel%2C+R%2E%2C+Heal%2C+K%2E%2C+Lonsdale%2C+D%2E+Seppi%2C+K%2E+and+Ringger%2C+E%2E+%282010%29%2E+A+Probabilistic+Morphological+Analyzer+for+Syriac%2E+In+Proceedings+of+EMNLP+2010%2C+pp%2E+810%2D820%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2010-mcclanahan-probabilistic-morphological-analyzer-syriac/)
12. Menke, J. and Martinez, T. R. (2004). Using Permutations Instead of Student's t Distribution for P-values in Paired-difference Algorithm Comparisons. In Proceedings of 2004 IEEE International Joint Conference on Neural Networks, pp. 1331-1335. Tov, E. editor. The Dead Sea Scrolls Electronic Library. [find](https://scholar.google.com/scholar?q=Menke%2C+J%2E+and+Martinez%2C+T%2E+R%2E+%282004%29%2E+Using+Permutations+Instead+of+Student%27s+t+Distribution+for+P%2Dvalues+in+Paired%2Ddifference+Algorithm+Comparisons%2E+In+Proceedings+of+2004+IEEE+International+Joint+Co)
13. Wright, William. Apocryphal Acts of the Apostles. 2 vols. London: Williams and Norgate, 1871, vol. 1, pp. 171-333; [find](https://scholar.google.com/scholar?q=Wright%2C+William%2E+Apocryphal+Acts+of+the+Apostles%2E+2+vols%2E+London%3A+Williams+and+Norgate%2C+1871%2C+vol%2E+1%2C+pp%2E+171%2D333%3B)