[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2012-dickinson-predicting-learner-levels-online-exercises/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/) / Predicting Learner Levels for Online Exercises of Hebrew

# Predicting Learner Levels for Online Exercises of Hebrew

**Authors:** [Markus Dickinson](https://md.mova.elamur.ai/free/en/docs/papers/authors/markus-dickinson/); [Sandra Kübler](https://md.mova.elamur.ai/free/en/docs/papers/authors/sandra-kubler/); Anthony Meyer · **Year:** 2012 · **Venue:** Proceedings of the Seventh Workshop on Building Educational Applications Using NLP **Link:** [aclanthology.org/W12-2011](https://aclanthology.org/W12-2011/) · library `aclbulk-2012-dickinson-predicting-learner-levels-online-exercises` · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, before 2016) — summary only

[**6** citations](https://openalex.org/works?filter=cites:W46330860) openalex-snapshot · 2026-10-14[**3** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2012-dickinson-predicting-learner-levels-online-exercises/) of the 140673 papers whose bibliographies we read[Proceedings of the Seventh Workshop on Building Educational Applications Using NLP](https://aclanthology.org/volumes/W12-20/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/W12-2011/)
- [PDF](https://aclanthology.org/W12-2011.pdf)
- [OpenAlex](https://openalex.org/W46330860)

**Authors (3)**

1. [Markus Dickinson](https://openalex.org/A5063365523) [Semantic Scholar](https://www.semanticscholar.org/author/144503727) · h 14 · 798 citations · 100 works
2. [Sandra Kübler](https://openalex.org/A5009027177) [ORCID](https://orcid.org/0000-0003-0885-5436) · [Semantic Scholar](https://www.semanticscholar.org/author/1804668) · h 25 · 4274 citations · 209 works
3. Anthony Meyer [Semantic Scholar](https://www.semanticscholar.org/author/143852922)

Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart.

## What it is about

The paper predicts the course level (100 to 300+) of university students of Hebrew from one placement-exam exercise: a scrambled sentence whose words must be put in order. The data are 128 sentences from 38 learners, so data is very sparse. Each learner answer is aligned to the closest of the gold orders, with a cost that mixes Levenshtein distance, longest common subsequence and displacement. Features are then extracted in three groups: intra-token errors (inside a word, with morphological slots), inter-token errors (shifts, transpositions and missing tokens, found by running the edit distance with substitutions turned off) and global sentence measures. A first memory-based classifier (TiMBL) maps each phenomenon to a level, and a second one maps the counts of phenomena to the learner's level. Accuracy is 60.5 percent against a majority baseline of 34.2 percent. Inter-token features matter most (dropping them gives 36.8 percent), and weighted kappa is 0.647.

## Ideas

- I1. A deterministic shift detector for word-order exercises: align the learner's words to a gold order with substitutions off, pair the deletions with the insertions, and report transpositions and shifts with their lengths. This is a rule that can be written in Rust in `prag` for English scrambled-sentence checks, and it is transparent: each flagged shift can be shown to the learner.
- I2. Store the acceptable word orders of an exercise once, with the morphology kept apart from the order, as the paper does (morphology is annotated once for all orders). In `world` this is a set of orders over one set of frames, held in `BTreeSet` so that the result is deterministic.
- I3. Report kappa next to accuracy, and use leave-one-out on small sets, because a 60 percent accuracy on 38 learners is not separated from a 34 percent baseline without a second number. This is the measurement rule for any Mova placement or level test.

## For Mova

I1 is a rule for `prag` (word-order checks). I2 is a data design for `world`. I3 is a measurement rule for any level test, to be stated in [12. Mova and Modern Hebrew: where it stands, what it would need](https://md.mova.elamur.ai/free/en/docs/languages/he/12-mova/). The paper is about Hebrew courses; Mova would use the same method only for its own test sets, not for any published exam.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{dickinson2012predicting,
  title = {{Predicting Learner Levels for Online Exercises of Hebrew}},
  author = {Dickinson, Markus and Kübler, Sandra and Meyer, Anthony},
  booktitle = {Proceedings of the Seventh Workshop on Building Educational Applications Using NLP},
  year = {2012},
  url = {https://aclanthology.org/W12-2011/}
}
```

- [← previous A Morphologically Annotated Hebrew CHILDES Corpus](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2012-albert-morphologically-annotated-hebrew-childes-corpus/)
- [next → Corpus-based Explorations of Affective Load Differences in Arabic-Hebrew-English](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2012-strapparava-corpus-based-explorations-affective-load/)
- [↑ back to Modern Hebrew](https://md.mova.elamur.ai/free/en/docs/papers/he/)

## References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dora Alexopoulou, Helen Yannakoudakis, and Ted Briscoe. 2010. From discriminative features to learner grammars: a data driven approach to learner corpora. Talk given at Second Language Research Forum, University of Maryland, October 2010. [find](https://scholar.google.com/scholar?q=Dora+Alexopoulou%2C+Helen+Yannakoudakis%2C+and+Ted+Briscoe%2E+2010%2E+From+discriminative+features+to+learner+grammars%3A+a+data+driven+approach+to+learner+corpora%2E+Talk+given+at+Second+Language+Research+Forum%2C)
2. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater v.2. Journal of Technology, Learning, and Assessment, 4(3), February. [find](https://scholar.google.com/scholar?q=Yigal+Attali+and+Jill+Burstein%2E+2006%2E+Automated+essay+scoring+with+e%2Drater+v%2E2%2E+Journal+of+Technology%2C+Learning%2C+and+Assessment%2C+4%283%29%2C+February%2E)
3. Michele Banko and Eric Brill. 2001. Mitigating the paucity-of-data problem: Exploring the effect of training corpus size on classifier performance for natural language processing. In Proceedings of HLT 2001, First International Conference on Human Language Technology Research, pages 253–257, San Diego, CA. [find](https://scholar.google.com/scholar?q=Michele+Banko+and+Eric+Brill%2E+2001%2E+Mitigating+the+paucity%2Dof%2Ddata+problem%3A+Exploring+the+effect+of+training+corpus+size+on+classifier+performance+for+natural+language+processing%2E+In+Proceedings+of+HL)
4. Adriane Boyd. 2010. EAGLE: an error-annotated corpus of beginning learner German. In Proceedings of LREC-10, Valetta, Malta. [find](https://scholar.google.com/scholar?q=Adriane+Boyd%2E+2010%2E+EAGLE%3A+an+error%2Dannotated+corpus+of+beginning+learner+German%2E+In+Proceedings+of+LREC%2D10%2C+Valetta%2C+Malta%2E)
5. Walter Daelemans, Antal van den Bosch, and Jakub Zavrel. 1999. Forgetting exceptions is harmful in language learning. Machine Learning, 34:11–41. [↗](https://doi.org/10.1023/a:1007585615670)
6. Walter Daelemans, Jakub Zavrel, Ko van der Sloot, and Antal van den Bosch. 2010. Timbl: Tilburg memory based learner, version 6.3, reference guide. Technical report, ILK Research Group. Technical Report Series no. 10-01. [find](https://scholar.google.com/scholar?q=Walter+Daelemans%2C+Jakub+Zavrel%2C+Ko+van+der+Sloot%2C+and+Antal+van+den+Bosch%2E+2010%2E+Timbl%3A+Tilburg+memory+based+learner%2C+version+6%2E3%2C+reference+guide%2E+Technical+report%2C+ILK+Research+Group%2E+Technical+Repo)
7. Robert Dale and Adam Kilgarriff. 2011. Helping our own: The HOO 2011 pilot shared task. In Proceedings of the Generation Challenges Session at the 13th European Workshop on Natural Language Generation, pages 242–249, Nancy, France, September. [find](https://scholar.google.com/scholar?q=Robert+Dale+and+Adam+Kilgarriff%2E+2011%2E+Helping+our+own%3A+The+HOO+2011+pilot+shared+task%2E+In+Proceedings+of+the+Generation+Challenges+Session+at+the+13th+European+Workshop+on+Natural+Language+Generation)
8. Ana Dı́az-Negrillo and Jesús Fernández-Domı́nguez. 2006. Error tagging systems for learner corpora. Spanish Journal of Applied Linguistics (RESLA), 19:83–102. [find](https://scholar.google.com/scholar?q=Ana+D%C4%B1%CC%81az%2DNegrillo+and+Jesu%CC%81s+Ferna%CC%81ndez%2DDom%C4%B1%CC%81nguez%2E+2006%2E+Error+tagging+systems+for+learner+corpora%2E+Spanish+Journal+of+Applied+Linguistics+%28RESLA%29%2C+19%3A83%E2%80%93102%2E)
9. Markus Dickinson, Ross Israel, and Sun-Hee Lee. 2011. Developing methodology for Korean particle error detection. In Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications, pages 81–86, Portland, OR, June. [find](https://scholar.google.com/scholar?q=Markus+Dickinson%2C+Ross+Israel%2C+and+Sun%2DHee+Lee%2E+2011%2E+Developing+methodology+for+Korean+particle+error+detection%2E+In+Proceedings+of+the+Sixth+Workshop+on+Innovative+Use+of+NLP+for+Building+Educational)
10. Markus Dickinson. 2011. On morphological analysis for learner language, focusing on Russian. Research on Language and Computation, 8(4):273–298. [↗](https://doi.org/10.1007/s11168-011-9079-0)
11. Glenn Fulcher. 1997. An English language placement test: issues in reliability and validity. Language Testing, 14(2):113–138. [↗](https://doi.org/10.1177/026553229701400201)
12. Yoav Goldberg and Michael Elhadad. 2011. Joint Hebrew segmentation and parsing using a PCFGLA lattice parser. In Proceedings of ACL-HLT, pages 704– 709, Portland, OR, June. [find](https://scholar.google.com/scholar?q=Yoav+Goldberg+and+Michael+Elhadad%2E+2011%2E+Joint+Hebrew+segmentation+and+parsing+using+a+PCFGLA+lattice+parser%2E+In+Proceedings+of+ACL%2DHLT%2C+pages+704%E2%80%93+709%2C+Portland%2C+OR%2C+June%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2011-goldberg-joint-hebrew-segmentation-parsing-pcfgla/)
13. Gintarè Grigonytè, João Paulo Cordeiro, Gaël Dias, Rumen Moraliyski, and Pavel Brazdil. 2010. Paraphrase alignment for synonym evidence discovery. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING), pages 403– 411, Beijing, China. [find](https://scholar.google.com/scholar?q=Gintare%CC%80+Grigonyte%CC%80%2C+Joa%CC%83o+Paulo+Cordeiro%2C+Gae%CC%88l+Dias%2C+Rumen+Moraliyski%2C+and+Pavel+Brazdil%2E+2010%2E+Paraphrase+alignment+for+synonym+evidence+discovery%2E+In+Proceedings+of+the+23rd+International+Conferen)
14. John A. Hawkins and Paula Buttery. 2010. Criterial features in learner corpora: Theory and illustrations. English Profile Journal, 1(1):1–23. [↗](https://doi.org/10.1017/s2041536210000103)
15. John A. Hawkins and Luna Filipović. 2010. Criterial Features in L2 English: Specifying the Reference Levels of the Common European Framework. Cambridge University Press. [find](https://scholar.google.com/scholar?q=John+A%2E+Hawkins+and+Luna+Filipovic%CC%81%2E+2010%2E+Criterial+Features+in+L2+English%3A+Specifying+the+Reference+Levels+of+the+Common+European+Framework%2E+Cambridge+University+Press%2E)
16. Alon Itai and Shuly Wintner. 2008. Language resources for Hebrew. Language Resources and Evaluation, 42(1):75–98. [↗](https://doi.org/10.1007/s10579-007-9050-8)
17. Alla Rozovskaya and Dan Roth. 2011. Algorithm selection and model adaptation for ESL correction tasks. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 924–933, Portland, OR, June. [find](https://scholar.google.com/scholar?q=Alla+Rozovskaya+and+Dan+Roth%2E+2011%2E+Algorithm+selection+and+model+adaptation+for+ESL+correction+tasks%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human)
18. Khalil Sima’an, Alon Itai, Yoad Winter, Alon Altman, and N. Nativ. 2001. Building a tree-bank of Modern Hebrew text. Traitment Automatique des Langues, 42(2). [find](https://scholar.google.com/scholar?q=Khalil+Sima%E2%80%99an%2C+Alon+Itai%2C+Yoad+Winter%2C+Alon+Altman%2C+and+N%2E+Nativ%2E+2001%2E+Building+a+tree%2Dbank+of+Modern+Hebrew+text%2E+Traitment+Automatique+des+Langues%2C+42%282%29%2E)
19. Joel Tetreault and Martin Chodorow. 2008. The ups and downs of preposition error detection in ESL writing. In Proceedings of COLING-08, Manchester. [find](https://scholar.google.com/scholar?q=Joel+Tetreault+and+Martin+Chodorow%2E+2008%2E+The+ups+and+downs+of+preposition+error+detection+in+ESL+writing%2E+In+Proceedings+of+COLING%2D08%2C+Manchester%2E)
20. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180–189, Portland, OR, June. [find](https://scholar.google.com/scholar?q=Helen+Yannakoudakis%2C+Ted+Briscoe%2C+and+Ben+Medlock%2E+2011%2E+A+new+dataset+and+method+for+automatically+grading+ESOL+texts%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+L)
21. Shlomo Yona and Shuly Wintner. 2008. A finite-state morphological grammar of Hebrew. Natural Language Engineering, 14(2):173–190. [↗](https://doi.org/10.1017/s1351324906004384) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-yona-finite-state-morphological-grammar-hebrew/)