[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/donati-2024-generation-evaluation-english-grammar-multiple/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Generation and Evaluation of English Grammar Multiple-Choice Cloze Exercises

# Generation and Evaluation of English Grammar Multiple-Choice Cloze Exercises

**Authors:** Nicolò Donati, Matteo Periani, Paolo Di Natale, Giuseppe Savino, Paolo Torroni · **Year:** 2024 · **Venue:** Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024) **Link:** [aclanthology.org/2024.clicit-1.39](https://aclanthology.org/2024.clicit-1.39/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology)

[Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024)](https://aclanthology.org/volumes/2024.clicit-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2024.clicit-1.39/)
- [PDF](https://aclanthology.org/2024.clicit-1.39.pdf)

**Authors (5)**

1. Nicolò Donati [Semantic Scholar](https://www.semanticscholar.org/author/2338997617)
2. Matteo Periani [Semantic Scholar](https://www.semanticscholar.org/author/2338996926)
3. Paolo Di Natale [Semantic Scholar](https://www.semanticscholar.org/author/2486760562)
4. Giuseppe Savino [Semantic Scholar](https://www.semanticscholar.org/author/2338996967)
5. Paolo Torroni [Semantic Scholar](https://www.semanticscholar.org/author/2286947191)

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

The paper fine-tunes Llama 3 8B (4-bit, QLoRA) to write English grammar multiple-choice cloze items (a sentence with a gap, one key, one or more distractors) for 19 grammar topics. Earlier systems used rules or fixed part-of-speech patterns, and the authors find their outputs creative but dull. The training data are the cleaned SC-Ques set, which has about 300k items; after removing near-duplicates only about 30% remains, and the least frequent topic is upsampled with GPT-4 output. Topics are assigned by a spaCy dependency matcher, by the overlap of the topic span with the key span. Two rule-based checks are proposed. Structural compliance checks that the gap is in the right place and that the distractor is homogeneous: same lemma for inflectional topics, or a word from a short admitted list for free-morpheme topics. Diversity is measured with Self-BLEU on 2- to 5-grams. In the evaluation, 50 items per topic were generated. The automatic check reaches 95% F1 against human structure labels (precision 98%, recall 91%). Averaged over topics, 79% of items pass the whole human review and the mean Self-BLEU is 7%. The main error is a meaningless sentence (75% of the errors).

## Ideas

- I1. `en/src/morph.rs`: the homogeneity test of the paper as a lemma check. A distractor is homogeneous when its lemma equals the key's lemma for inflectional topics. Count how often the test passes on the school exercises, and check the failures by hand.
- I2. Structural compliance as a rule set with a measured F1: write the gap-position and key-form checks over the UD tree (`en/src/parse.rs`), label 100 school items by hand, and report precision and recall against those labels, as the paper does.
- I3. Topic labels by a tree-pattern matcher over UD relations rather than a spaCy matcher: a fixed table of patterns (for example an auxiliary plus past participle for the perfect), applied in order, with the first match winning. The key-span overlap rule is kept as it is.
- I4. A Self-BLEU-like repetition count over 2- to 5-grams for generated school questions, with a fixed threshold, to catch near-duplicate items before they reach the lessons (`world/src/knowledge.rs`).

## For Mova

Exercise generation for the school sits in `world/src/knowledge.rs` (lessons, exercises and tests). Morphology and the lemma test belong to `en/src/morph.rs`. Tree patterns for topics belong to `en/src/parse.rs` and the UD trees of [20. Syntax: the Universal Dependencies frame](https://md.mova.elamur.ai/free/en/docs/languages/en/20-syntax-overview/). Topic chapters are [13. Verbs](https://md.mova.elamur.ai/free/en/docs/languages/en/13-verbs/), [12. Pronouns, determiners and numerals](https://md.mova.elamur.ai/free/en/docs/languages/en/12-pronouns-determiners-numerals/) (possessives and quantifiers) and [26. Questions](https://md.mova.elamur.ai/free/en/docs/languages/en/26-questions/) (WH-questions). Measures: lemma-test pass rate, precision and recall of the rule checks against 100 hand-labelled items, and the repetition rate, run twice for determinism.

Topics: [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{donati2024generation,
  title = {{Generation and Evaluation of English Grammar Multiple-Choice Cloze Exercises}},
  author = {Donati, Nicolò and Periani, Matteo and Natale, Paolo Di and Savino, Giuseppe and Torroni, Paolo},
  booktitle = {Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024)},
  year = {2024},
  url = {https://aclanthology.org/2024.clicit-1.39/}
}
```

- [← previous Bias in polar questions: Evidence from English and German production experiments](https://md.mova.elamur.ai/free/en/docs/papers/domaneschi-2017-bias-polar-questions-evidence-english/)
- [next → DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://md.mova.elamur.ai/free/en/docs/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

36 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. H. G. Widdowson, Teaching Language as Communication, Oxford University Press, Oxford, 1978. [find](https://scholar.google.com/scholar?q=H%2E+G%2E+Widdowson%2C+Teaching+Language+as+Communication%2C+Oxford+University+Press%2C+Oxford%2C+1978%2E)
2. H. G. Widdowson, Explorations in Applied Linguistics, Oxford University Press, Oxford, 1979. [find](https://scholar.google.com/scholar?q=H%2E+G%2E+Widdowson%2C+Explorations+in+Applied+Linguistics%2C+Oxford+University+Press%2C+Oxford%2C+1979%2E)
3. W. Gan, Z. Qi, J. Wu, J. C.-W. Lin, Large language models in education: Vision and opportunities, in: 2023 IEEE International Conference on Big Data (BigData), 2023, pp. 4776–4785. doi: 10. 1109/BigData59044.2023.10386291 . [find](https://scholar.google.com/scholar?q=W%2E+Gan%2C+Z%2E+Qi%2C+J%2E+Wu%2C+J%2E+C%2E%2DW%2E+Lin%2C+Large+language+models+in+education%3A+Vision+and+opportunities%2C+in%3A+2023+IEEE+International+Conference+on+Big+Data+%28BigData%29%2C+2023%2C+pp%2E+4776%E2%80%934785%2E+doi%3A+10%2E+1109%2FBigDa)
4. Q. Wang, R. Rose, N. Orita, A. Sugawara, Automated generation of multiple-choice cloze questions for assessing english vocabulary using gpt-turbo 3.5, 2024. URL: https://arxiv.org/abs/2403. 02078. arXiv:2403.02078 . [↗](https://arxiv.org/abs/2403.02078)
5. P. Chomphooyod, A. Suchato, N. Tuaycharoen, P. Punyabukkana, English grammar multiple-choice question generation using text-to-text transfer transformer, Computers and Education: Artificial Intelligence 5 (2023) 100158. URL: https://www.sciencedirect.com/science/ article/pii/S2666920X23000371. doi: https://doi.org/10.1016/j.caeai.2023.100158 . [↗](https://doi.org/10.1016/j.caeai.2023.100158)
6. Y. Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, Y. Yu, Texygen: A benchmarking platform for text generation models, 2018. URL: https://arxiv.org/abs/ 1802.01886. arXiv:1802.01886 . [↗](https://arxiv.org/abs/1802.01886)
7. D. H. Hymes, On communicative competence, in: J. B. Pride, J. Holmes (Eds.), Sociolinguistics. Selected Readings, Penguin, Harmondsworth, 1972, pp. 269–293. [find](https://scholar.google.com/scholar?q=D%2E+H%2E+Hymes%2C+On+communicative+competence%2C+in%3A+J%2E+B%2E+Pride%2C+J%2E+Holmes+%28Eds%2E%29%2C+Sociolinguistics%2E+Selected+Readings%2C+Penguin%2C+Harmondsworth%2C+1972%2C+pp%2E+269%E2%80%93293%2E)
8. Q. Liu, Y. Huang, Z. Liu, S. Huang, J. Chen, X. Zhao, G. Lin, Y. Zhou, W. Luo, Sc-ques: A sentence completion question dataset for english as a second language learners, in: C. Frasson, P. Mylonas, C. Trous-sas (Eds.), Augmented Intelligence and Intelligent Tutoring Systems, Springer Nature Switzerland, Cham, 2023, pp. 678–690. [find](https://scholar.google.com/scholar?q=Q%2E+Liu%2C+Y%2E+Huang%2C+Z%2E+Liu%2C+S%2E+Huang%2C+J%2E+Chen%2C+X%2E+Zhao%2C+G%2E+Lin%2C+Y%2E+Zhou%2C+W%2E+Luo%2C+Sc%2Dques%3A+A+sentence+completion+question+dataset+for+english+as+a+second+language+learners%2C+in%3A+C%2E+Frasson%2C+P%2E+Mylonas%2C+C%2E)
9. L. F. Bachman, Fundamental Considerations in Language Testing, Oxford University Press, Oxford, 1990. [find](https://scholar.google.com/scholar?q=L%2E+F%2E+Bachman%2C+Fundamental+Considerations+in+Language+Testing%2C+Oxford+University+Press%2C+Oxford%2C+1990%2E)
10. J. E. Purpura, Assessing Grammar, Cambridge Language Assessment, Cambridge University Press, 2004. [find](https://scholar.google.com/scholar?q=J%2E+E%2E+Purpura%2C+Assessing+Grammar%2C+Cambridge+Language+Assessment%2C+Cambridge+University+Press%2C+2004%2E)
11. G. Fulcher, G. Fulcher, Practical Language Testing, 1st ed., Routledge, 2010. doi: 10.4324/ 980203767399 . [find](https://scholar.google.com/scholar?q=G%2E+Fulcher%2C+G%2E+Fulcher%2C+Practical+Language+Testing%2C+1st+ed%2E%2C+Routledge%2C+2010%2E+doi%3A+10%2E4324%2F+980203767399+%2E)
12. V.-M. Pho, T. André, A.-L. Ligozat, B. Grau, G. Il-louz, T. François, Multiple choice question corpus analysis for distractor characterization, in: N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, S. Piperidis (Eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), European Language Resources Association (ELRA), Reykjavik, Iceland, 2014, pp. 4284– 4291. URL: [http://www.lrec-conf.org/proceedings/](http://www.lrec-conf.org/proceedings/) lrec2014/pdf/692_Paper.pdf. [↗](http://www.lrec-conf.org/proceedings/)
13. E. Sumita, F. Sugaya, S. Yamamoto, Measuring non-native speakers’ proficiency of english by using a test with automatically-generated fill-in-the-blank questions (2005). doi: 10.3115/1609829. 1609839 . [↗](https://doi.org/10.3115/1609829)
14. J. Brown, G. A. Frishkoff, M. Eskénazi, Automatic question generation for vocabulary assessment, in: HLT/EMNLP 2005, Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, 6-8 October 2005, Vancouver, British Columbia, Canada, The Association for Computational Linguistics, 2005, pp. 819–826. URL: https://aclanthology.org/H05-1103/. [↗](https://aclanthology.org/H05-1103/)
15. S. Smith, A. P.V.S, A. Kilgarriff, Gap-fill tests for language learners: Corpus-driven item generation, 2010. URL: [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:61531901. [↗](https://api.semanticscholar.org/)
16. M. Majumder, S. K. Saha, A system for generating multiple choice questions: With a novel approach for sentence selection, in: H. Chen, Y. Tseng, Y. Matsumoto, L. Wong (Eds.), Proceedings of the 2nd Workshop on Natural Language Processing Techniques for Educational Applications, NLP-TEA@ACL/IJCNLP, Beijing, China, July 31, 2015, Association for Computational Linguistics, 2015, pp. 64–72. URL: https://doi.org/10.18653/v1/W15-4410. doi: 10.18653/V1/W15-4410 . [↗](https://doi.org/10.18653/v1/w15-4410)
17. M. Majumder, S. K. Saha, A system for generating multiple choice questions: With a novel approach for sentence selection, in: H. Chen, Y. Tseng, Y. Matsumoto, L. Wong (Eds.), Proceedings of the 2nd Workshop on Natural Language Processing Techniques for Educational Applications, NLP-TEA@ACL/IJCNLP, Beijing, China, July 31, 2015, Association for Computational Linguistics, 2015, pp. 64–72. URL: https://doi.org/10.18653/v1/W15-4410. doi: 10.18653/V1/W15-4410 . [↗](https://doi.org/10.18653/v1/w15-4410)
18. T. Goto, T. Kojiri, T. Watanabe, T. Iwata, T. Yamada, Automatic generation system of multiple-choice cloze questions and its evaluation, Knowledge Management & E-Learning: An International Journal 2 (2010) 210–224. URL: [https://api.semanticscholar.org/CorpusID:15482954](https://api.semanticscholar.org/CorpusID:15482954). [↗](https://api.semanticscholar.org/CorpusID:15482954)
19. J. D. Lafferty, A. McCallum, F. C. N. Pereira, Conditional random fields: Probabilistic models for segmenting and labeling sequence data, in: C. E. Brodley, A. P. Danyluk (Eds.), Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, Morgan Kaufmann, 2001, pp. 282–289. [find](https://scholar.google.com/scholar?q=J%2E+D%2E+Lafferty%2C+A%2E+McCallum%2C+F%2E+C%2E+N%2E+Pereira%2C+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2C+in%3A+C%2E+E%2E+Brodley%2C+A%2E+P%2E+Danyluk+%28Eds%2E%29%2C+Proceedings+of+the+Ei)
20. S. K. Bitew, J. Deleu, A. S. Doğruöz, C. Develder, T. Demeester, Learning from partially annotated data: Example-aware creation of gap-filling exercises for language learning, in: E. Kochmar, J. Burstein, A. Horbach, R. Laarmann-Quante, N. Madnani, A. Tack, V. Yaneva, Z. Yuan, T. Zesch (Eds.), Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, BEA@ACL 2023, Toronto, Canada, 13 July 2023, Association for Computational Linguistics, 2023, pp. 598–609. URL: https://doi.org/10. 18653/v1/2023.bea-1.51. doi: 10.18653/V1/2023. BEA-1.51 . [↗](https://doi.org/10.18653/v1/2023)
21. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, 2019. URL: https://arxiv.org/abs/1907.11692. arXiv:1907.11692 . [↗](https://arxiv.org/abs/1907.11692)
22. S. Matsumori, K. Okuoka, R. Shibata, M. Inoue, Y. Fukuchi, M. Imai, Mask and cloze: Automatic open cloze question generation using a masked language model, IEEE Access 11 (2023) 9835–9850. URL: http://dx.doi.org/10.1109/ACCESS.2023.3239005. doi: 10.1109/ access.2023.3239005 . [↗](https://doi.org/10.1109/access.2023.3239005)
23. P. Chomphooyod, A. Suchato, N. Tuaycharoen, P. Punyabukkana, English grammar multiple-choice question generation using text-to-text transfer transformer, Comput. Educ. Artif. Intell. 5 (2023) 100158. URL: https://doi.org/10.1016/j.caeai.2023. 100158. doi: 10.1016/J.CAEAI.2023.100158 . [↗](https://doi.org/10.1016/j.caeai.2023)
24. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, 2023. URL: https://arxiv.org/ abs/1706.03762. arXiv:1706.03762 . [↗](https://arxiv.org/abs/1706.03762)
25. V. Slavuj, L. Nacinovic Prskalo, M. Brkic Bakaric, Automatic generation of language exercises based on a universal methodology: An analysis of possibilities, Bulletin of the Transilvania University of Brasov. Series IV: Philology and Cultural Studies 14 (63) (2022) 29–48. doi: 10.31926/but.pcs.2021. 63.14.2.3 . [↗](https://doi.org/10.31926/but.pcs.2021)
26. A. Malafeev, Language exercise generation, International Journal of Conceptual Structures and Smart Applications 2 (2014) 20–35. doi: 10.4018/ IJCSSA.2014070102 . [find](https://scholar.google.com/scholar?q=A%2E+Malafeev%2C+Language+exercise+generation%2C+International+Journal+of+Conceptual+Structures+and+Smart+Applications+2+%282014%29+20%E2%80%9335%2E+doi%3A+10%2E4018%2F+IJCSSA%2E2014070102+%2E)
27. D. Perrett, D. March, An evidence-based approach to distractor generation in multiple-choice language tests, 2019. doi: 10.13140/RG.2.2.22779. 16165 . [↗](https://doi.org/10.13140/rg.2.2.22779)
28. K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: P. Isabelle, E. Charniak, D. Lin (Eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002, pp. 311–318. URL: https://aclanthology.org/P02-1040. doi: 10.3115/ 1073083.1073135 . [↗](https://aclanthology.org/P02-1040/) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
29. C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, 2004, pp. 74–81. URL: https://aclanthology.org/W04-1013. [↗](https://aclanthology.org/W04-1013/)
30. S. Banerjee, A. Lavie, METEOR: An automatic metric for MT evaluation with improved correlation with human judgments, in: J. Goldstein, A. Lavie, C.-Y. Lin, C. Voss (Eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Association for Computational Linguistics, Ann Arbor, Michigan, 2005, pp. 65–72. URL: https://aclanthology.org/W05-0909. [↗](https://aclanthology.org/W05-0909/) [our card](https://md.mova.elamur.ai/free/en/docs/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
31. Meta, Introducing Meta Llama 3: The most capable openly available LLM to date, [https://ai.meta.com/](https://ai.meta.com/) blog/meta-llama-3/, April 2024. [↗](https://ai.meta.com/)
32. K. Tirumala, D. Simig, A. Aghajanyan, A. Morcos, D4: Improving llm pretraining via document deduplication and diversification, in: A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, S. Levine (Eds.), Advances in Neural Information Processing Systems, volume 36, Curran Associates, Inc., 2023, pp. 53983–53995. URL: [https://proceedings.neurips.cc/paper_files/paper/2023/file/](https://proceedings.neurips.cc/paper_files/paper/2023/file/) a8f8cbd7f7a5fb2c837e578c75e5b615-Paper-Datasets_ and_Benchmarks.pdf. [↗](https://proceedings.neurips.cc/paper_files/paper/2023/file/)
33. O. et al., Gpt-4 technical report, 2024. URL: https://arxiv.org/abs/2303.08774. arXiv:2303.08774 . [↗](https://arxiv.org/abs/2303.08774)
34. T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, Qlora: Efficient finetuning of quantized llms, 2023. URL: https://arxiv.org/abs/2305.14314. arXiv:2305.14314 . [↗](https://arxiv.org/abs/2305.14314)
35. Z. Fu, W. Lam, A. M.-C. So, B. Shi, A theoretical analysis of the repetition problem in text generation, Proceedings of the AAAI Conference on Artificial Intelligence 35 (2021) 12848–12856. URL: https://ojs.aaai.org/index.php/AAAI/article/ view/17520. doi: 10.1609/aaai.v35i14.17520 . [↗](https://doi.org/10.1609/aaai.v35i14.17520)
36. Z. Xu, S. Jain, M. Kankanhalli, Hallucination is inevitable: An innate limitation of large language models, 2024. URL: https://arxiv.org/abs/2401.11817. arXiv:2401.11817 . [↗](https://arxiv.org/abs/2401.11817)