[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/berzak-2020-starc-structured-annotations-reading-comprehension/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / STARC: Structured Annotations for Reading Comprehension

# STARC: Structured Annotations for Reading Comprehension

**Authors:** [Yevgeni Berzak](https://md.mova.elamur.ai/free/en/docs/papers/authors/yevgeni-berzak/), Jonathan Malmaud, [Roger Levy](https://md.mova.elamur.ai/free/en/docs/papers/authors/roger-levy/) · **Year:** 2020 · **Venue:** ACL 2020 **Link:** [aclanthology.org/2020.acl-main.507](https://aclanthology.org/2020.acl-main.507/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (paper); OneStopQA's own licence to be checked — tests only

[**19** citations](https://openalex.org/works?filter=cites:W3022179588) openalex · 2026-10-19[**19** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/berzak-2020-starc-structured-annotations-reading-comprehension/) of the 140673 papers whose bibliographies we read[Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://aclanthology.org/volumes/2020.acl-main/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2020.acl-main.507/)
- [PDF](https://aclanthology.org/2020.acl-main.507.pdf)
- [DOI 10.18653/v1/2020.acl-main.507](https://doi.org/10.18653/v1/2020.acl-main.507)
- [OpenAlex](https://openalex.org/W3022179588)

**Authors (3)**

1. [Yevgeni Berzak](https://openalex.org/A5023679744) [ORCID](https://orcid.org/0000-0003-4474-1727) · [Semantic Scholar](https://www.semanticscholar.org/author/1908728) · h 15 · 608 citations · 72 works
2. [Jonathan Malmaud](https://openalex.org/A5001030838) [ORCID](https://orcid.org/0000-0002-5352-2086) · [Semantic Scholar](https://www.semanticscholar.org/author/3274291) · h 12 · 1413 citations · 31 works
3. [Roger Levy](https://openalex.org/A5090215557) [ORCID](https://orcid.org/0000-0002-4493-8864) · [Semantic Scholar](https://www.semanticscholar.org/author/50007746) · h 56 · 23375 citations · 434 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

STARC is a scheme for four-option reading questions in which the options have fixed roles tied to marked text spans: A is right and rests on a «critical span»; B mentions the critical span but misunderstands it; C refers to a separate «distractor span»; D has no support in the text. Questions avoid arithmetic, outside knowledge and speculation, and options are written so that elimination tricks do not work. OneStopQA applies it to 30 Guardian articles in three difficulty versions (162 paragraphs, 486 questions, 1,458 question–paragraph pairs). Ablations show the value: without the passage RoBERTa guesses 37.2% on OneStopQA but 47.1% on RACE; removing the critical span drops accuracy towards chance, as it should. Crowd judgements find 18.3% of RACE questions without a single right answer against 3.4% for OneStopQA, giving approximate ceilings of 88.8% and 97.9%; the structured options also show what kind of misreading each wrong choice is.

## Ideas

- I1. Classify the student's wrong answers with the STARC types, by rule from the trace: the chosen option matched the frame that answers the question but with a wrong role or polarity (B — misread the right place), matched another frame of the lesson (C — wrong place), or matched nothing (D — guess). Add the split to `world knowledge diagnose`; B-type errors point at role and negation handling, C-type at retrieval and salience.
- I2. Right for the right reason: when lessons and questions are made (the teacher writes them, docs/school-making.md), also store the critical sentence for each question. Then check that the frame the student used comes from that sentence; an answer right but resting on another sentence is counted apart. This is a cheap guard against lucky matches in the course numbers.
- I3. Critical-span ablation as a validity check of course questions: remove the critical sentence; the student should fall to guessing on that question. Questions still answered right are guessable from elsewhere or from the options and should be fixed or marked.
- I4. Ask the teacher to write options in the A/B/C/D structure for new courses, so B-type options test reading of the right sentence directly; check on a pilot that the student's score on B-type distractors differs from C and D.

## For Mova

[30. Pragmatics and discourse: overview](https://md.mova.elamur.ai/free/en/docs/languages/en/30-pragmatics-overview/) §9 (measurements). Code: `world/src/knowledge.rs` (diagnosis, evidence check), `world/src/school.rs`, `world/src/lessons.rs`; docs/school-making.md for the teacher's question format. Measures: error-type shares B/C/D per course, share of right answers resting on the critical sentence, accuracy with the critical sentence removed.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2020.acl-main.507](https://doi.org/10.18653/v1/2020.acl-main.507)

```
@inproceedings{berzak2020starc,
  title = {{STARC: Structured Annotations for Reading Comprehension}},
  author = {Berzak, Yevgeni and Malmaud, Jonathan and Levy, Roger},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.507},
  url = {https://doi.org/10.18653/v1/2020.acl-main.507}
}
```

- [← previous On Old English swa 'so'](https://md.mova.elamur.ai/free/en/docs/papers/beck-2023-old-english-em-swa-em/)
- [next → Abductive Commonsense Reasoning](https://md.mova.elamur.ai/free/en/docs/papers/bhagavatula-2019-abductive-commonsense-reasoning/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. Analyzing the behavior of visual question answering models. In EMNLP, pages 1955–1960. [find](https://scholar.google.com/scholar?q=Aishwarya+Agrawal%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+2016%2E+Analyzing+the+behavior+of+visual+question+answering+models%2E+In+EMNLP%2C+pages+1955%E2%80%931960%2E)
2. Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint arXiv:1610.00956. — [arXiv:1610.00956](https://arxiv.org/abs/1610.00956) [find](https://scholar.google.com/scholar?q=Ondrej+Bajgar%2C+Rudolf+Kadlec%2C+and+Jan+Kleindienst%2E+2016%2E+Embracing+data+abundance%3A+Booktest+dataset+for+reading+comprehension%2E+arXiv+preprint+arXiv%3A1610%2E00956%2E+%E2%80%94+%5BarXiv%3A1610%2E00956%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
3. Wei-Lun Chao, Hexiang Hu, and Fei Sha. 2018. Being negative but constructively: Lessons learnt from creating better visual question answering datasets. In NAACL-HLT, pages 431–441. [find](https://scholar.google.com/scholar?q=Wei%2DLun+Chao%2C+Hexiang+Hu%2C+and+Fei+Sha%2E+2018%2E+Being+negative+but+constructively%3A+Lessons+learnt+from+creating+better+visual+question+answering+datasets%2E+In+NAACL%2DHLT%2C+pages+431%E2%80%93441%2E)
4. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the CNN/Daily mail reading comprehension task. In ACL, pages 2358–2367. [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Jason+Bolton%2C+and+Christopher+D+Manning%2E+2016%2E+A+thorough+examination+of+the+CNN%2FDaily+mail+reading+comprehension+task%2E+In+ACL%2C+pages+2358%E2%80%932367%2E)
5. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In CVPR. [find](https://scholar.google.com/scholar?q=Yash+Goyal%2C+Tejas+Khot%2C+Douglas+Summers%2DStay%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+2017%2E+Making+the+V+in+VQA+matter%3A+Elevating+the+role+of+image+understanding+in+Visual+Question+Answering%2E+In+CVPR%2E)
6. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NeurIPS, pages 1693–1701. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+NeurIPS%2C+pages+1693%E2%80%931701%2E)
7. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The Goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301. [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Antoine+Bordes%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+The+Goldilocks+principle%3A+Reading+children%E2%80%99s+books+with+explicit+memory+representations%2E+arXiv+preprint+arXiv%3A1511%2E02301%2E)
8. Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. 2016. Revisiting visual question answering baselines. In ECCV, pages 727–739. Springer. [find](https://scholar.google.com/scholar?q=Allan+Jabri%2C+Armand+Joulin%2C+and+Laurens+Van+Der+Maaten%2E+2016%2E+Revisiting+visual+question+answering+baselines%2E+In+ECCV%2C+pages+727%E2%80%93739%2E+Springer%2E)
9. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+S+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+arXiv+preprint+arXiv%3A1705%2E03551%2E)
10. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In NAACL, pages 252–262. [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Snigdha+Chaturvedi%2C+Michael+Roth%2C+Shyam+Upadhyay%2C+and+Dan+Roth%2E+2018%2E+Looking+beyond+the+surface%3A+A+challenge+set+for+reading+comprehension+over+multiple+sentences%2E+In+NAACL%2C+pages+25) [our card](https://md.mova.elamur.ai/free/en/docs/papers/khashabi-2018-looking-beyond-surface-challenge-set/)
11. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (Automated Readability Index, Fog count and Flesch reading ease formula) for navy enlisted personnel. [find](https://scholar.google.com/scholar?q=J+Peter+Kincaid%2C+Robert+P+Fishburne+Jr%2C+Richard+L+Rogers%2C+and+Brad+S+Chissom%2E+1975%2E+Derivation+of+new+readability+formulas+%28Automated+Readability+Index%2C+Fog+count+and+Flesch+reading+ease+formula%29+for+)
12. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Redfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Matthew+Kelcey%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+N%2E+Toutanova)
13. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale reading comprehension dataset from examinations. In EMNLP, pages 785–794. [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2017%2E+RACE%3A+Large%2Dscale+reading+comprehension+dataset+from+examinations%2E+In+EMNLP%2C+pages+785%E2%80%93794%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lai-2017-race-large-scale-reading-comprehension/)
14. G. Harry MC Laughlin. 1969. SMOG grading-a new readability formula. Journal of Reading, 12(8):639– 646. [find](https://scholar.google.com/scholar?q=G%2E+Harry+MC+Laughlin%2E+1969%2E+SMOG+grading%2Da+new+readability+formula%2E+Journal+of+Reading%2C+12%288%29%3A639%E2%80%93+646%2E)
15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
16. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human-generated machine reading comprehension dataset. [find](https://scholar.google.com/scholar?q=Tri+Nguyen%2C+Mir+Rosenberg%2C+Xia+Song%2C+Jianfeng+Gao%2C+Saurabh+Tiwary%2C+Rangan+Majumder%2C+and+Li+Deng%2E+2016%2E+MS+MARCO%3A+A+human%2Dgenerated+machine+reading+comprehension+dataset%2E)
17. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, pages 2383–2392. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+EMNLP%2C+pages+2383%E2%80%932392%2E)
18. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, pages 193–203. [find](https://scholar.google.com/scholar?q=Matthew+Richardson%2C+Christopher+JC+Burges%2C+and+Erin+Renshaw%2E+2013%2E+MCTest%3A+A+challenge+dataset+for+the+open%2Ddomain+machine+comprehension+of+text%2E+In+EMNLP%2C+pages+193%E2%80%93203%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/missing-2013-richardson-mctest-challenge-dataset-open-domain/)
19. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. [find](https://scholar.google.com/scholar?q=Adam+Trischler%2C+Tong+Wang%2C+Xingdi+Yuan%2C+Justin+Harris%2C+Alessandro+Sordoni%2C+Philip+Bachman%2C+and+Kaheer+Suleman%2E+2017%2E+NewsQA%3A+A+machine+comprehension+dataset%2E)
20. Sowmya Vajjala and Ivana Lučić. 2018. OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification. Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 297–304. [find](https://scholar.google.com/scholar?q=Sowmya+Vajjala+and+Ivana+Luc%CC%8Cic%CC%81%2E+2018%2E+OneStopEnglish+corpus%3A+A+new+corpus+for+automatic+readability+assessment+and+text+simplification%2E+Proceedings+of+the+Thirteenth+Workshop+on+Innovative+Use+of+NL)
21. Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, David McAllester, and Dan Roth. 2019. Evidence sentence extraction for machine reading comprehension. In CoNLL, pages 696–707. [find](https://scholar.google.com/scholar?q=Hai+Wang%2C+Dian+Yu%2C+Kai+Sun%2C+Jianshu+Chen%2C+Dong+Yu%2C+David+McAllester%2C+and+Dan+Roth%2E+2019%2E+Evidence+sentence+extraction+for+machine+reading+comprehension%2E+In+CoNLL%2C+pages+696%E2%80%93707%2E)