[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/dasigi-2019-quoref-reading-comprehension-dataset-questions/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

# Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

**Authors:** [Pradeep Dasigi](https://md.mova.elamur.ai/free/en/docs/papers/authors/pradeep-dasigi/), [Nelson F. Liu](https://md.mova.elamur.ai/free/en/docs/papers/authors/nelson-f-liu/), Ana Marasović, [Noah A. Smith](https://md.mova.elamur.ai/free/en/docs/papers/authors/noah-a-smith/), [Matt Gardner](https://md.mova.elamur.ai/free/en/docs/papers/authors/matt-gardner/) · **Year:** 2019 · **Venue:** EMNLP 2019 (arXiv:1908.05803) **Link:** [arxiv.org/abs/1908.05803](https://arxiv.org/abs/1908.05803) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (paper); the paragraphs are Wikipedia text (CC BY-SA) — tests only

[**147** citations](https://openalex.org/works?filter=cites:W2968154628) openalex-snapshot · 2026-10-14[**109** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/dasigi-2019-quoref-reading-comprehension-dataset-questions/) of the 140673 papers whose bibliographies we readarXiv:1908.05803[licence](http://creativecommons.org/licenses/by-sa/4.0/)

- [Publisher's page](https://arxiv.org/abs/1908.05803)
- [PDF](https://arxiv.org/pdf/1908.05803)
- [arXiv 1908.05803](https://arxiv.org/abs/1908.05803)
- [OpenAlex](https://openalex.org/W2968154628)

**Authors (5)**

1. [Pradeep Dasigi](https://openalex.org/A5029074038) [ORCID](https://orcid.org/0000-0001-7127-1316) · h 20 · 3171 citations · 90 works
2. [Nelson F. Liu](https://openalex.org/A5031981513) h 13 · 3238 citations · 37 works
3. [Ana Marasovic](https://openalex.org/A5087098432) [ORCID](https://orcid.org/0009-0000-2502-9262) · h 16 · 3644 citations · 57 works
4. [Noah A. Smith](https://openalex.org/A5088517824) [ORCID](https://orcid.org/0000-0002-2310-6380) · h 93 · 38892 citations · 484 works
5. [Matt Gardner](https://openalex.org/A5035088083) [ORCID](https://orcid.org/0000-0001-8458-1727) · h 42 · 20076 citations · 99 works

Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart.

## What it is about

Quoref has over 24K span-answer questions on 4.7K Wikipedia paragraphs (about 40% film plot summaries, which mention many characters), written by crowd workers so that answering needs linking two mentions of one entity. To keep out questions answerable by surface cues, a BERT model trained on SQuAD ran in the background and a question was accepted only if that model got it wrong. In a sample of 100, 78% truly need coreference: pronominal 69%, nominal 54%, both 32%, commonsense 10%; about 10% of answers are several spans. Questions often name the target by a description instead of a name («the person who is defied by her daughter» → Hippolyta, from «Hippolyta was granted a daughter, Diana … Diana defies her mother»). The best model reached 70.5 F1 against 93.4 for humans.

## Ideas

- I1. Count, in `world knowledge diagnose`, which course questions can be answered only through a resolved pronoun or nominal (the remembered frame that matches the question has the answer role filled by a mention the resolver linked, not by a name). Report the share right among those separately. It tells how much of the student's score depends on `Ctx` resolution, and gives the coref work a course-level yardstick next to GUM CoNLL F1.
- I2. Read questions that refer to an entity through a description: a relative clause on *the person/character/animal who …* in the question is itself a frame to be matched («who is defied by her daughter» → find X with a daughter-frame and a *defy* frame whose patient is X). This needs the kin inverse (daughter of X ⇔ X is mother) from the possessive and kin nouns; the parser side exists (`world/src/ans.rs` handles possessives). Measure on course questions of the form «What is the name of the … who …».
- I3. Use the adversarial filter idea to grade Mova's own course questions: mark as «shallow» every question that a trivial baseline (the option sharing most words with one lesson sentence, or the nearest name to a question word) answers right, and report the student on deep questions only. A gain that is only on shallow questions is not reading.

## For Mova

[35. Reference and coreference](https://md.mova.elamur.ai/free/en/docs/languages/en/35-reference-and-coreference/) §10 (the resolver inside the frame reader), [Watching the SLM think: the diagnosis of its answers](https://md.mova.elamur.ai/free/en/docs/docs/mmm-diagnosis/) (the diagnosis steps). Code: `world/src/knowledge.rs` (diagnose split), `prag/src/frames.rs` (`Ctx`, relative clauses in questions), `world/src/ans.rs`. Measures: course test counts on pronoun-dependent and deep questions; Quoref dev only as an outside test set, given the Wikipedia licence.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 1908.05803](https://arxiv.org/abs/1908.05803)

```
@misc{dasigi2019quoref,
  title = {{Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning}},
  author = {Dasigi, Pradeep and Liu, Nelson F. and Marasovic, Ana and Smith, Noah A. and Gardner, Matt},
  year = {2019},
  eprint = {1908.05803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.05803}
}
```

- [← previous Constructing a Lexicon of English Discourse Connectives](https://md.mova.elamur.ai/free/en/docs/papers/das-2018-constructing-lexicon-english-discourse-connectives/)
- [next → Bible English: Chapters on Old and Disused Expressions in the Authorized Version of the Scriptures and the Book of Common Prayer](https://md.mova.elamur.ai/free/en/docs/papers/davies-1875-bible-english-chapters-old/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rahul Aralikatte and Anders Søgaard. 2019. Model-based annotation of coreference. arXiv:1906.10724. [↗](https://arxiv.org/abs/1906.10724)
2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. MS MARCO: A human generated machine reading comprehension dataset. In NeurIPS. [find](https://scholar.google.com/scholar?q=Payal+Bajaj%2C+Daniel+Campos%2C+Nick+Craswell%2C+Li+Deng%2C+Jianfeng+Gao%2C+Xiaodong+Liu%2C+Rangan+Majumder%2C+Andrew+McNamara%2C+Bhaskar+Mitra%2C+Tri+Nguyen%2C+et+al%2E+2016%2E+MS+MARCO%3A+A+human+generated+machine+reading+co)
3. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Hong+Chen%2C+Zhenhua+Fan%2C+Hao+Lu%2C+Alan+Yuille%2C+and+Shu+Rong%2E+2018%2E+PreCo%3A+A+large%2Dscale+dataset+in+preschool+vocabulary+for+coreference+resolution%2E+In+Proc%2E+of+EMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/chen-2018-preco-large-scale-dataset-preschool/)
4. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Christopher+Clark+and+Matt+Gardner%2E+2018%2E+Simple+and+effective+multi%2Dparagraph+reading+comprehension%2E+In+Proc%2E+of+ACL%2E)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proc%2E+of+NAACL%2E)
6. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL. [↗](https://doi.org/10.48550/arxiv.1903.00161) [our card](https://md.mova.elamur.ai/free/en/docs/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
7. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proc. of Workshop for NLP Open Source Software. [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Joel+Grus%2C+Mark+Neumann%2C+Oyvind+Tafjord%2C+Pradeep+Dasigi%2C+Nelson+F%2E+Liu%2C+Matthew+Peters%2C+Michael+Schmitz%2C+and+Luke+Zettlemoyer%2E+2018%2E+AllenNLP%3A+A+deep+semantic+natural+language+processing)
8. Abbas Ghaddar and Phillippe Langlais. 2016. Wiki-Coref: An English coreference-annotated corpus of Wikipedia articles. In Proc. of LREC. [find](https://scholar.google.com/scholar?q=Abbas+Ghaddar+and+Phillippe+Langlais%2E+2016%2E+Wiki%2DCoref%3A+An+English+coreference%2Dannotated+corpus+of+Wikipedia+articles%2E+In+Proc%2E+of+LREC%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ghaddar-2016-wikicoref-english-coreference-annotated-corpus/)
9. Anupam Guha, Mohit Iyyer, Danny Bouman, and Jordan Boyd-Graber. 2015. Removing the training wheels: A coreference dataset that entertains humans and challenges computers. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Anupam+Guha%2C+Mohit+Iyyer%2C+Danny+Bouman%2C+and+Jordan+Boyd%2DGraber%2E+2015%2E+Removing+the+training+wheels%3A+A+coreference+dataset+that+entertains+humans+and+challenges+computers%2E+In+Proc%2E+of+NAACL%2E)
10. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+Bowman%2C+and+Noah+A+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+Proc%2E+of+NAACL%2E)
11. Mandar S. Joshi, Eunsol Choi, Daniel S. Weld, and Luke S. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Mandar+S%2E+Joshi%2C+Eunsol+Choi%2C+Daniel+S%2E+Weld%2C+and+Luke+S%2E+Zettlemoyer%2E+2017%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+Proc%2E+of+ACL%2E)
12. Divyansh Kaushik and Zachary Chase Lipton. 2018. How much reading does reading comprehension require? A critical investigation of popular benchmarks. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Divyansh+Kaushik+and+Zachary+Chase+Lipton%2E+2018%2E+How+much+reading+does+reading+comprehension+require%3F+A+critical+investigation+of+popular+benchmarks%2E+In+Proc%2E+of+EMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/kaushik-2018-how-much-reading-does-reading/)
13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A benchmark for question answering research. In TACL. [↗](https://doi.org/10.1162/tacl_a_00276)
14. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv:1906.08237. [↗](https://arxiv.org/abs/1906.08237)
15. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2017%2E+RACE%3A+Large%2Dscale+ReAding+comprehension+dataset+from+examinations%2E+In+Proc%2E+of+EMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lai-2017-race-large-scale-reading-comprehension/)
16. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining local convolution with global self-attention for reading comprehension. In Proc. of ICLR. [↗](https://doi.org/10.48550/arxiv.1804.09541)
17. Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Nelson+F%2E+Liu%2C+Roy+Schwartz%2C+and+Noah+A%2E+Smith%2E+2019%2E+Inoculation+by+fine%2Dtuning%3A+A+method+for+analyzing+challenge+datasets%2E+In+Proc%2E+of+NAACL%2E)
18. Courtney Napoles, Matthew R. Gormley, and Benjamin Van Durme. 2012. Annotated Gigaword. Technical report. [find](https://scholar.google.com/scholar?q=Courtney+Napoles%2C+Matthew+R%2E+Gormley%2C+and+Benjamin+Van+Durme%2E+2012%2E+Annotated+Gigaword%2E+Technical+report%2E)
19. Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proc. of NAACL. [↗](https://doi.org/10.5283/epub.43420)
20. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the ARRAU corpus. In Proc. of CRAC Workshop. [find](https://scholar.google.com/scholar?q=Massimo+Poesio%2C+Yulia+Grishina%2C+Varada+Kolhatkar%2C+Nafise+Moosavi%2C+Ina+Roesiger%2C+Adam+Roussel%2C+Fabian+Simonjetz%2C+Alexandra+Uma%2C+Olga+Uryupina%2C+Juntao+Yu%2C+and+Heike+Zinsmeister%2E+2018%2E+Anaphora+resolutio)
21. Sameer S. Pradhan, Lance Ramshaw, Ralph Weischedel, Jessica MacBride, and Linnea Micciulla. 2007. Unrestricted coreference: Identifying entities and events in OntoNotes. In Proc. of ICSC. [find](https://scholar.google.com/scholar?q=Sameer+S%2E+Pradhan%2C+Lance+Ramshaw%2C+Ralph+Weischedel%2C+Jessica+MacBride%2C+and+Linnea+Micciulla%2E+2007%2E+Unrestricted+coreference%3A+Identifying+entities+and+events+in+OntoNotes%2E+In+Proc%2E+of+ICSC%2E)
22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. of EMNLP. [↗](https://doi.org/10.48550/arxiv.1606.05250)
23. Marta Recasens, Eduard Hovy, and M Antònia Martı́. 2011. Identity, non-identity, and near-identity: Addressing the complexity of coreference. Lingua, 121(6):1138–1152. [↗](https://doi.org/10.1016/j.lingua.2011.02.004)
24. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Matthew+Richardson%2C+Christopher+J%2E+C%2E+Burges%2C+and+Erin+Renshaw%2E+2013%2E+MCTest%3A+A+challenge+dataset+for+the+open%2Ddomain+machine+comprehension+of+text%2E+In+Proc%2E+of+EMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/missing-2013-richardson-mctest-challenge-dataset-open-domain/)
25. Mark D. Smucker, Charles L. A. Clarke, and Gordon V. Cormack. 2009. Experiments with clueweb09: Relevance feedback and web tracks. In Proc. of TREC. [find](https://scholar.google.com/scholar?q=Mark+D%2E+Smucker%2C+Charles+L%2E+A%2E+Clarke%2C+and+Gordon+V%2E+Cormack%2E+2009%2E+Experiments+with+clueweb09%3A+Relevance+feedback+and+web+tracks%2E+In+Proc%2E+of+TREC%2E)
26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NeurIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Proc%2E+of+NeurIPS%2E)
27. Yannick Versley. 2008. Vagueness and referential ambiguity in a large-scale annotated corpus. Research on Language and Computation, 6(3):333–353. [↗](https://doi.org/10.1007/s11168-008-9059-1)
28. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Proc. of ICCV. [find](https://scholar.google.com/scholar?q=Yukun+Zhu%2C+Ryan+Kiros%2C+Richard+S%2E+Zemel%2C+Ruslan+Salakhutdinov%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+m)