[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/huang-2019-cosmos-qa-machine-reading-comprehension/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

# Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

**Authors:** Lifu Huang, [Ronan Le Bras](https://md.mova.elamur.ai/free/en/docs/papers/authors/ronan-le-bras/), [Chandra Bhagavatula](https://md.mova.elamur.ai/free/en/docs/papers/authors/chandra-bhagavatula/), [Yejin Choi](https://md.mova.elamur.ai/free/en/docs/papers/authors/yejin-choi/) · **Year:** 2019 · **Venue:** EMNLP-IJCNLP 2019 **Link:** [aclanthology.org/D19-1243](https://aclanthology.org/D19-1243/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (paper); data licence to be checked before test use

[**270** citations](https://openalex.org/works?filter=cites:W2970780738) openalex-snapshot · 2026-10-14[**266** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/huang-2019-cosmos-qa-machine-reading-comprehension/) of the 140673 papers whose bibliographies we read[Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)](https://aclanthology.org/volumes/D19-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/D19-1243/)
- [PDF](https://aclanthology.org/D19-1243.pdf)
- [DOI 10.18653/v1/d19-1243](https://doi.org/10.18653/v1/d19-1243)
- [arXiv 1909.00277](https://arxiv.org/abs/1909.00277)
- [OpenAlex](https://openalex.org/W2970780738)

**Authors (4)**

1. [Lifu Huang](https://openalex.org/A5042819803) [ORCID](https://orcid.org/0000-0002-2743-7718) · [Semantic Scholar](https://www.semanticscholar.org/author/2319173963) · h 28 · 2676 citations · 157 works
2. [Ronan Le Bras](https://openalex.org/A5024879161) [ORCID](https://orcid.org/0000-0003-2439-6938) · [Semantic Scholar](https://www.semanticscholar.org/author/39227408) · h 34 · 7375 citations · 188 works
3. [Chandra Bhagavatula](https://openalex.org/A5044250030) [ORCID](https://orcid.org/0000-0001-6264-0378) · [Semantic Scholar](https://www.semanticscholar.org/author/1857797) · h 32 · 6116 citations · 103 works
4. [Yejin Choi](https://openalex.org/A5102992157) [ORCID](https://orcid.org/0000-0003-3032-5378) · [Semantic Scholar](https://www.semanticscholar.org/author/1699545) · h 84 · 33876 citations · 437 works

Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart.

## What it is about

Cosmos QA has 35,588 four-option questions on 21,886 paragraphs from personal-narrative blogs, written by crowd workers to need commonsense «between the lines»: likely causes of events, what happened before or after, plausible facts about people and things, and counterfactuals («What would have happened if…?»). Over 83% of right answers are not stated in the paragraph, and 93.8% of questions need contextual commonsense (motivation, reaction, pre/post-condition, temporal, situational facts, counterfactuals). «None of the above» is used as an option: questions answerable without the paragraph had their right answer replaced by it, and in 70% of the others it replaced the weakest distractor. BERT with multiway attention reaches 68.4% against 94% for humans; word-overlap readers do far worse because the right answer often overlaps less with the text. A partial-input ablation shows that removing the question hurts little, since question types are few and guessable from the options.

## Ideas

- I1. Partial-input controls for every course, after the paper's ablation: run the student with the lesson removed (question + options only) and with the question removed (lesson + options only). Questions still answered right without the lesson measure prior or option bias, not reading; report them per course and subtract them when claiming a gain. A deterministic, cheap gate that can go red.
- I2. Handle «none of the above» options explicitly: when no option has frame support (the diagnosis step «no option supported», 29.4% on the Torah course) and such an option exists, choose it with the reason «no remembered frame supports A–C»; when one option is supported, never choose it. Count first how many course questions carry such an option.
- I3. Use the absurdity matrix (`world/src/domains.rs`) to demote options a person would not choose under common sense (leaving a baby alone at home), the error class that made a third of the paper's model errors. Only as a tie-breaker between options with equal frame support, with the score shown in the trace; measure on the «right option outscored» cases.

## For Mova

[33. Implicature and presupposition](https://md.mova.elamur.ai/free/en/docs/languages/en/33-implicature-and-presupposition/) (inference beyond the text), `36-discourse-relations.md` §8, §11. Code: `world/src/knowledge.rs` (partial-input runs, none-of-the-above), `world/src/domains.rs`, `world/src/report.rs`. Measures: course scores with and without lesson and question, diagnosis step shares, «right option outscored» count.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/d19-1243](https://doi.org/10.18653/v1/d19-1243) · [arXiv 1909.00277](https://arxiv.org/abs/1909.00277)

```
@inproceedings{huang2019cosmos,
  title = {{Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning}},
  author = {Huang, Lifu and Bras, Ronan Le and Bhagavatula, Chandra and Choi, Yejin},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1243},
  eprint = {1909.00277},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1243}
}
```

- [← previous A fine-grained comparison of pragmatic language understanding in humans and language models](https://md.mova.elamur.ai/free/en/docs/papers/hu-2023-fine-grained-comparison-pragmatic-language/)
- [next → Multiword Expressions: A Pain in the Neck for NLP](https://md.mova.elamur.ai/free/en/docs/papers/idioms-2002-sag-multiword-expressions-pain-neck-nlp/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

47 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+Meteor%3A+An+automatic+metric+for+mt+evaluation+with+improved+correlation+with+human+judgments%2E+In+Proceedings+of+the+acl+workshop+on+intrinsic+and+extrinsic+eva) [our card](https://md.mova.elamur.ai/free/en/docs/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
2. Kevin Burton, Akshay Java, Ian Soboroff, et al. 2009. The icwsm 2009 spinn3r dataset. In Proceedings of ICWSM 2009. [find](https://scholar.google.com/scholar?q=Kevin+Burton%2C+Akshay+Java%2C+Ian+Soboroff%2C+et+al%2E+2009%2E+The+icwsm+2009+spinn3r+dataset%2E+In+Proceedings+of+ICWSM+2009%2E)
3. Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the ending: Strong neural baselines for the roc story cloze task. In Proceedings of ACL 2017, pages 616–622. [find](https://scholar.google.com/scholar?q=Zheng+Cai%2C+Lifu+Tu%2C+and+Kevin+Gimpel%2E+2017%2E+Pay+attention+to+the+ending%3A+Strong+neural+baselines+for+the+roc+story+cloze+task%2E+In+Proceedings+of+ACL+2017%2C+pages+616%E2%80%93622%2E)
4. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of ACL 2016, pages 2358–2367. [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Jason+Bolton%2C+and+Christopher+D+Manning%2E+2016%2E+A+thorough+examination+of+the+cnn%2Fdaily+mail+reading+comprehension+task%2E+In+Proceedings+of+ACL+2016%2C+pages+2358%E2%80%932367%2E)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
6. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Proceedings of ACL 2017, pages 1832–1846. [find](https://scholar.google.com/scholar?q=Bhuwan+Dhingra%2C+Hanxiao+Liu%2C+Zhilin+Yang%2C+William+Cohen%2C+and+Ruslan+Salakhutdinov%2E+2017%2E+Gated%2Dattention+readers+for+text+comprehension%2E+In+Proceedings+of+ACL+2017%2C+pages+1832%E2%80%931846%2E)
7. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. [find](https://scholar.google.com/scholar?q=Matthew+Dunn%2C+Levent+Sagun%2C+Mike+Higgins%2C+V+Ugur+Guney%2C+Volkan+Cirik%2C+and+Kyunghyun+Cho%2E+2017%2E+Searchqa%3A+A+new+q%26a+dataset+augmented+with+context+from+a+search+engine%2E+arXiv+preprint+arXiv%3A1704%2E05179%2E)
8. Andrew Gordon and Reid Swanson. 2009. Identifying personal stories in millions of weblog entries. In Third International Conference on Weblogs and Social Media, Data Challenge Workshop. [find](https://scholar.google.com/scholar?q=Andrew+Gordon+and+Reid+Swanson%2E+2009%2E+Identifying+personal+stories+in+millions+of+weblog+entries%2E+In+Third+International+Conference+on+Weblogs+and+Social+Media%2C+Data+Challenge+Workshop%2E)
9. Rudolf Kadlec, Martin Schmid, Ondřej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In Proceedings of ACL 2016, pages 908–918. [find](https://scholar.google.com/scholar?q=Rudolf+Kadlec%2C+Martin+Schmid%2C+Ondr%CC%8Cej+Bajgar%2C+and+Jan+Kleindienst%2E+2016%2E+Text+understanding+with+the+attention+sum+reader+network%2E+In+Proceedings+of+ACL+2016%2C+pages+908%E2%80%93918%2E)
10. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gáabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Proceedings of TACL 2018, pages 317–328. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Koc%CC%8Cisky%CC%80%2C+Jonathan+Schwarz%2C+Phil+Blunsom%2C+Chris+Dyer%2C+Karl+Moritz+Hermann%2C+Ga%CC%81abor+Melis%2C+and+Edward+Grefenstette%2E+2018%2E+The+narrativeqa+reading+comprehension+challenge%2E+Proceedings+of+TACL+2) [our card](https://md.mova.elamur.ai/free/en/docs/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)
11. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of EMNLP 2017, pages 785–794. [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2017%2E+Race%3A+Large%2Dscale+reading+comprehension+dataset+from+examinations%2E+In+Proceedings+of+EMNLP+2017%2C+pages+785%E2%80%93794%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lai-2017-race-large-scale-reading-comprehension/)
12. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E)
13. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+Text+Summarization+Branches+Out%2E)
14. Sewon Min, Minjoon Seo, and Hannaneh Hajishirzi. 2017. Question answering through transfer learning from large fine-grained supervision data. In Proceedings of ACL 2017, pages 510–517. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Minjoon+Seo%2C+and+Hannaneh+Hajishirzi%2E+2017%2E+Question+answering+through+transfer+learning+from+large+fine%2Dgrained+supervision+data%2E+In+Proceedings+of+ACL+2017%2C+pages+510%E2%80%93517%2E)
15. Bhavana Dalvi Mishra, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. arXiv preprint arXiv:1805.06975. [find](https://scholar.google.com/scholar?q=Bhavana+Dalvi+Mishra%2C+Lifu+Huang%2C+Niket+Tandon%2C+Wen%2Dtau+Yih%2C+and+Peter+Clark%2E+2018%2E+Tracking+state+changes+in+procedural+text%3A+a+challenge+dataset+and+models+for+process+paragraph+comprehension%2E+arXiv)
16. Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Michael+Roth%2C+Annie+Louis%2C+Nathanael+Chambers%2C+and+James+Allen%2E+2017%2E)
17. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46–51. [find](https://scholar.google.com/scholar?q=Lsdsem+2017+shared+task%3A+The+story+cloze+test%2E+In+Proceedings+of+the+2nd+Workshop+on+Linking+Models+of+Lexical%2C+Sentential+and+Discourse%2Dlevel+Semantics%2C+pages+46%E2%80%9351%2E)
18. Andrew S Gordon. 2016. Commonsense interpretation of triangle behavior. In Proceedings of AAAI 2016. [find](https://scholar.google.com/scholar?q=Andrew+S+Gordon%2E+2016%2E+Commonsense+interpretation+of+triangle+behavior%2E+In+Proceedings+of+AAAI+2016%2E)
19. Peter Norvig. 1987. A Unified Theory of Inference for Text Understanding. Ph.D. thesis, EECS Department, University of California, Berkeley. [find](https://scholar.google.com/scholar?q=Peter+Norvig%2E+1987%2E+A+Unified+Theory+of+Inference+for+Text+Understanding%2E+Ph%2ED%2E+thesis%2C+EECS+Department%2C+University+of+California%2C+Berkeley%2E)
20. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of NAACL 2018, pages 107–112. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+Bowman%2C+and+Noah+A+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+Proceedings+of+NAACL+2018%2C+pages+107%E2%80%93)
21. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018. Mcscript: A novel dataset for assessing machine comprehension using script knowledge. In Proceedings of LREC 2018. [find](https://scholar.google.com/scholar?q=Simon+Ostermann%2C+Ashutosh+Modi%2C+Michael+Roth%2C+Stefan+Thater%2C+and+Manfred+Pinkal%2E+2018%2E+Mcscript%3A+A+novel+dataset+for+assessing+machine+comprehension+using+script+knowledge%2E+In+Proceedings+of+LREC+2018)
22. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of NIPS 2015, pages 1693–1701. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+Proceedings+of+NIPS+2015%2C+page)
23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL 2002, pages 311–318. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+ACL+2002%2C+pages+311%E2%80%93318%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
24. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301. [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Antoine+Bordes%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+The+goldilocks+principle%3A+Reading+children%E2%80%99s+books+with+explicit+memory+representations%2E+arXiv+preprint+arXiv%3A1511%2E02301%2E)
25. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of ACL 2018, pages 328–339. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+ACL+2018%2C+pages+328%E2%80%93339%2E)
26. Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Time+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+with+unsupervised+learning%2E+Technical+report%2C+Technical+report%2C+OpenAI%2E)
27. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, page 8. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+page+8%2E)
28. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: the winograd schema challenge. In Proceedings of EMNLP 2012, pages 777–789. [find](https://scholar.google.com/scholar?q=Altaf+Rahman+and+Vincent+Ng%2E+2012%2E+Resolving+complex+cases+of+definite+pronouns%3A+the+winograd+schema+challenge%2E+In+Proceedings+of+EMNLP+2012%2C+pages+777%E2%80%93789%2E)
29. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you dont know: Unanswerable questions for squad. In Proceedings of ACL 2018, pages 784–789. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Know+what+you+dont+know%3A+Unanswerable+questions+for+squad%2E+In+Proceedings+of+ACL+2018%2C+pages+784%E2%80%93789%2E)
30. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP 2016, pages 2383–2392. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+EMNLP+2016%2C+pages+2383%E2%80%932392%2E)
31. Hannah Rashkin, Antoine Bosselut, Maarten Sap, Kevin Knight, and Yejin Choi. 2018. Modeling naive psychology of characters in simple commonsense stories. In Proceedings of ACL 2018, pages 2289–2299. [find](https://scholar.google.com/scholar?q=Hannah+Rashkin%2C+Antoine+Bosselut%2C+Maarten+Sap%2C+Kevin+Knight%2C+and+Yejin+Choi%2E+2018%2E+Modeling+naive+psychology+of+characters+in+simple+commonsense+stories%2E+In+Proceedings+of+ACL+2018%2C+pages+2289%E2%80%932299%2E)
32. Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042. [find](https://scholar.google.com/scholar?q=Siva+Reddy%2C+Danqi+Chen%2C+and+Christopher+D+Manning%2E+2018%2E+Coqa%3A+A+conversational+question+answering+challenge%2E+arXiv+preprint+arXiv%3A1808%2E07042%2E)
33. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of EMNLP 2013, pages 193–203. [find](https://scholar.google.com/scholar?q=Matthew+Richardson%2C+Christopher+JC+Burges%2C+and+Erin+Renshaw%2E+2013%2E+Mctest%3A+A+challenge+dataset+for+the+open%2Ddomain+machine+comprehension+of+text%2E+In+Proceedings+of+EMNLP+2013%2C+pages+193%E2%80%93203%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/missing-2013-richardson-mctest-challenge-dataset-open-domain/)
34. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+2011%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+2011+AAAI+Spring+Symposium+Series%2E)
35. Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2018. Atomic: An atlas of machine commonsense for if-then reasoning. arXiv preprint arXiv:1811.00146. [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Ronan+LeBras%2C+Emily+Allaway%2C+Chandra+Bhagavatula%2C+Nicholas+Lourie%2C+Hannah+Rashkin%2C+Brendan+Roof%2C+Noah+A+Smith%2C+and+Yejin+Choi%2E+2018%2E+Atomic%3A+An+atlas+of+machine+commonsense+for+if%2Dthen+re)
36. Liang Wang, Meng Sun, Wei Zhao, Kewei Shen, and Jingming Liu. 2018a. Yuanfudao at semeval-2018 task 11: Three-way attention and relational knowledge for commonsense machine comprehension. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 758–762. [find](https://scholar.google.com/scholar?q=Liang+Wang%2C+Meng+Sun%2C+Wei+Zhao%2C+Kewei+Shen%2C+and+Jingming+Liu%2E+2018a%2E+Yuanfudao+at+semeval%2D2018+task+11%3A+Three%2Dway+attention+and+relational+knowledge+for+commonsense+machine+comprehension%2E+In+Proceedin)
37. Shuohang Wang, Mo Yu, Jing Jiang, and Shiyu Chang. 2018b. A co-matching model for multi-choice reading comprehension. In Proceedings of ACL 2018, pages 746–751. [find](https://scholar.google.com/scholar?q=Shuohang+Wang%2C+Mo+Yu%2C+Jing+Jiang%2C+and+Shiyu+Chang%2E+2018b%2E+A+co%2Dmatching+model+for+multi%2Dchoice+reading+comprehension%2E+In+Proceedings+of+ACL+2018%2C+pages+746%E2%80%93751%2E)
38. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of EMNLP 2018, pages 93–104. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+2018%2E+Swag%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+In+Proceedings+of+EMNLP+2018%2C+pages+93%E2%80%93104%2E)
39. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885. [find](https://scholar.google.com/scholar?q=Sheng+Zhang%2C+Xiaodong+Liu%2C+Jingjing+Liu%2C+Jianfeng+Gao%2C+Kevin+Duh%2C+and+Benjamin+Van+Durme%2E+2018%2E+Record%3A+Bridging+the+gap+between+human+and+machine+commonsense+reading+comprehension%2E+arXiv+preprint+arX)
40. Shuailiang Zhang, Hai Zhao, Yuwei Wu, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. 2019a. Dual co-matching network for multi-choice reading comprehension. arXiv preprint arXiv:1901.09381. — [arXiv:1901.09381](https://arxiv.org/abs/1901.09381) [find](https://scholar.google.com/scholar?q=Shuailiang+Zhang%2C+Hai+Zhao%2C+Yuwei+Wu%2C+Zhuosheng+Zhang%2C+Xi+Zhou%2C+and+Xiang+Zhou%2E+2019a%2E+Dual+co%2Dmatching+network+for+multi%2Dchoice+reading+comprehension%2E+arXiv+preprint+arXiv%3A1901%2E09381%2E+%E2%80%94+%5BarXiv%3A1901%2E0)
41. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019b. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019b%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2E)
42. Haichao Zhu, Furu Wei, Bing Qin, and Ting Liu. 2018. Hierarchical attention flow for multiple-choice reading comprehension. In Proceedings of AAAI 2018. [find](https://scholar.google.com/scholar?q=Haichao+Zhu%2C+Furu+Wei%2C+Bing+Qin%2C+and+Ting+Liu%2E+2018%2E+Hierarchical+attention+flow+for+multiple%2Dchoice+reading+comprehension%2E+In+Proceedings+of+AAAI+2018%2E)
43. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728. [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Hannah+Rashkin%2C+Derek+Chen%2C+Ronan+LeBras%2C+and+Yejin+Choi%2E+2019%2E+Socialiqa%3A+Commonsense+reasoning+about+social+interactions%2E+arXiv+preprint+arXiv%3A1904%2E09728%2E)
44. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603. [find](https://scholar.google.com/scholar?q=Minjoon+Seo%2C+Aniruddha+Kembhavi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2016%2E+Bidirectional+attention+flow+for+machine+comprehension%2E+arXiv+preprint+arXiv%3A1611%2E01603%2E)
45. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937. [find](https://scholar.google.com/scholar?q=Alon+Talmor%2C+Jonathan+Herzig%2C+Nicholas+Lourie%2C+and+Jonathan+Berant%2E+2018%2E+Commonsenseqa%3A+A+question+answering+challenge+targeting+commonsense+knowledge%2E+arXiv+preprint+arXiv%3A1811%2E00937%2E)
46. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200. [find](https://scholar.google.com/scholar?q=Adam+Trischler%2C+Tong+Wang%2C+Xingdi+Yuan%2C+Justin+Harris%2C+Alessandro+Sordoni%2C+Philip+Bachman%2C+and+Kaheer+Suleman%2E+2017%2E+Newsqa%3A+A+machine+comprehension+dataset%2E+In+Proceedings+of+the+2nd+Workshop+on+Repr)
47. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of CVPR 2015, pages 4566–4575. [find](https://scholar.google.com/scholar?q=Ramakrishna+Vedantam%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+2015%2E+Cider%3A+Consensus%2Dbased+image+description+evaluation%2E+In+Proceedings+of+CVPR+2015%2C+pages+4566%E2%80%934575%2E)