[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/lal-2021-tellmewhy-dataset-answering-why-questions/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / TellMeWhy: A Dataset for Answering Why-Questions in Narratives

# TellMeWhy: A Dataset for Answering Why-Questions in Narratives

**Authors:** [Yash Kumar Lal](https://md.mova.elamur.ai/free/en/docs/papers/authors/yash-kumar-lal/), [Nathanael Chambers](https://md.mova.elamur.ai/free/en/docs/papers/authors/nathanael-chambers/), [Raymond Mooney](https://md.mova.elamur.ai/free/en/docs/papers/authors/raymond-mooney/), Niranjan Balasubramanian · **Year:** 2021 · **Venue:** Findings of ACL-IJCNLP 2021 **Link:** [aclanthology.org/2021.findings-acl.53](https://aclanthology.org/2021.findings-acl.53/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (paper); the stories are ROCStories — test use only, after its terms are checked

[**24** citations](https://openalex.org/works?filter=cites:W3171639130) openalex · 2026-10-18[**30** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/lal-2021-tellmewhy-dataset-answering-why-questions/) of the 140673 papers whose bibliographies we read[Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021](https://aclanthology.org/volumes/2021.findings-acl/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2021.findings-acl.53/)
- [PDF](https://aclanthology.org/2021.findings-acl.53.pdf)
- [DOI 10.18653/v1/2021.findings-acl.53](https://doi.org/10.18653/v1/2021.findings-acl.53)
- [arXiv 2106.06132](https://arxiv.org/abs/2106.06132)
- [OpenAlex](https://openalex.org/W3171639130)

**Authors (4)**

1. [Yash Kumar Lal](https://openalex.org/A5064437096) h 8 · 254 citations · 43 works
2. [Nathanael Chambers](https://openalex.org/A5105307360) h 28 · 4779 citations · 92 works
3. [Raymond Mooney](https://openalex.org/A5008715111) [ORCID](https://orcid.org/0000-0002-4504-0490) · h 80 · 31134 citations · 320 works
4. [Niranjan Balasubramanian](https://openalex.org/A5101768349) [ORCID](https://orcid.org/0000-0003-4187-9368) · h 24 · 3211 citations · 185 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

TellMeWhy has 30,519 why-questions about the actions of characters in 9,636 five-sentence ROCStories, generated by templates from dependency parses («Why did {subject} {verb} {object}?») and answered by three crowd workers each, who start from a prompt «{subject} {verb} {object} because…», mark the sentences that helped and say whether the answer is in the story. In 28.8% of questions at least two of three workers found no explicit answer; 43% of answers cite no supporting sentence; answers share only 57% of their words with the story; and for the action in the first sentence, 55% of questions need outside knowledge. Human answers are judged 99% grammatical and 95% valid. Fine-tuned T5 and UnifiedQA beat untuned models but stay far below humans, and do much worse on implicit-answer questions (purpose of buying several brands = to taste-test them, told only later).

## Ideas

- I1. Infer the purpose of an action from what the same agent does later with the same object, a deterministic «plan» rule over frames: if the agent of frame A later uses A's patient or destination in frame B (buys water → tastes the water; goes to the store → buys at the store), link B as A's purpose (`cause`, marked as inferred, with the shared entity printed). The story's own later sentence is then the answer to «Why did Rudy buy several brands?». Measure `Ask::Cause` course questions right and the frame bench cause recall; inferred links must not override explicit *because*.
- I2. Use stated wants and feelings as causes of later actions: a prior `want`/`need`/`love`/`be hungry` frame of the same agent whose object or goal matches a later action explains it («She loved seeing the animals… She went to look at the polar bears»). Rank explicit *because* > purpose *to* > stated want > later use (I1), and print the rank in the answer's reason.
- I3. Generate why-questions from Mova's own tale frames with the paper's templates and let the teacher write reference answers once (silver for measuring, never used at test time), labelled explicit/implicit as the paper does. Report the student's accuracy separately on explicit and implicit questions, so gains from I1–I2 are seen where they should be.

## For Mova

[36. Discourse relations and coherence](https://md.mova.elamur.ai/free/en/docs/languages/en/36-discourse-relations/) §8 (time and narrative), §11 (links between frames, story time), `33-implicature-and-presupposition.md` (inferences beyond the text). Code: `prag/src/frames.rs` (cause links, inferred purpose), `world/src/narr.rs`, `world/src/knowledge.rs` (cause questions). Measures: course why-questions right, frame bench cause recall/precision, explicit vs implicit accuracy on the generated set.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2021.findings-acl.53](https://doi.org/10.18653/v1/2021.findings-acl.53) · [arXiv 2106.06132](https://arxiv.org/abs/2106.06132)

```
@inproceedings{lal2021tellmewhy,
  title = {{TellMeWhy: A Dataset for Answering Why-Questions in Narratives}},
  author = {Lal, Yash Kumar and Chambers, Nathanael and Mooney, Raymond and Balasubramanian, Niranjan},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.53},
  eprint = {2106.06132},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.53}
}
```

- [← previous RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://md.mova.elamur.ai/free/en/docs/papers/lai-2017-race-large-scale-reading-comprehension/)
- [next → Whit's the Richt Pairt o Speech: PoS tagging for Scots](https://md.mova.elamur.ai/free/en/docs/papers/lameris-2021-whit-s-richt-pairt-o/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

58 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Sant… [find](https://scholar.google.com/scholar?q=Lo%C3%AFc+Barrault%2C+Magdalena+Biesialska%2C+Ondr%CC%8Cej+Bojar%2C+Marta+R%2E+Costa%2Djuss%C3%A0%2C+Christian+Federmann%2C+Yvette+Graham%2C+Roman+Grundkiewicz%2C+Barry+Haddow%2C+Matthias+Huck%2C+Eric+Joanis%2C+Tom+Kocmi%2C+Philipp+Koehn%2C+Ch)
2. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (… [find](https://scholar.google.com/scholar?q=Lo%C3%AFc+Barrault%2C+Ondr%CC%8Cej+Bojar%2C+Marta+R%2E+Costa%2Djuss%C3%A0%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Philipp+Koehn%2C+Shervin+Malmasi%2C+Christof+Monz%2C+Mathias+M%C3%BCller%2C+Santanu)
3. Mohaddeseh Bastan, Mahnaz Koupaee, Youngseo Son, Richard Sicoli, and Niranjan Balasubramanian. 2020. Author’s sentiment prediction. In Proceedings of the 28th International Conference on Computational Linguistics, pages 604–615, Barcelona, Spain (Online). International Committee on Computational Lin… [find](https://scholar.google.com/scholar?q=Mohaddeseh+Bastan%2C+Mahnaz+Koupaee%2C+Youngseo+Son%2C+Richard+Sicoli%2C+and+Niranjan+Balasubramanian%2E+2020%2E+Author%E2%80%99s+sentiment+prediction%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational)
4. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Chandra+Bhagavatula%2C+Ronan+Le+Bras%2C+Chaitanya+Malaviya%2C+Keisuke+Sakaguchi%2C+Ari+Holtzman%2C+Hannah+Rashkin%2C+Doug+Downey%2C+Wen+tau+Yih%2C+and+Yejin+Choi%2E+2020%2E+Abductive+commonsense+reasoning%2E+In+Internation) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bhagavatula-2019-abductive-commonsense-reasoning/)
5. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine transla… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Yvette+Graham%2C+Barry+Haddow%2C+Shujian+Huang%2C+Matthias+Huck%2C+Philipp+Koehn%2C+Qun+Liu%2C+Varvara+Logacheva%2C+Christof+Monz%2C+Matteo+Negri%2C+Matt+Post%2C+Raph)
6. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi,… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Antonio+Jimeno+Yepes%2C+Philipp+Koehn%2C+Varvara+Logacheva%2C+Christof+Monz%2C+Matteo+Negri%2C+Aur%C3%A9lie+N%C3%A9v%C3%A9ol%2C+M)
7. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272–303, Belgium, Brussels. A… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Philipp+Koehn%2C+and+Christof+Monz%2E+2018%2E+Findings+of+the+2018+conference+on+machine+translation+%28WMT18%29%2E+In+Proceedings+of+)
8. Ozan Caglayan, Pranava Madhyastha, and Lucia Specia. 2020. Curious case of language generation evaluation metrics: A cautionary tale. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2322–2328, Barcelona, Spain (Online). International Committee on Computational… [find](https://scholar.google.com/scholar?q=Ozan+Caglayan%2C+Pranava+Madhyastha%2C+and+Lucia+Specia%2E+2020%2E+Curious+case+of+language+generation+evaluation+metrics%3A+A+cautionary+tale%2E+In+Proceedings+of+the+28th+International+Conference+on+Computation)
9. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799. [find](https://scholar.google.com/scholar?q=Asli+Celikyilmaz%2C+Elizabeth+Clark%2C+and+Jianfeng+Gao%2E+2020%2E+Evaluation+of+text+generation%3A+A+survey%2E+arXiv+preprint+arXiv%3A2006%2E14799%2E)
10. Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 602–610, Suntec, Si… [find](https://scholar.google.com/scholar?q=Nathanael+Chambers+and+Dan+Jurafsky%2E+2009%2E+Unsupervised+learning+of+narrative+schemas+and+their+participants%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+ACL+and+the+4th+I)
11. Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Evaluating question answering evaluation. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 119–124, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anthony+Chen%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2019%2E+Evaluating+question+answering+evaluation%2E+In+Proceedings+of+the+2nd+Workshop+on+Machine+Reading+for+Question+Answering%2C+pages+119)
12. Andrei Dulceanu, Thang Le Dinh, Walter Chang, Trung Bui, Doo Soon Kim, Manh Chien Vu, and Seokhwan Kim. 2018. PhotoshopQuiA: A corpus of non-factoid questions and answers for why-question answering. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 20… [find](https://scholar.google.com/scholar?q=Andrei+Dulceanu%2C+Thang+Le+Dinh%2C+Walter+Chang%2C+Trung+Bui%2C+Doo+Soon+Kim%2C+Manh+Chien+Vu%2C+and+Seokhwan+Kim%2E+2018%2E+PhotoshopQuiA%3A+A+corpus+of+non%2Dfactoid+questions+and+answers+for+why%2Dquestion+answering%2E+I)
13. Jesse Dunietz, Greg Burnham, Akash Bharadwaj, Owen Rambow, Jennifer Chu-Carroll, and Dave Ferrucci. 2020. To test machine comprehension, start by defining comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7839–7859, Online. Association f… [find](https://scholar.google.com/scholar?q=Jesse+Dunietz%2C+Greg+Burnham%2C+Akash+Bharadwaj%2C+Owen+Rambow%2C+Jennifer+Chu%2DCarroll%2C+and+Dave+Ferrucci%2E+2020%2E+To+test+machine+comprehension%2C+start+by+defining+comprehension%2E+In+Proceedings+of+the+58th+Ann)
14. J. Fukumoto, T. Kato, F. Masui, and T. Mori. 2007. An overview of the 4th question answering challenge (qac-4) at ntcir workshop 6. In NTCIR. [find](https://scholar.google.com/scholar?q=J%2E+Fukumoto%2C+T%2E+Kato%2C+F%2E+Masui%2C+and+T%2E+Mori%2E+2007%2E+An+overview+of+the+4th+question+answering+challenge+%28qac%2D4%29+at+ntcir+workshop+6%2E+In+NTCIR%2E)
15. Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. J. Artif. Int. Res., 61(1):65–170. [find](https://scholar.google.com/scholar?q=Albert+Gatt+and+Emiel+Krahmer%2E+2018%2E+Survey+of+the+state+of+the+art+in+natural+language+generation%3A+Core+tasks%2C+applications+and+evaluation%2E+J%2E+Artif%2E+Int%2E+Res%2E%2C+61%281%29%3A65%E2%80%93170%2E)
16. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics, 9:346–361. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+2021%2E+Did+Aristotle+Use+a+Laptop%3F+A+Question+Answering+Benchmark+with+Implicit+Reasoning+Strategies%2E+Transactions+of+)
17. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational… [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2013%2E+Continuous+measurement+scales+in+human+evaluation+of+machine+translation%2E+In+Proceedings+of+the+7th+Linguistic+Annotation+Works)
18. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014. Is machine translation getting better over time? In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 443–451, Gothenburg, Sweden. Association for Computational L… [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2014%2E+Is+machine+translation+getting+better+over+time%3F+In+Proceedings+of+the+14th+Conference+of+the+European+Chapter+of+the+Associati)
19. Ryuichiro Higashinaka and Hideki Isozaki. 2008. Corpus-based question answering for why-questions. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I. [find](https://scholar.google.com/scholar?q=Ryuichiro+Higashinaka+and+Hideki+Isozaki%2E+2008%2E+Corpus%2Dbased+question+answering+for+why%2Dquestions%2E+In+Proceedings+of+the+Third+International+Joint+Conference+on+Natural+Language+Processing%3A+Volume%2DI%2E)
20. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal%2C+Ines+Montani%2C+Sofie+Van+Lan%2Ddeghem%2C+and+Adriane+Boyd%2E+2020%2E+spaCy%3A+Industrial%2Dstrength+Natural+Language+Processing+in+Python%2E)
21. David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings… [find](https://scholar.google.com/scholar?q=David+M%2E+Howcroft%2C+Anya+Belz%2C+Miruna%2DAdriana+Clinciu%2C+Dimitra+Gkatzia%2C+Sadid+A%2E+Hasan%2C+Saad+Mahamood%2C+Simon+Mille%2C+Emiel+van+Miltenburg%2C+Sashank+Santhanam%2C+and+Verena+Rieser%2E+2020%2E+Twenty+years+of+con)
22. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907, Online. Association for Com… [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Sewon+Min%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Hannaneh+Hajishirzi%2E+2020%2E+UNIFIEDQA%3A+Crossing+format+boundaries+with+a+single+QA+system%2E+In+Findings+of+the)
23. Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, and Daniel S. Weld. 2021. Genie: A leaderboard for human-in-the-loop evaluation of text generation. [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Gabriel+Stanovsky%2C+Jonathan+Bragg%2C+Nicholas+Lourie%2C+Jungo+Kasai%2C+Yejin+Choi%2C+Noah+A%2E+Smith%2C+and+Daniel+S%2E+Weld%2E+2021%2E+Genie%3A+A+leaderboard+for+human%2Din%2Dthe%2Dloop+evaluation+of+text+gen)
24. W. Kintsch and T. Dijk. 1978. Toward a model of text comprehension and production. Psychological Review, 85:363–394. [find](https://scholar.google.com/scholar?q=W%2E+Kintsch+and+T%2E+Dijk%2E+1978%2E+Toward+a+model+of+text+comprehension+and+production%2E+Psychological+Review%2C+85%3A363%E2%80%93394%2E)
25. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317– 328. [find](https://scholar.google.com/scholar?q=Tom%C3%A1%C5%A1+Koc%CC%8Cisk%C3%BD%2C+Jonathan+Schwarz%2C+Phil+Blunsom%2C+Chris+Dyer%2C+Karl+Moritz+Hermann%2C+G%C3%A1bor+Melis%2C+and+Edward+Grefenstette%2E+2018%2E+The+NarrativeQA+reading+comprehension+challenge%2E+Transactions+of+the+Associ) [our card](https://md.mova.elamur.ai/free/en/docs/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)
26. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2017%2E+RACE%3A+Large%2Dscale+ReAding+comprehension+dataset+from+examinations%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lai-2017-race-large-scale-reading-comprehension/)
27. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E+Association+for+Computational+Linguistics%2E)
28. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. Asso… [find](https://scholar.google.com/scholar?q=Qingsong+Ma%2C+Johnny+Wei%2C+Ondr%CC%8Cej+Bojar%2C+and+Yvette+Graham%2E+2019%2E+Results+of+the+WMT19+metrics+shared+task%3A+Segment%2Dlevel+and+strong+MT+systems+pose+big+challenges%2E+In+Proceedings+of+the+Fourth+Confere)
29. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849, San Diego, California. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technologies%2C+pages+839%E2%80%93849%2C+San+Diego%2C+California%2E+Association+for+Computational+Linguistics%2E)
30. Nasrin Mostafazadeh, Alyson Grealish, Nathanael Chambers, James Allen, and Lucy Vanderwende. 2016b. CaTeRS: Causal and temporal relation scheme for semantic annotation of event structures. In Proceedings of the Fourth Workshop on Events, pages 51–61, San Diego, California. Association for Computatio… [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Alyson+Grealish%2C+Nathanael+Chambers%2C+James+Allen%2C+and+Lucy+Vanderwende%2E+2016b%2E+CaTeRS%3A+Causal+and+temporal+relation+scheme+for+semantic+annotation+of+event+structures%2E+In+Proceedi)
31. Nasrin Mostafazadeh, Aditya Kalyanpur, Lori Moon, David Buchanan, Lauren Berkowitz, Or Biran, and Jennifer Chu-Carroll. 2020. GLUCOSE: GeneraLized and COntextualized story explanations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4569–4586… [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Aditya+Kalyanpur%2C+Lori+Moon%2C+David+Buchanan%2C+Lauren+Berkowitz%2C+Or+Biran%2C+and+Jennifer+Chu%2DCarroll%2E+2020%2E+GLUCOSE%3A+GeneraLized+and+COntextualized+story+explanations%2E+In+Proceedings) [our card](https://md.mova.elamur.ai/free/en/docs/papers/commonsense-2020-mostafazadeh-glucose-generalized-contextualized-story-explanations/)
32. Joanna Mrozinski, Edward Whittaker, and Sadaoki Furui. 2008. Collecting a why-question corpus for development and evaluation of an automatic QA-system. In Proceedings of ACL-08: HLT, pages 443–451, Columbus, Ohio. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Joanna+Mrozinski%2C+Edward+Whittaker%2C+and+Sadaoki+Furui%2E+2008%2E+Collecting+a+why%2Dquestion+corpus+for+development+and+evaluation+of+an+automatic+QA%2Dsystem%2E+In+Proceedings+of+ACL%2D08%3A+HLT%2C+pages+443%E2%80%93451%2C+Co)
33. Allen Nie, Erin Bennett, and Noah Goodman. 2019. Learning to explain: Answering why-questions via rephrasing. In Proceedings of the First Workshop on NLP for Conversational AI, pages 113–120, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Allen+Nie%2C+Erin+Bennett%2C+and+Noah+Goodman%2E+2019%2E+Learning+to+explain%3A+Answering+why%2Dquestions+via+rephrasing%2E+In+Proceedings+of+the+First+Workshop+on+NLP+for+Conversational+AI%2C+pages+113%E2%80%93120%2C+Florence)
34. Peter Norvig. 1987. Inference in text understanding. In Proceedings of the Sixth National Conference on Artificial Intelligence - Volume 2, AAAI’87, page 561–565. AAAI Press. [find](https://scholar.google.com/scholar?q=Peter+Norvig%2E+1987%2E+Inference+in+text+understanding%2E+In+Proceedings+of+the+Sixth+National+Conference+on+Artificial+Intelligence+%2D+Volume+2%2C+AAAI%E2%80%9987%2C+page+561%E2%80%93565%2E+AAAI+Press%2E)
35. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2018. RankME: Reliable human ratings for natural language generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pag… [find](https://scholar.google.com/scholar?q=Jekaterina+Novikova%2C+Ondr%CC%8Cej+Du%C5%A1ek%2C+and+Verena+Rieser%2E+2018%2E+RankME%3A+Reliable+human+ratings+for+natural+language+generation%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+)
36. D. Marasini, P. Quatto, and E. Ripamonti. 2016. Assessing the inter-rater agreement for ordinal data through weighted indexes. Statistical Methods in Medical Research, 25:2611 – 2633. [find](https://scholar.google.com/scholar?q=D%2E+Marasini%2C+P%2E+Quatto%2C+and+E%2E+Ripamonti%2E+2016%2E+Assessing+the+inter%2Drater+agreement+for+ordinal+data+through+weighted+indexes%2E+Statistical+Methods+in+Medical+Research%2C+25%3A2611+%E2%80%93+2633%2E)
37. Jong-Hoon Oh, Kazuma Kadowaki, Julien Kloetzer, Ryu Iida, and Kentaro Torisawa. 2019. Open-domain why-question answering with adversarial learning to encode answer texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4227–4237, Florence, Italy. Ass… [find](https://scholar.google.com/scholar?q=Jong%2DHoon+Oh%2C+Kazuma+Kadowaki%2C+Julien+Kloetzer%2C+Ryu+Iida%2C+and+Kentaro+Torisawa%2E+2019%2E+Open%2Ddomain+why%2Dquestion+answering+with+adversarial+learning+to+encode+answer+texts%2E+In+Proceedings+of+the+57th+An)
38. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016a. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+Allen%2E+2016a%2E+A+corpus+and+cloze+evaluation+for+deeper+understanding+of+comm)
39. Jong-Hoon Oh, Kentaro Torisawa, Chikara Hashimoto, Takuya Kawada, Stijn De Saeger, Jun’ichi Kazama, and Yiou Wang. 2012. Why question answering using sentiment analysis and word classes. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational… [find](https://scholar.google.com/scholar?q=Jong%2DHoon+Oh%2C+Kentaro+Torisawa%2C+Chikara+Hashimoto%2C+Takuya+Kawada%2C+Stijn+De+Saeger%2C+Jun%E2%80%99ichi+Kazama%2C+and+Yiou+Wang%2E+2012%2E+Why+question+answering+using+sentiment+analysis+and+word+classes%2E+In+Proceeding)
40. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Comp) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
41. Karl Pichotta and Raymond Mooney. 2014. Statistical script learning with multi-argument events. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 220–229, Gothenburg, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Karl+Pichotta+and+Raymond+Mooney%2E+2014%2E+Statistical+script+learning+with+multi%2Dargument+events%2E+In+Proceedings+of+the+14th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Lingu)
42. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matt+Post%2E+2018%2E+A+call+for+clarity+in+reporting+BLEU+scores%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%3A+Research+Papers%2C+pages+186%E2%80%93+191%2C+Brussels%2C+Belgium%2E+Association+for+Computa) [our card](https://md.mova.elamur.ai/free/en/docs/papers/post-2018-call-clarity-reporting-bleu-scores/)
43. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
44. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
45. Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. 2020. Getting closer to ai complete question answering: A set of prerequisite real tasks. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8722–8731. [find](https://scholar.google.com/scholar?q=Anna+Rogers%2C+Olga+Kovaleva%2C+Matthew+Downey%2C+and+Anna+Rumshisky%2E+2020%2E+Getting+closer+to+ai+complete+question+answering%3A+A+set+of+prerequisite+real+tasks%2E+Proceedings+of+the+AAAI+Conference+on+Artifici)
46. Keisuke Sakaguchi and Benjamin Van Durme. 2018. Efficient online scalar annotation with bounded support. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 208–218, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi+and+Benjamin+Van+Durme%2E+2018%2E+Efficient+online+scalar+annotation+with+bounded+support%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vol)
47. Sashank Santhanam and Samira Shaikh. 2019. Towards best experiment design for evaluating dialogue system output. In Proceedings of the 12th International Conference on Natural Language Generation, pages 88–94, Tokyo, Japan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sashank+Santhanam+and+Samira+Shaikh%2E+2019%2E+Towards+best+experiment+design+for+evaluating+dialogue+system+output%2E+In+Proceedings+of+the+12th+International+Conference+on+Natural+Language+Generation%2C+pag)
48. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):3027–3035. [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Ronan+Le+Bras%2C+Emily+Allaway%2C+Chandra+Bhagavatula%2C+Nicholas+Lourie%2C+Hannah+Rashkin%2C+Brendan+Roof%2C+Noah+A%2E+Smith%2C+and+Yejin+Choi%2E+2019%2E+Atomic%3A+An+atlas+of+machine+commonsense+for+if%2Dthen+)
49. R. Schank and R. P. Abelson. 1977. Scripts, plans, goals and understanding: an inquiry into human knowledge structures. [find](https://scholar.google.com/scholar?q=R%2E+Schank+and+R%2E+P%2E+Abelson%2E+1977%2E+Scripts%2C+plans%2C+goals+and+understanding%3A+an+inquiry+into+human+knowledge+structures%2E)
50. Roger C. Schank and Robert P. Abelson. 1975. Scripts, plans, and knowledge. In Proceedings of the 4th International Joint Conference on Artificial Intelligence - Volume 1, IJCAI’75, page 151–157, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. [find](https://scholar.google.com/scholar?q=Roger+C%2E+Schank+and+Robert+P%2E+Abelson%2E+1975%2E+Scripts%2C+plans%2C+and+knowledge%2E+In+Proceedings+of+the+4th+International+Joint+Conference+on+Artificial+Intelligence+%2D+Volume+1%2C+IJCAI%E2%80%9975%2C+page+151%E2%80%93157%2C+San+)
51. João Sedoc, Daphne Ippolito, Arun Kirubarajan, Jai Thirani, Lyle Ungar, and Chris Callison-Burch. 2019. ChatEval: A tool for chatbot evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 60–65, Minneap… [find](https://scholar.google.com/scholar?q=Jo%C3%A3o+Sedoc%2C+Daphne+Ippolito%2C+Arun+Kirubarajan%2C+Jai+Thirani%2C+Lyle+Ungar%2C+and+Chris+Callison%2DBurch%2E+2019%2E+ChatEval%3A+A+tool+for+chatbot+evaluation%2E+In+Proceedings+of+the+2019+Conference+of+the+North+Amer)
52. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+Parikh%2E+2020%2E+BLEURT%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+)
53. Suzan Verberne, Lou Boves, Nelleke Oostdijk, and Peter-Arno Coppen. 2006. Data for question answering: The case of why. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Suzan+Verberne%2C+Lou+Boves%2C+Nelleke+Oostdijk%2C+and+Peter%2DArno+Coppen%2E+2006%2E+Data+for+question+answering%3A+The+case+of+why%2E+In+Proceedings+of+the+Fifth+International+Conference+on+Language+Resources+and+E)
54. Suzan Verberne, Lou Boves, Nelleke Oostdijk, and Peter-Arno Coppen. 2007. Evaluating discourse-based answer extraction for why-question answering. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’07, page 735–736, New Y… [find](https://scholar.google.com/scholar?q=Suzan+Verberne%2C+Lou+Boves%2C+Nelleke+Oostdijk%2C+and+Peter%2DArno+Coppen%2E+2007%2E+Evaluating+discourse%2Dbased+answer+extraction+for+why%2Dquestion+answering%2E+In+Proceedings+of+the+30th+Annual+International+ACM+S)
55. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+Remi+Louf%2C+Morgan+Funtow%2Dicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara)
56. Rowan Zellers, Ari Holtzman, Elizabeth Clark, Lianhui Qin, Ali Farhadi, and Yejin Choi. 2020. Evaluating machines by their real-world language use. arXiv preprint arXiv:2004.03607. — [arXiv:2004.03607](https://arxiv.org/abs/2004.03607) [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Elizabeth+Clark%2C+Lianhui+Qin%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2020%2E+Evaluating+machines+by+their+real%2Dworld+language+use%2E+arXiv+preprint+arXiv%3A2004%2E03607%2E+%E2%80%94+%5BarXiv%3A2004%2E03607)
57. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2A%2C+Varsha+Kishore%2A%2C+Felix+Wu%2A%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
58. Rolf A. Zwaan, Mark C. Langston, and Arthur C. Graesser. 1995. The construction of situation models in narrative comprehension: An event-indexing model. Psychological Science, 6(5):292–297. [find](https://scholar.google.com/scholar?q=Rolf+A%2E+Zwaan%2C+Mark+C%2E+Langston%2C+and+Arthur+C%2E+Graesser%2E+1995%2E+The+construction+of+situation+models+in+narrative+comprehension%3A+An+event%2Dindexing+model%2E+Psychological+Science%2C+6%285%29%3A292%E2%80%93297%2E)