[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/ning-2020-torque-reading-comprehension-dataset-temporal/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions

# TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions

**Authors:** [Qiang Ning](https://md.mova.elamur.ai/free/en/docs/papers/authors/qiang-ning/), Hao Wu, Rujun Han, [Nanyun Peng](https://md.mova.elamur.ai/free/en/docs/papers/authors/nanyun-peng/), [Matt Gardner](https://md.mova.elamur.ai/free/en/docs/papers/authors/matt-gardner/), [Dan Roth](https://md.mova.elamur.ai/free/en/docs/papers/authors/dan-roth/) · **Year:** 2020 · **Venue:** EMNLP 2020 **Link:** [aclanthology.org/2020.emnlp-main.88](https://aclanthology.org/2020.emnlp-main.88/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (paper); the passages come from TempEval-3 news — test use only, after the data terms are checked

[**83** citations](https://openalex.org/works?filter=cites:W3099246072) openalex · 2026-10-18[**68** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/ning-2020-torque-reading-comprehension-dataset-temporal/) of the 140673 papers whose bibliographies we read[Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://aclanthology.org/volumes/2020.emnlp-main/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2020.emnlp-main.88/)
- [PDF](https://aclanthology.org/2020.emnlp-main.88.pdf)
- [DOI 10.18653/v1/2020.emnlp-main.88](https://doi.org/10.18653/v1/2020.emnlp-main.88)
- [arXiv 2005.00242](https://arxiv.org/abs/2005.00242)
- [OpenAlex](https://openalex.org/W3099246072)

**Authors (6)**

1. [Qiang Ning](https://openalex.org/A5032597148) [ORCID](https://orcid.org/0000-0001-8321-866X) · h 24 · 2056 citations · 119 works
2. [Hao Wu](https://openalex.org/A5058457898) [ORCID](https://orcid.org/0009-0006-2083-3001) · h 7 · 475 citations · 24 works
3. [Rujun Han](https://openalex.org/A5025334096) h 12 · 555 citations · 32 works
4. [Nanyun Peng](https://openalex.org/A5030248499) [ORCID](https://orcid.org/0000-0002-8509-6595) · h 42 · 8030 citations · 351 works
5. [Matt Gardner](https://openalex.org/A5035088083) [ORCID](https://orcid.org/0000-0001-8458-1727) · h 42 · 21027 citations · 136 works
6. [Dan Roth](https://openalex.org/A5023802054) [ORCID](https://orcid.org/0009-0002-1447-5173) · h 85 · 31311 citations · 731 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

TORQUE asks temporal relations as reading-comprehension questions instead of labelling event pairs with Allen-style relations. Crowd workers marked events (verbs and nouns) in 3.2k news snippets and wrote 21k questions of the form «What happened before/after/while E?», «What had started before E?», «What has begun but not finished?», each answered with the set of all events that fit (possibly none). Natural-language questions handle what fixed labels could not: vague order, and events in different modes — negated, uncertain, hypothetical, repeated, generic — can be asked about directly («What didn't the lion do after a large meal?»). Workers also wrote contrast questions that change only the temporal word (before → after → while), so that an answer copied by position is penalised. RoBERTa-large reaches 51% exact match against 84% for humans; readers trained on SQuAD return the same answer for before and after questions.

## Ideas

- I1. A contrast gate for the student's temporal questions: for each course question «What happened after X?», generate «…before X?» and «…while X?» on the same lesson; the answers must differ in the way the frames' `next` order says (or be empty). An answer that does not change with the temporal word is a sign of word matching. Report the share of consistent triples per course; run twice for determinism.
- I2. Answer *before* and *while* questions, not only *what happened next* (`Ask::Next`): *before X* follows `next` links backwards to the frame that precedes X, *while X* takes frames with the same `when:` or a *while*/*as* clause, and an empty set is a valid answer («nothing happened before the snow started»). Measure on course questions with *before*/*while* and on the frame bench `next` role.
- I3. Keep events in non-real modes out of «what happened» answers: negated frames (`neg`), conditional ones (`cond`) and the unrealised complements proposed in `white-2016-universal-decompositional-semantics-universal-dependencies` I1 do not answer «What did the lion do after the meal?» but do answer «What would the lion do if…?». A minimal-pair test per mode.

## For Mova

[36. Discourse relations and coherence](https://md.mova.elamur.ai/free/en/docs/languages/en/36-discourse-relations/) §8.1 (temporal relations), §11.4 (story time), `13-verbs.md` (tense and aspect), `27-negation.md`. Code: `prag/src/frames.rs` (`next`, `when:`, `neg`, `cond`, `Ask::Next`), `world/src/knowledge.rs`, `world/src/narr.rs`. Measures: contrast-triple consistency, course temporal questions right, frame bench next F1.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2020.emnlp-main.88](https://doi.org/10.18653/v1/2020.emnlp-main.88) · [arXiv 2005.00242](https://arxiv.org/abs/2005.00242)

```
@inproceedings{ning2020torque,
  title = {{TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions}},
  author = {Ning, Qiang and Wu, Hao and Han, Rujun and Peng, Nanyun and Gardner, Matt and Roth, Dan},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.88},
  eprint = {2005.00242},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.88}
}
```

- [← previous DisSim: A Discourse-Aware Syntactic Text Simplification Framework for English and German](https://md.mova.elamur.ai/free/en/docs/papers/niklaus-2019-dissim-discourse-aware-syntactic-text/)
- [next → A Paradigm Change for Formal Syntax: Computational Algorithms in the Grammar of English](https://md.mova.elamur.ai/free/en/docs/papers/ninio-2022-paradigm-change-formal-syntax-computational/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

50 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2005. The ACE 2005 (ACE 05) Evaluation Plan. Technical report. [find](https://scholar.google.com/scholar?q=2005%2E+The+ACE+2005+%28ACE+05%29+Evaluation+Plan%2E+Technical+report%2E)
2. James F Allen. 1984. Towards a general theory of action and time. Artificial Intelligence, 23(2):123– 154. [find](https://scholar.google.com/scholar?q=James+F+Allen%2E+1984%2E+Towards+a+general+theory+of+action+and+time%2E+Artificial+Intelligence%2C+23%282%29%3A123%E2%80%93+154%2E)
3. Steven Bethard, James H Martin, and Sara Klingenstein. 2007. Timelines from text: Identification of syntactic temporal relations. In IEEE International Conference on Semantic Computing (ICSC), pages 11–18. [find](https://scholar.google.com/scholar?q=Steven+Bethard%2C+James+H+Martin%2C+and+Sara+Klingenstein%2E+2007%2E+Timelines+from+text%3A+Identification+of+syntactic+temporal+relations%2E+In+IEEE+International+Conference+on+Semantic+Computing+%28ICSC%29%2C+pages+1)
4. Steven Bethard, Guergana Savova, Wei-Te Chen, Leon Derczynski, James Pustejovsky, and Marc Verhagen. 2016. SemEval-2016 Task 12: Clinical TempEval. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 1052–1062, San Diego, California. Association for Computa… [find](https://scholar.google.com/scholar?q=Steven+Bethard%2C+Guergana+Savova%2C+Wei%2DTe+Chen%2C+Leon+Derczynski%2C+James+Pustejovsky%2C+and+Marc+Verhagen%2E+2016%2E+SemEval%2D2016+Task+12%3A+Clinical+TempEval%2E+In+Proceedings+of+the+10th+International+Workshop+on)
5. Steven Bethard, Guergana Savova, Martha Palmer, and James Pustejovsky. 2017. Semeval-2017 task 12: Clinical tempeval. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 565–572. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Steven+Bethard%2C+Guergana+Savova%2C+Martha+Palmer%2C+and+James+Pustejovsky%2E+2017%2E+Semeval%2D2017+task+12%3A+Clinical+tempeval%2E+In+Proceedings+of+the+11th+International+Workshop+on+Semantic+Evaluation+%28SemEval%2D)
6. Taylor Cassidy, Bill McDowell, Nathanel Chambers, and Steven Bethard. 2014. An annotation framework for dense event ordering. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 501–506. [find](https://scholar.google.com/scholar?q=Taylor+Cassidy%2C+Bill+McDowell%2C+Nathanel+Chambers%2C+and+Steven+Bethard%2E+2014%2E+An+annotation+framework+for+dense+event+ordering%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+)
7. Nathanael Chambers, Taylor Cassidy, Bill McDowell, and Steven Bethard. 2014. Dense event ordering with a multi-pass architecture. Transactions of the Association for Computational Linguistics (TACL), 2:273–284. [find](https://scholar.google.com/scholar?q=Nathanael+Chambers%2C+Taylor+Cassidy%2C+Bill+McDowell%2C+and+Steven+Bethard%2E+2014%2E+Dense+event+ordering+with+a+multi%2Dpass+architecture%2E+Transactions+of+the+Association+for+Computational+Linguistics+%28TACL%29%2C+)
8. Fei Cheng and Yusuke Miyao. 2017. Classifying temporal relations by bidirectional LSTM over dependency paths. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), volume 2, pages 1–6. [find](https://scholar.google.com/scholar?q=Fei+Cheng+and+Yusuke+Miyao%2E+2017%2E+Classifying+temporal+relations+by+bidirectional+LSTM+over+dependency+paths%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL)
9. Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5925–5932. [find](https://scholar.google.com/scholar?q=Pradeep+Dasigi%2C+Nelson+F%2E+Liu%2C+Ana+Marasovic%CC%81%2C+Noah+A%2E+Smith%2C+and+Matt+Gardner%2E+2019%2E+Quoref%3A+A+reading+comprehension+dataset+with+questions+requiring+coreferential+reasoning%2E+In+Proceedings+of+the+Co) [our card](https://md.mova.elamur.ai/free/en/docs/papers/dasigi-2019-quoref-reading-comprehension-dataset-questions/)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+Conference+of+the+North+)
11. Dmitriy Dligach, Timothy Miller, Chen Lin, Steven Bethard, and Guergana Savova. 2017. Neural temporal relation extraction. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL), volume 2, pages 746–751. [find](https://scholar.google.com/scholar?q=Dmitriy+Dligach%2C+Timothy+Miller%2C+Chen+Lin%2C+Steven+Bethard%2C+and+Guergana+Savova%2E+2017%2E+Neural+temporal+relation+extraction%2E+In+Proceedings+of+the+Conference+of+the+European+Chapter+of+the+Association+f)
12. Quang Do, Wei Lu, and Dan Roth. 2012. Joint inference for event timeline construction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Quang+Do%2C+Wei+Lu%2C+and+Dan+Roth%2E+2012%2E+Joint+inference+for+event+timeline+construction%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
13. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NA… [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Yizhong+Wang%2C+Pradeep+Dasigi%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2019%2E+DROP%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+In+Proceedings+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
14. Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noa… [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Yoav+Artzi%2C+Victoria+Basmova%2C+Jonathan+Berant%2C+Ben+Bogin%2C+Sihao+Chen%2C+Pradeep+Dasigi%2C+Dheeru+Dua%2C+Yanai+Elazar%2C+Ananth+Gottumukkala%2C+Nitish+Gupta%2C+Hanna+Hajishirzi%2C+Gabriel+Ilharco%2C+Dani) [our card](https://md.mova.elamur.ai/free/en/docs/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
15. Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min. 2019a. On making reading comprehension more comprehensive. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering. [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Jonathan+Berant%2C+Hannaneh+Hajishirzi%2C+Alon+Talmor%2C+and+Sewon+Min%2E+2019a%2E+On+making+reading+comprehension+more+comprehensive%2E+In+Proceedings+of+the+2nd+Workshop+on+Machine+Reading+for+Que)
16. Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min. 2019b. Question answering is a format; when is it useful? arXiv preprint arXiv:1909.11291. — [arXiv:1909.11291](https://arxiv.org/abs/1909.11291) [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Jonathan+Berant%2C+Hannaneh+Hajishirzi%2C+Alon+Talmor%2C+and+Sewon+Min%2E+2019b%2E+Question+answering+is+a+format%3B+when+is+it+useful%3F+arXiv+preprint+arXiv%3A1909%2E11291%2E+%E2%80%94+%5BarXiv%3A1909%2E11291%5D%28https%3A%2F%2F)
17. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Yoav+Goldberg%2C+and+Jonathan+Berant%2E+2019%2E+Are+we+modeling+the+task+or+the+annotator%3F+an+investigation+of+annotator+bias+in+natural+language+understanding+datasets%2E+In+Proceedings+of+the+Conf)
18. Hangfeng He, Qiang Ning, and Dan Roth. 2020. QuASE: Question-answer driven sentence encoding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 8743–8758. [find](https://scholar.google.com/scholar?q=Hangfeng+He%2C+Qiang+Ning%2C+and+Dan+Roth%2E+2020%2E+QuASE%3A+Question%2Danswer+driven+sentence+encoding%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+pages+8743%E2%80%9387)
19. Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 643–653. [find](https://scholar.google.com/scholar?q=Luheng+He%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2015%2E+Question%2Danswer+driven+semantic+role+labeling%3A+Using+natural+language+to+annotate+natural+language%2E+In+Proceedings+of+the+Conference+on+Empirical+Met) [our card](https://md.mova.elamur.ai/free/en/docs/papers/he-2015-question-answer-driven-semantic-role/)
20. Egoitz Laparra, Dongfang Xu, Ahmed Elsayed, Steven Bethard, and Martha Palmer. 2018. [find](https://scholar.google.com/scholar?q=Egoitz+Laparra%2C+Dongfang+Xu%2C+Ahmed+Elsayed%2C+Steven+Bethard%2C+and+Martha+Palmer%2E+2018%2E)
21. SemEval 2018 task 6: Parsing time normalizations. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 88–96. [find](https://scholar.google.com/scholar?q=SemEval+2018+task+6%3A+Parsing+time+normalizations%2E+In+Proceedings+of+The+12th+International+Workshop+on+Semantic+Evaluation%2C+pages+88%E2%80%9396%2E)
22. Artuur Leeuwenberg and Marie-Francine Moens. 2017. Structured learning for temporal relation extraction from clinical records. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Artuur+Leeuwenberg+and+Marie%2DFrancine+Moens%2E+2017%2E+Structured+learning+for+temporal+relation+extraction+from+clinical+records%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Asso)
23. Artuur Leeuwenberg and Marie-Francine Moens. 2018. Temporal information extraction by predicting relative time-lines. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Artuur+Leeuwenberg+and+Marie%2DFrancine+Moens%2E+2018%2E+Temporal+information+extraction+by+predicting+relative+time%2Dlines%2E+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+)
24. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the SIGNLL Conference on Computational Natural Language Learning (CoNLL), pages 333–342. [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Minjoon+Seo%2C+Eunsol+Choi%2C+and+Luke+Zettlemoyer%2E+2017%2E+Zero%2Dshot+relation+extraction+via+reading+comprehension%2E+In+Proceedings+of+the+SIGNLL+Conference+on+Computational+Natural+Language+Lear)
25. Chen Lin, Timothy Miller, Dmitriy Dligach, Steven Bethard, and Guergana Savova. 2017. Representations of time expressions for temporal relation extraction with convolutional neural networks. [find](https://scholar.google.com/scholar?q=Chen+Lin%2C+Timothy+Miller%2C+Dmitriy+Dligach%2C+Steven+Bethard%2C+and+Guergana+Savova%2E+2017%2E+Representations+of+time+expressions+for+temporal+relation+extraction+with+convolutional+neural+networks%2E)
26. Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019. Reasoning over paragraph effects in situations. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 58–62. [find](https://scholar.google.com/scholar?q=Kevin+Lin%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Matt+Gardner%2E+2019%2E+Reasoning+over+paragraph+effects+in+situations%2E+In+Proceedings+of+the+2nd+Workshop+on+Machine+Reading+for+Question+Answering%2C+pages+58%E2%80%936)
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
28. Hector Llorens, Nathanael Chambers, Naushad UzZaman, Nasrin Mostafazadeh, James Allen, and James Pustejovsky. 2015. SemEval-2015 Task 5: QA TEMPEVAL - evaluating temporal information understanding with question answering. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEv… [find](https://scholar.google.com/scholar?q=Hector+Llorens%2C+Nathanael+Chambers%2C+Naushad+UzZaman%2C+Nasrin+Mostafazadeh%2C+James+Allen%2C+and+James+Pustejovsky%2E+2015%2E+SemEval%2D2015+Task+5%3A+QA+TEMPEVAL+%2D+evaluating+temporal+information+understanding+wit)
29. Yuanliang Meng and Anna Rumshisky. 2018. Context-aware neural model for temporal information extraction. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), volume 1, pages 527–536. [find](https://scholar.google.com/scholar?q=Yuanliang+Meng+and+Anna+Rumshisky%2E+2018%2E+Context%2Daware+neural+model+for+temporal+information+extraction%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+vo)
30. Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2017. Crowdsourcing question-answer meaning representations. arXiv preprint arXiv:1711.05885. [find](https://scholar.google.com/scholar?q=Julian+Michael%2C+Gabriel+Stanovsky%2C+Luheng+He%2C+Ido+Dagan%2C+and+Luke+Zettlemoyer%2E+2017%2E+Crowdsourcing+question%2Danswer+meaning+representations%2E+arXiv+preprint+arXiv%3A1711%2E05885%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/michael-2018-crowdsourcing-question-answer-meaning-representations/)
31. Anne-Lyse Minard, Manuela Speranza, Eneko Agirre, Itziar Aldabe, Marieke van Erp, Bernardo Magnini, German Rigau, Ruben Urizar, and Fondazione Bruno Kessler. 2015. SemEval-2015 Task 4: TimeLine: Cross-document event ordering. In Proceedings of the 9th International Workshop on Semantic Evaluation (S… [find](https://scholar.google.com/scholar?q=Anne%2DLyse+Minard%2C+Manuela+Speranza%2C+Eneko+Agirre%2C+Itziar+Aldabe%2C+Marieke+van+Erp%2C+Bernardo+Magnini%2C+German+Rigau%2C+Ruben+Urizar%2C+and+Fondazione+Bruno+Kessler%2E+2015%2E+SemEval%2D2015+Task+4%3A+TimeLine%3A+Cross)
32. T. Mitamura, Y. Yamakawa, S. Holm, Z. Song, A. Bies, S. Kulick, and S. Strassel. 2015. Event nugget annotation: Processes and issues. In Proceedings of the Workshop on Events at NAACL-HLT. [find](https://scholar.google.com/scholar?q=T%2E+Mitamura%2C+Y%2E+Yamakawa%2C+S%2E+Holm%2C+Z%2E+Song%2C+A%2E+Bies%2C+S%2E+Kulick%2C+and+S%2E+Strassel%2E+2015%2E+Event+nugget+annotation%3A+Processes+and+issues%2E+In+Proceedings+of+the+Workshop+on+Events+at+NAACL%2DHLT%2E)
33. Jessica Semega, Melissa Kollar, John Creamer, and Abinash Mohanty. 2019. Income and Poverty in the United States: 2018. U.S. Department of Commerce. [find](https://scholar.google.com/scholar?q=Jessica+Semega%2C+Melissa+Kollar%2C+John+Creamer%2C+and+Abinash+Mohanty%2E+2019%2E+Income+and+Poverty+in+the+United+States%3A+2018%2E+U%2ES%2E+Department+of+Commerce%2E)
34. Nasrin Mostafazadeh, Alyson Grealish, Nathanael Chambers, James Allen, and Lucy Vanderwende. 2016. CaTeRS: Causal and temporal relation scheme for semantic annotation of event structures. In Proceedings of the 4th Workshop on Events: Definition, Detection, Coreference, and Representation, pages 51–6… [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Alyson+Grealish%2C+Nathanael+Chambers%2C+James+Allen%2C+and+Lucy+Vanderwende%2E+2016%2E+CaTeRS%3A+Causal+and+temporal+relation+scheme+for+semantic+annotation+of+event+structures%2E+In+Proceedin)
35. William F Styler IV, Steven Bethard, Sean Finan, Martha Palmer, Sameer Pradhan, Piet C de Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, et al. 2014. Temporal annotation in the clinical domain. Transactions of the Association for Computational Linguistics (TACL), 2:143. [find](https://scholar.google.com/scholar?q=William+F+Styler+IV%2C+Steven+Bethard%2C+Sean+Finan%2C+Martha+Palmer%2C+Sameer+Pradhan%2C+Piet+C+de+Groen%2C+Brad+Erickson%2C+Timothy+Miller%2C+Chen+Lin%2C+Guergana+Savova%2C+et+al%2E+2014%2E+Temporal+annotation+in+the+clini)
36. Qiang Ning, Zhili Feng, and Dan Roth. 2017. A structured learning approach to temporal relation extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1038–1048, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qiang+Ning%2C+Zhili+Feng%2C+and+Dan+Roth%2E+2017%2E+A+structured+learning+approach+to+temporal+relation+extraction%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29)
37. Qiang Ning, Zhili Feng, Hao Wu, and Dan Roth. 2018a. Joint reasoning for temporal and causal relations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 2278–2288. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qiang+Ning%2C+Zhili+Feng%2C+Hao+Wu%2C+and+Dan+Roth%2E+2018a%2E+Joint+reasoning+for+temporal+and+causal+relations%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+pag)
38. Qiang Ning, Sanjay Subramanian, and Dan Roth. 2019. An Improved Neural Baseline for Temporal Relation Extraction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Qiang+Ning%2C+Sanjay+Subramanian%2C+and+Dan+Roth%2E+2019%2E+An+Improved+Neural+Baseline+for+Temporal+Relation+Extraction%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28)
39. Julien Tourille, Olivier Ferret, Aurelie Neveol, and Xavier Tannier. 2017. Neural architecture for temporal relation extraction: A Bi-LSTM approach for detecting narrative containers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), volume 2, pages 224–230… [find](https://scholar.google.com/scholar?q=Julien+Tourille%2C+Olivier+Ferret%2C+Aurelie+Neveol%2C+and+Xavier+Tannier%2E+2017%2E+Neural+architecture+for+temporal+relation+extraction%3A+A+Bi%2DLSTM+approach+for+detecting+narrative+containers%2E+In+Proceedings+o)
40. Naushad UzZaman, Hector Llorens, James Allen, Leon Derczynski, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 Task 1: TEMPEVAL-3: Evaluating time expressions, events, and temporal relations. Proceedings of the Joint Conference on Lexical and Computational Semantics (*SEM), 2:1–9. [find](https://scholar.google.com/scholar?q=Naushad+UzZaman%2C+Hector+Llorens%2C+James+Allen%2C+Leon+Derczynski%2C+Marc+Verhagen%2C+and+James+Pustejovsky%2E+2013%2E+SemEval%2D2013+Task+1%3A+TEMPEVAL%2D3%3A+Evaluating+time+expressions%2C+events%2C+and+temporal+relations%2E)
41. Alakananda Vempala, Eduardo Blanco, and Alexis Palmer. 2018. Determining event durations: Models and error analysis. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), volume 2, pages 164–168. [find](https://scholar.google.com/scholar?q=Alakananda+Vempala%2C+Eduardo+Blanco%2C+and+Alexis+Palmer%2E+2018%2E+Determining+event+durations%3A+Models+and+error+analysis%2E+In+Proceedings+of+the+Conference+of+the+North+American+Chapter+of+the+Association+f)
42. Qiang Ning, Hao Wu, and Dan Roth. 2018b. A multi-axis annotation scheme for event temporal relations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1318–1328. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qiang+Ning%2C+Hao+Wu%2C+and+Dan+Roth%2E+2018b%2E+A+multi%2Daxis+annotation+scheme+for+event+temporal+relations%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+pages)
43. Marc Verhagen, Robert Gaizauskas, Frank Schilder, Mark Hepple, Graham Katz, and James Pustejovsky. 2007. SemEval-2007 Task 15: TempEval temporal relation identification. In Proceedings of the 4th International Workshop on Semantic Evaluations, pages 75–80. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marc+Verhagen%2C+Robert+Gaizauskas%2C+Frank+Schilder%2C+Mark+Hepple%2C+Graham+Katz%2C+and+James+Pustejovsky%2E+2007%2E+SemEval%2D2007+Task+15%3A+TempEval+temporal+relation+identification%2E+In+Proceedings+of+the+4th+Inte)
44. Tim O’Gorman, Kristin Wright-Bettner, and Martha Palmer. 2016. Richer Event Description: Integrating event coreference with temporal, causal and bridging annotation. In Proceedings of the 2nd Workshop on Computing News Storylines (CNS 2016), pages 47–56, Austin, Texas. Association for Computational… [find](https://scholar.google.com/scholar?q=Tim+O%E2%80%99Gorman%2C+Kristin+Wright%2DBettner%2C+and+Martha+Palmer%2E+2016%2E+Richer+Event+Description%3A+Integrating+event+coreference+with+temporal%2C+causal+and+bridging+annotation%2E+In+Proceedings+of+the+2nd+Workshop)
45. Marc Verhagen, Roser Sauri, Tommaso Caselli, and James Pustejovsky. 2010. SemEval-2010 Task 13: TempEval-2. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 57– 62. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marc+Verhagen%2C+Roser+Sauri%2C+Tommaso+Caselli%2C+and+James+Pustejovsky%2E+2010%2E+SemEval%2D2010+Task+13%3A+TempEval%2D2%2E+In+Proceedings+of+the+5th+International+Workshop+on+Semantic+Evaluation%2C+pages+57%E2%80%93+62%2E+Assoc)
46. James Pustejovsky, Patrick Hanks, Roser Sauri, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, et al. 2003. The TIMEBANK corpus. In Corpus Linguistics, page 40. [find](https://scholar.google.com/scholar?q=James+Pustejovsky%2C+Patrick+Hanks%2C+Roser+Sauri%2C+Andrew+See%2C+Robert+Gaizauskas%2C+Andrea+Setzer%2C+Dragomir+Radev%2C+Beth+Sundheim%2C+David+Day%2C+Lisa+Ferro%2C+et+al%2E+2003%2E+The+TIMEBANK+corpus%2E+In+Corpus+Linguisti)
47. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 784–789. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+SQuAD%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28A)
48. Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. “Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Ben+Zhou%2C+Daniel+Khashabi%2C+Qiang+Ning%2C+and+Dan+Roth%2E+2019%2E+%E2%80%9CGoing+on+a+vacation%E2%80%9D+takes+longer+than+%E2%80%9CGoing+for+a+walk%E2%80%9D%3A+A+Study+of+Temporal+Commonsense+Understanding%2E+In+Proceedings+of+the+Conference+o)
49. Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal Common Sense Acquisition with Minimal Supervision. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Ben+Zhou%2C+Qiang+Ning%2C+Daniel+Khashabi%2C+and+Dan+Roth%2E+2020%2E+Temporal+Common+Sense+Acquisition+with+Minimal+Supervision%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguis)
50. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2383–2392. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+L)