[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/mirzaee-2021-spartqa-textual-question-answering-benchmark/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning

# SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning

**Authors:** Roshanak Mirzaee, Hossein Rajaby Faghihi, [Qiang Ning](https://md.mova.elamur.ai/free/en/docs/papers/authors/qiang-ning/), Parisa Kordjamshidi · **Year:** 2021 · **Venue:** Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) **Link:** [aclanthology.org/2021.naacl-main.364](https://aclanthology.org/2021.naacl-main.364/) (ACL Anthology 2021.naacl-main.364; DOI [10.18653/v1/2021.naacl-main.364](https://doi.org/10.18653/v1/2021.naacl-main.364)) **License of the paper:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology) — [creativecommons.org/licenses/by/4.0](https://creativecommons.org/licenses/by/4.0/)

[**41** citations](https://openalex.org/works?filter=cites:W3153839026) openalex · 2026-10-18[**52** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/mirzaee-2021-spartqa-textual-question-answering-benchmark/) of the 140673 papers whose bibliographies we read[Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://aclanthology.org/volumes/2021.naacl-main/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2021.naacl-main.364/)
- [PDF](https://aclanthology.org/2021.naacl-main.364.pdf)
- [DOI 10.18653/v1/2021.naacl-main.364](https://doi.org/10.18653/v1/2021.naacl-main.364)
- [arXiv 2104.05832](https://arxiv.org/abs/2104.05832)
- [OpenAlex](https://openalex.org/W3153839026)

**Authors (4)**

1. [Roshanak Mirzaee](https://openalex.org/A5055413119) h 5 · 102 citations · 11 works
2. [Hossein Rajaby Faghihi](https://openalex.org/A5037637824) [ORCID](https://orcid.org/0000-0003-4283-8816) · h 5 · 99 citations · 14 works
3. [Qiang Ning](https://openalex.org/A5032597148) [ORCID](https://orcid.org/0000-0001-8321-866X) · h 24 · 2056 citations · 119 works
4. [Parisa Kordjamshidi](https://openalex.org/A5087682739) [ORCID](https://orcid.org/0000-0002-4606-1824) · h 19 · 1284 citations · 168 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## Summary

The paper introduces a question-answering benchmark for spatial reasoning over text that is richer than the earlier synthetic bAbI spatial task. Scene descriptions are based on NLVR images: a small human-written part (SpartQA-Human, about 1.1k questions) and a large automatically generated part (SpartQA-Auto) built with context-free grammars and spatial inference rules. Descriptions are deliberately incomplete, so answers require transitivity, symmetry, inverse relations, inclusion and exclusion. There are four question types: which relation holds, which block an object is in, which of two objects fits, and yes/no with a "don't know" option under an open-world assumption. Because language models do poorly when trained only on the small human set, the authors propose distant supervision: further pretraining on the auto-generated data. This improves results on SpartQA-Human and also transfers to bAbI and BoolQ.

## How Mova uses it

- [Scene: shapes, colours, sizes, proximity](https://md.mova.elamur.ai/free/en/docs/global/seeds/shortcuts/scene/) — a first-level knowledge seed for scene descriptions: categories of colours, sizes and shapes, generic nouns ("thing", "object", "one"), synonyms of above/below, and the relations near, far and touching declared symmetric (but not transitive).
- SpartQA-Human is used as an evaluation set for the `world spartqa` reader in the `world` crate: mentions are built from the seed categories, scoped to a block, relations are composed, and the output is three-valued (follows / opposite / unknown).
- The dataset is under a non-commercial license, so it is used only for measurement, not shipped or trained on beyond rule fixes guided by training-split errors.

## Effectiveness in Mova

Measured on SpartQA-Human (test): 59.3% overall (majority baseline 34.9%; first version 50.5%), by type YN 60.1%, FB 67.1%, CO 58.2%, FR 50.6%; training split 61.3%. Fixes were made only from training-split errors. Mova's test split is the 2022 version (YN 143, FR 77 questions), so the overall figure is not comparable with the 2021 BERT transfer result on the 510-question test. Comparable figures: YN 60.1% with DK in the gold labels, or 68.5% two-class (DK mapped to No), versus BERT+Q-Chain 59.44, Llama-3-8B CoT 67.83, PistaQ 75.52 and GPT-4 77.62; FR 50.6% (exact set match) versus BERT with SpaRTUN 50.64. This is achieved with no neural network and no training on this data. The ceiling is limited: questions were written from images, so some answers are not derivable from (or contradict) the text.

---

👨‍🔬💥

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2021.naacl-main.364](https://doi.org/10.18653/v1/2021.naacl-main.364) · [arXiv 2104.05832](https://arxiv.org/abs/2104.05832)

```
@inproceedings{mirzaee2021spartqa,
  title = {{SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning}},
  author = {Mirzaee, Roshanak and Faghihi, Hossein Rajaby and Ning, Qiang and Kordjamshidi, Parisa},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.364},
  eprint = {2104.05832},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.364}
}
```

- [← previous Assessing the Reliability of Statistical Software: Part I](https://md.mova.elamur.ai/free/en/docs/papers/mccullough-1998-assessing-reliability-statistical-software/)
- [next → Learning To Use Formulas To Solve Simple Arithmetic Problems](https://md.mova.elamur.ai/free/en/docs/papers/mitra-2016-learning-use-formulas-solve-simple/)
- [↑ back to papers](https://md.mova.elamur.ai/free/en/docs/papers/)

## References

56 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. GLUE (Wang et al., 2018). We observe that many of the boolQ examples answered correctly by the BERT further pretrained on S PART QA-A UTO require multi-step reasoning. Our hypothesis is that since solving S PART QA-A UTO questions needs multi-step reasoning, fine-tuning BERT on S PART QA-A UTO gener… [find](https://scholar.google.com/scholar?q=GLUE+%28Wang+et+al%2E%2C+2018%29%2E+We+observe+that+many+of+the+boolQ+examples+answered+correctly+by+the+BERT+further+pretrained+on+S+PART+QA%2DA+UTO+require+multi%2Dstep+reasoning%2E+Our+hypothesis+is+that+since+sol)
2. Model Majority baseline Recurrent model (ReM) ReM fine-tuned on SQuAD ReM fine-tuned on QNLI ReM fine-tuned on NQ BERT (our setup) BERT (S PART QA-A UTO ) Accuracy 62.2 62.2 69.8 71.4 72.8 71.9 74.2 *Table 6: System performances on the dev set of boolQ (since the test set is not available to us). To… [find](https://scholar.google.com/scholar?q=Model+Majority+baseline+Recurrent+model+%28ReM%29+ReM+fine%2Dtuned+on+SQuAD+ReM+fine%2Dtuned+on+QNLI+ReM+fine%2Dtuned+on+NQ+BERT+%28our+setup%29+BERT+%28S+PART+QA%2DA+UTO+%29+Accuracy+62%2E2+62%2E2+69%2E8+71%2E4+72%2E8+71%2E9+74%2E2+%2A)
3. Kaustubh Dhole and Christopher D. Manning. 2020. Syn-QG: Syntactic and shallow semantic rules for question generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 752–765. [find](https://scholar.google.com/scholar?q=Kaustubh+Dhole+and+Christopher+D%2E+Manning%2E+2020%2E+Syn%2DQG%3A+Syntactic+and+shallow+semantic+rules+for+question+generation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Li)
4. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+2015%2E+VQA%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE+international+confe)
5. Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Xinya+Du+and+Claire+Cardie%2E+2020%2E+Event+extraction+by+answering+%28almost%29+natural+questions%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
6. Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. TOUCHDOWN: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12538–12547. [find](https://scholar.google.com/scholar?q=Howard+Chen%2C+Alane+Suhr%2C+Dipendra+Misra%2C+Noah+Snavely%2C+and+Yoav+Artzi%2E+2019%2E+TOUCHDOWN%3A+Natural+language+navigation+and+spatial+reasoning+in+visual+street+environments%2E+In+Proceedings+of+the+IEEE+Conf)
7. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:… [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+2019%2E+BoolQ%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+In+Proceedings+of)
8. Douglas H Clements and Michael T Battista. 1992. Geometry and spatial reasoning. Handbook of research on mathematics teaching and learning, pages 420– 464. [find](https://scholar.google.com/scholar?q=Douglas+H+Clements+and+Michael+T+Battista%2E+1992%2E+Geometry+and+spatial+reasoning%2E+Handbook+of+research+on+mathematics+teaching+and+learning%2C+pages+420%E2%80%93+464%2E)
9. Soham Dan, Parisa Kordjamshidi, Julia Bonn, Archna Bhatia, Zheng Cai, Martha Palmer, and Dan Roth. 2020. From spatial relations to spatial configurations. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5855– 5864, Marseille, France. European Language Resources Associa… [find](https://scholar.google.com/scholar?q=Soham+Dan%2C+Parisa+Kordjamshidi%2C+Julia+Bonn%2C+Archna+Bhatia%2C+Zheng+Cai%2C+Martha+Palmer%2C+and+Dan+Roth%2E+2020%2E+From+spatial+relations+to+spatial+configurations%2E+In+Proceedings+of+the+12th+Language+Resources)
10. Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Co… [find](https://scholar.google.com/scholar?q=Pradeep+Dasigi%2C+Nelson+F%2E+Liu%2C+Ana+Marasovic%CC%81%2C+Noah+A%2E+Smith%2C+and+Matt+Gardner%2E+2019%2E+Quoref%3A+A+reading+comprehension+dataset+with+questions+requiring+coreferential+reasoning%2E+In+Proceedings+of+the+20) [our card](https://md.mova.elamur.ai/free/en/docs/papers/dasigi-2019-quoref-reading-comprehension-dataset-questions/)
11. Surabhi Datta and Kirk Roberts. 2020. A hybrid deep learning approach for spatial trigger extraction from radiology reports. In Proceedings of the Third International Workshop on Spatial Language Understanding, pages 50–55, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Surabhi+Datta+and+Kirk+Roberts%2E+2020%2E+A+hybrid+deep+learning+approach+for+spatial+trigger+extraction+from+radiology+reports%2E+In+Proceedings+of+the+Third+International+Workshop+on+Spatial+Language+Unde)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
13. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to Ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1342– 1352. [find](https://scholar.google.com/scholar?q=Xinya+Du%2C+Junru+Shao%2C+and+Claire+Cardie%2E+2017%2E+Learning+to+Ask%3A+Neural+question+generation+for+reading+comprehension%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Lin)
14. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Yizhong+Wang%2C+Pradeep+Dasigi%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2019%2E+DROP%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+In+Proceedings+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
15. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, N… [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Yoav+Artzi%2C+Victoria+Basmov%2C+Jonathan+Berant%2C+Ben+Bogin%2C+Sihao+Chen%2C+Pradeep+Dasigi%2C+Dheeru+Dua%2C+Yanai+Elazar%2C+Ananth+Gottumukkala%2C+Nitish+Gupta%2C+Hannaneh+Hajishirzi%2C+Gabriel+Ilharco%2C+Da) [our card](https://md.mova.elamur.ai/free/en/docs/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
16. Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min. 2019. Question Answering is a Format; when is it useful? ArXiv, abs/1909.11291. — [arXiv:1909.11291](https://arxiv.org/abs/1909.11291) [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Jonathan+Berant%2C+Hannaneh+Hajishirzi%2C+Alon+Talmor%2C+and+Sewon+Min%2E+2019%2E+Question+Answering+is+a+Format%3B+when+is+it+useful%3F+ArXiv%2C+abs%2F1909%2E11291%2E+%E2%80%94+%5BarXiv%3A1909%2E11291%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
17. Mehdi Ghanimifard and Simon Dobnik. 2017. Learning to compose spatial relations with grounded neural language models. In IWCS 2017-12th International Conference on Computational Semantics-Long papers. [find](https://scholar.google.com/scholar?q=Mehdi+Ghanimifard+and+Simon+Dobnik%2E+2017%2E+Learning+to+compose+spatial+relations+with+grounded+neural+language+models%2E+In+IWCS+2017%2D12th+International+Conference+on+Computational+Semantics%2DLong+papers%2E)
18. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t Stop Pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+2020%2E+Don%E2%80%99t+Stop+Pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+In+Proceedings+of+)
19. Hangfeng He, Qiang Ning, and Dan Roth. 2020a. QuASE: Question-answer driven sentence encoding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8743–8758, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hangfeng+He%2C+Qiang+Ning%2C+and+Dan+Roth%2E+2020a%2E+QuASE%3A+Question%2Danswer+driven+sentence+encoding%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+8743%E2%80%9387)
20. Hangfeng He, Mingyuan Zhang, Qiang Ning, and Dan Roth. 2020b. Foreshadowing the benefits of incidental supervision. arXiv preprint arXiv:2006.05500. — [arXiv:2006.05500](https://arxiv.org/abs/2006.05500) [find](https://scholar.google.com/scholar?q=Hangfeng+He%2C+Mingyuan+Zhang%2C+Qiang+Ning%2C+and+Dan+Roth%2E+2020b%2E+Foreshadowing+the+benefits+of+incidental+supervision%2E+arXiv+preprint+arXiv%3A2006%2E05500%2E+%E2%80%94+%5BarXiv%3A2006%2E05500%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006%2E055)
21. Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-Answer Driven Semantic Role Labeling: Using natural language to annotate natural language. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 643–653. [find](https://scholar.google.com/scholar?q=Luheng+He%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2015%2E+Question%2DAnswer+Driven+Semantic+Role+Labeling%3A+Using+natural+language+to+annotate+natural+language%2E+In+Proceedings+of+the+2015+Conference+on+Empirica) [our card](https://md.mova.elamur.ai/free/en/docs/papers/he-2015-question-answer-driven-semantic-role/)
22. Michael Heilman and Noah A Smith. 2009. Question generation via overgenerating transformations and ranking. Technical report, CARNEGIE-MELLON UNIV PITTSBURGH PA LANGUAGE TECH-NOLOGIES INST. [find](https://scholar.google.com/scholar?q=Michael+Heilman+and+Noah+A+Smith%2E+2009%2E+Question+generation+via+overgenerating+transformations+and+ranking%2E+Technical+report%2C+CARNEGIE%2DMELLON+UNIV+PITTSBURGH+PA+LANGUAGE+TECH%2DNOLOGIES+INST%2E)
23. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
24. Zixian Huang, Yulin Shen, Xiao Li, Yu’ang Wei, Gong Cheng, Lin Zhou, Xinyu Dai, and Yuzhong Qu. 2019. GeoSQA: A benchmark for scenario-based question answering in the geography domain at high school level. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and… [find](https://scholar.google.com/scholar?q=Zixian+Huang%2C+Yulin+Shen%2C+Xiao+Li%2C+Yu%E2%80%99ang+Wei%2C+Gong+Cheng%2C+Lin+Zhou%2C+Xinyu+Dai%2C+and+Yuzhong+Qu%2E+2019%2E+GeoSQA%3A+A+benchmark+for+scenario%2Dbased+question+answering+in+the+geography+domain+at+high+school+l)
25. Drew A Hudson and Christopher D Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6700–6709. [find](https://scholar.google.com/scholar?q=Drew+A+Hudson+and+Christopher+D+Manning%2E+2019%2E+GQA%3A+A+new+dataset+for+real%2Dworld+visual+reasoning+and+compositional+question+answering%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pat)
26. Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018. Tem-pQuestions: A benchmark for temporal question answering. In Companion Proceedings of the The Web Conference 2018, pages 1057–1062. [find](https://scholar.google.com/scholar?q=Zhen+Jia%2C+Abdalghani+Abujabal%2C+Rishiraj+Saha+Roy%2C+Jannik+Str%C3%B6tgen%2C+and+Gerhard+Weikum%2E+2018%2E+Tem%2DpQuestions%3A+A+benchmark+for+temporal+question+answering%2E+In+Companion+Proceedings+of+the+The+Web+Confer)
27. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901–291… [find](https://scholar.google.com/scholar?q=Justin+Johnson%2C+Bharath+Hariharan%2C+Laurens+van+der+Maaten%2C+Li+Fei%2DFei%2C+C+Lawrence+Zitnick%2C+and+Ross+Girshick%2E+2017%2E+CLEVR%3A+A+diagnostic+dataset+for+compositional+language+and+elementary+visual+reasoni)
28. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking Beyond the Surface:a challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Snigdha+Chaturvedi%2C+Michael+Roth%2C+Shyam+Upadhyay%2C+and+Dan+Roth%2E+2018%2E+Looking+Beyond+the+Surface%3Aa+challenge+set+for+reading+comprehension+over+multiple+sentences%2E+In+Proceedings+of+N) [our card](https://md.mova.elamur.ai/free/en/docs/papers/khashabi-2018-looking-beyond-surface-challenge-set/)
29. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UnifiedQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907. [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Sewon+Min%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Hannaneh+Hajishirzi%2E+2020%2E+UnifiedQA%3A+Crossing+format+boundaries+with+a+single+QA+system%2E+In+Findings+of+the)
30. Hyounghun Kim, Abhaysinh Zala, Graham Burri, Hao Tan, and Mohit Bansal. 2020. ArraMon: A joint navigation-assembly instruction interpretation task in dynamic environments. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3910–3927, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Hyounghun+Kim%2C+Abhaysinh+Zala%2C+Graham+Burri%2C+Hao+Tan%2C+and+Mohit+Bansal%2E+2020%2E+ArraMon%3A+A+joint+navigation%2Dassembly+instruction+interpretation+task+in+dynamic+environments%2E+In+Findings+of+the+Associati)
31. Parisa Kordjamshidi, Marie-Francine Moens, and Martijn van Otterlo. 2010. Spatial Role Labeling: Task definition and annotation scheme. In Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10), pages 413–420. European Language Resources Association (ELRA)… [find](https://scholar.google.com/scholar?q=Parisa+Kordjamshidi%2C+Marie%2DFrancine+Moens%2C+and+Martijn+van+Otterlo%2E+2010%2E+Spatial+Role+Labeling%3A+Task+definition+and+annotation+scheme%2E+In+Proceedings+of+the+Seventh+conference+on+International+Langua)
32. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural Questions: a benchmark for question answering research. Transactions of the Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Red%2Dfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Jacob+Devlin%2C+Kenton+Lee%2C+et+al%2E+2019%2E+Natural+Questions%3A+a+be)
33. Igor Labutov, Sumit Basu, and Lucy Vanderwende. 2015. Deep questions without deep understanding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 889–898… [find](https://scholar.google.com/scholar?q=Igor+Labutov%2C+Sumit+Basu%2C+and+Lucy+Vanderwende%2E+2015%2E+Deep+questions+without+deep+understanding%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+7th+)
34. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+ALBERT%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+International+Confe)
35. Christian Landsiedel, Verena Rieser, Matthew Walter, and Dirk Wollherr. 2017. A review of spatial reasoning and interaction for real-world robotics. Advanced Robotics, 31(5):222–242. [find](https://scholar.google.com/scholar?q=Christian+Landsiedel%2C+Verena+Rieser%2C+Matthew+Walter%2C+and+Dirk+Wollherr%2E+2017%2E+A+review+of+spatial+reasoning+and+interaction+for+real%2Dworld+robotics%2E+Advanced+Robotics%2C+31%285%29%3A222%E2%80%93242%2E)
36. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-Shot relation extraction via reading comprehension. In CONLL, pages 333–342. [find](https://scholar.google.com/scholar?q=Omer+Levy%2C+Minjoon+Seo%2C+Eunsol+Choi%2C+and+Luke+Zettlemoyer%2E+2017%2E+Zero%2DShot+relation+extraction+via+reading+comprehension%2E+In+CONLL%2C+pages+333%E2%80%93342%2E)
37. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980– 2988. [find](https://scholar.google.com/scholar?q=Tsung%2DYi+Lin%2C+Priya+Goyal%2C+Ross+Girshick%2C+Kaiming+He%2C+and+Piotr+Doll%C3%A1r%2E+2017%2E+Focal+loss+for+dense+object+detection%2E+In+Proceedings+of+the+IEEE+international+conference+on+computer+vision%2C+pages+2980%E2%80%93)
38. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2E)
39. Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2018. Crowdsourcing question-answer meaning representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Sho… [find](https://scholar.google.com/scholar?q=Julian+Michael%2C+Gabriel+Stanovsky%2C+Luheng+He%2C+Ido+Dagan%2C+and+Luke+Zettlemoyer%2E+2018%2E+Crowdsourcing+question%2Danswer+meaning+representations%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/michael-2018-crowdsourcing-question-answer-meaning-representations/)
40. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 10… [find](https://scholar.google.com/scholar?q=Mike+Mintz%2C+Steven+Bills%2C+Rion+Snow%2C+and+Dan+Jurafsky%2E+2009%2E+Distant+supervision+for+relation+extraction+without+labeled+data%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+)
41. Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1158–1172. [find](https://scholar.google.com/scholar?q=Qiang+Ning%2C+Hao+Wu%2C+Rujun+Han%2C+Nanyun+Peng%2C+Matt+Gardner%2C+and+Dan+Roth%2E+2020%2E+TORQUE%3A+A+reading+comprehension+dataset+of+temporal+ordering+questions%2E+In+Proceedings+of+the+2020+Conference+on+Empirical) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ning-2020-torque-reading-comprehension-dataset-temporal/)
42. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Thibault+F%C3%A9vry%2C+and+Samuel+R+Bowman%2E+2018%2E+Sentence+encoders+on+stilts%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+arXiv+preprint+arXiv%3A1811%2E01088%2E)
43. Taher Rahgooy, Umar Manzoor, and Parisa Kordjamshidi. 2018. Visually guided spatial relation extraction from text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 788–794… [find](https://scholar.google.com/scholar?q=Taher+Rahgooy%2C+Umar+Manzoor%2C+and+Parisa+Kordjamshidi%2E+2018%2E+Visually+guided+spatial+relation+extraction+from+text%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Associatio)
44. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu)
45. Homero Roman Roman, Yonatan Bisk, Jesse Thomason, Asli Celikyilmaz, and Jianfeng Gao. 2020. RMM: A recursive mental model for dialogue navigation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1732–1745, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Homero+Roman+Roman%2C+Yonatan+Bisk%2C+Jesse+Thomason%2C+Asli+Celikyilmaz%2C+and+Jianfeng+Gao%2E+2020%2E+RMM%3A+A+recursive+mental+model+for+dialogue+navigation%2E+In+Findings+of+the+Association+for+Computational+Ling)
46. Priyanka Sen and Amir Saffari. 2020. What do models learn from question answering datasets? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2429–2438. [find](https://scholar.google.com/scholar?q=Priyanka+Sen+and+Amir+Saffari%2E+2020%2E+What+do+models+learn+from+question+answering+datasets%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+242)
47. Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 217–223, Vancouver, Canada. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Alane+Suhr%2C+Mike+Lewis%2C+James+Yeh%2C+and+Yoav+Artzi%2E+2017%2E+A+corpus+of+natural+language+for+visual+reasoning%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28)
48. Takuma Udagawa, Takato Yamazaki, and Akiko Aizawa. 2020. A linguistic analysis of visually grounded dialogues based on spatial expressions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 750–765, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Takuma+Udagawa%2C+Takato+Yamazaki%2C+and+Akiko+Aizawa%2E+2020%2E+A+linguistic+analysis+of+visually+grounded+dialogues+based+on+spatial+expressions%2E+In+Findings+of+the+Association+for+Computational+Linguistics)
49. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+the)
50. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. [find](https://scholar.google.com/scholar?q=Jason+Weston%2C+Antoine+Bordes%2C+Sumit+Chopra%2C+Alexander+M+Rush%2C+Bart+van+Merri%C3%ABnboer%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2015%2E+Towards+ai%2Dcomplete+question+answering%3A+A+set+of+prerequisite+toy+tasks%2E+arX) [our card](https://md.mova.elamur.ai/free/en/docs/papers/weston-2015-towards-ai-complete-question-answering/)
51. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754–5764. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Russ+R+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+XLNet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+Advances+in+neural+inform)
52. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. pages 2369–2380. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D%2E+Manning%2E+2018%2E+HotpotQA%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+p)
53. Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1753– 1762. [find](https://scholar.google.com/scholar?q=Daojian+Zeng%2C+Kang+Liu%2C+Yubo+Chen%2C+and+Jun+Zhao%2E+2015%2E+Distant+supervision+for+relation+extraction+via+piecewise+convolutional+neural+networks%2E+In+Proceedings+of+the+2015+conference+on+empirical+metho)
54. Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with minimal supervision. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7579–7589. [find](https://scholar.google.com/scholar?q=Ben+Zhou%2C+Qiang+Ning%2C+Daniel+Khashabi%2C+and+Dan+Roth%2E+2020%2E+Temporal+common+sense+acquisition+with+minimal+supervision%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Li)
55. Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418–6428. [find](https://scholar.google.com/scholar?q=Alane+Suhr%2C+Stephanie+Zhou%2C+Ally+Zhang%2C+Iris+Zhang%2C+Huajun+Bai%2C+and+Yoav+Artzi%2E+2019%2E+A+corpus+for+reasoning+about+natural+language+grounded+in+photographs%2E+In+Proceedings+of+the+57th+Annual+Meeting+o) [our card](https://md.mova.elamur.ai/free/en/docs/papers/suhr-2019-corpus-reasoning-natural-language-grounded/)
56. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Patrick+Xia%2C+Berlin+Chen%2C+Alex+Wang%2C+Adam+Poliak%2C+R+Thomas+McCoy%2C+Najoung+Kim%2C+Benjamin+Van+Durme%2C+Samuel+R+Bowman%2C+Dipanjan+Das%2C+et+al%2E+2019%2E+What+do+you+learn+from)