[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/yu-2023-ifqa-dataset-open-domain-question/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions

# IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions

**Authors:** Wenhao Yu, Meng Jiang, Peter Clark and Ashish Sabharwal · **Year:** 2023 · **Venue:** Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing **Link:** [aclanthology.org/2023.emnlp-main.515](https://aclanthology.org/2023.emnlp-main.515/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology)

[**8** citations](https://openalex.org/works?filter=cites:W4389520750) openalex · 2026-10-18[**18** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/yu-2023-ifqa-dataset-open-domain-question/) of the 140673 papers whose bibliographies we read[Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://aclanthology.org/volumes/2023.emnlp-main/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2023.emnlp-main.515/)
- [PDF](https://aclanthology.org/2023.emnlp-main.515.pdf)
- [DOI 10.18653/v1/2023.emnlp-main.515](https://doi.org/10.18653/v1/2023.emnlp-main.515)
- [arXiv 2305.14010](https://arxiv.org/abs/2305.14010)
- [OpenAlex](https://openalex.org/W4389520750)

**Authors (4)**

1. [Wenhao Yu](https://openalex.org/A5055111969) [ORCID](https://orcid.org/0000-0003-4894-4601) · h 18 · 1362 citations · 74 works
2. [Meng Jiang](https://openalex.org/A5074821819) [ORCID](https://orcid.org/0000-0002-3009-519X) · h 41 · 6376 citations · 256 works
3. Peter Clark
4. [Ashish Sabharwal](https://openalex.org/A5077726785) h 46 · 8833 citations · 260 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

IfQA has about 3,800 open-domain questions, each with a counterfactual "if" clause, such as "If the movement of the earth's crust caused Everest to drop by 300 metres, what would be the highest mountain?". Crowdworkers wrote the questions on Wikipedia passages selected by causal keywords, and one or two further workers verified them. Answers are mostly spans (75.1%); by type, 49.7% are entities, 15.9% numbers, 14.5% dates and 19.9% others. The data come in a supervised split (2,400/700/700) and a few-shot split. Retrieval is the first bottleneck: at 20 passages both BM25 and DPR reach only about 60% recall, and DPR misses the gold passage in about 35% of questions. Retrieve-then-read systems are weak as well, and the best system pairs BM25 retrieval with a ChatGPT reader, with a few-shot test F1 of 57.81. The abstract reports F1 44.5 for the supervised pipeline and 57.2 for chain-of-thought ChatGPT.

## Ideas

- I1. A deterministic counterfactual operation in `world/src/qa.rs`: parse the if-clause, find the fact it replaces in the world, change the numeric or entity slot, and answer by a rule such as a comparison of heights. The paper's worked example (8,848 minus 300 against K2 at 8,611) is a good hand-checked test.
- I2. Report school question answering by answer type (entity, number, date, other), as the paper does, so that a weak type is visible rather than hidden in one score.
- I3. Retrieval miss analysis: count questions whose if-clause content words do not occur in the gold passage (the "sea level" case). Use `world/src/knowledge.rs` to map counterfactual wording to passage wording by a fixed lexicon.
- I4. Use the mood of the if-clause as a feature ("were" and "had" in the subjunctive), the feature chapter [9. The subjunctive, conditions and wishes](https://md.mova.elamur.ai/free/en/docs/languages/en-x-emode/09-subjunctive/) describes for older English.

## For Mova

The chapters are [25. Relative clauses](https://md.mova.elamur.ai/free/en/docs/languages/en/25-relative-clauses/) (the clause) and [26. Questions](https://md.mova.elamur.ai/free/en/docs/languages/en/26-questions/). The code is `world/src/qa.rs` and `world/src/knowledge.rs`. The measures are accuracy by answer type on a hand-checked set of counterfactual school questions, and the retrieval miss count. Run each twice for determinism.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2023.emnlp-main.515](https://doi.org/10.18653/v1/2023.emnlp-main.515) · [arXiv 2305.14010](https://arxiv.org/abs/2305.14010)

```
@inproceedings{yu2023ifqa,
  title = {{IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions}},
  author = {Yu, Wenhao and Jiang, Meng and Clark, Peter and Sabharwal, Ashish},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.515},
  eprint = {2305.14010},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.515}
}
```

- [← previous CREPE: Open-Domain Question Answering with False Presuppositions](https://md.mova.elamur.ai/free/en/docs/papers/yu-2023-crepe-open-domain-question-answering/)
- [next → AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference](https://md.mova.elamur.ai/free/en/docs/papers/yuan-2023-ambicoref-evaluating-human-model-sensitivity/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

49 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2022. Task-aware retrieval with instructions. arXiv preprint arXiv:2211.09260. [↗](https://arxiv.org/abs/2211.09260)
2. Andrew Atherton. 2005. A future for small business? prospective scenarios for the development of the economy based on current policy thinking and counterfactual reasoning. Futures, 37(8):777–794. [find](https://scholar.google.com/scholar?q=Andrew+Atherton%2E+2005%2E+A+future+for+small+business%3F+prospective+scenarios+for+the+development+of+the+economy+based+on+current+policy+thinking+and+counterfactual+reasoning%2E+Futures%2C+37%288%29%3A777%E2%80%93794%2E)
3. Robert A Baron. 2000. Counterfactual thinking and venture formation: The potential effects of thinking about “what might have been”. Journal of business venturing, 15(1):79–91. [find](https://scholar.google.com/scholar?q=Robert+A+Baron%2E+2000%2E+Counterfactual+thinking+and+venture+formation%3A+The+potential+effects+of+thinking+about+%E2%80%9Cwhat+might+have+been%E2%80%9D%2E+Journal+of+business+venturing%2C+15%281%29%3A79%E2%80%9391%2E)
4. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In EMNLP, pages 1533– 1544. [find](https://scholar.google.com/scholar?q=Jonathan+Berant%2C+Andrew+Chou%2C+Roy+Frostig%2C+and+Percy+Liang%2E+2013%2E+Semantic+parsing+on+freebase+from+question%2Danswer+pairs%2E+In+EMNLP%2C+pages+1533%E2%80%93+1544%2E)
5. Eric Brill, Susan Dumais, and Michele Banko. 2002. An analysis of the askmsr question-answering system. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 257–264. [find](https://scholar.google.com/scholar?q=Eric+Brill%2C+Susan+Dumais%2C+and+Michele+Banko%2E+2002%2E+An+analysis+of+the+askmsr+question%2Danswering+system%2E+In+Proceedings+of+the+2002+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+lear)
7. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879. [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Adam+Fisch%2C+Jason+Weston%2C+and+Antoine+Bordes%2E+2017%2E+Reading+wikipedia+to+answer+open%2Ddomain+questions%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Lingui)
8. Danqi Chen and Wen-tau Yih. 2020. Open-domain question answering. In Proceedings of the 58th annual meeting of the association for computational linguistics: tutorial abstracts, pages 34–37. [find](https://scholar.google.com/scholar?q=Danqi+Chen+and+Wen%2Dtau+Yih%2E+2020%2E+Open%2Ddomain+question+answering%2E+In+Proceedings+of+the+58th+annual+meeting+of+the+association+for+computational+linguistics%3A+tutorial+abstracts%2C+pages+34%E2%80%9337%2E)
9. Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W Cohen. 2020. Open question answering over tables and text. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Ming%2DWei+Chang%2C+Eva+Schlinger%2C+William+Yang+Wang%2C+and+William+W+Cohen%2E+2020%2E+Open+question+answering+over+tables+and+text%2E+In+International+Conference+on+Learning+Representations%2E)
10. Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive questions. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Xinyi+Wang%2C+and+William+Yang+Wang%2E+2021%2E+A+dataset+for+answering+time%2Dsensitive+questions%2E+In+Thirty%2Dfifth+Conference+on+Neural+Information+Processing+Systems+Datasets+and+Benchmarks+Track)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171– 4186. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
12. Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. 2022. Causal inference in natural language processing: Estimation, prediction, interpretation and beyond. Transactions of the Association for Computational Linguistics, 10:1138–1158. [find](https://scholar.google.com/scholar?q=Amir+Feder%2C+Katherine+A+Keith%2C+Emaad+Manzoor%2C+Reid+Pryzant%2C+Dhanya+Sridhar%2C+Zach+Wood%2DDoughty%2C+Jacob+Eisenstein%2C+Justin+Grimmer%2C+Roi+Reichart%2C+Margaret+E+Roberts%2C+et+al%2E+2022%2E+Causal+inference+in+natu)
13. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909. [↗](https://arxiv.org/abs/2002.08909)
14. Stephen J Hoch. 1985. Counterfactual reasoning and accuracy in predicting personal events. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11(4):719. [find](https://scholar.google.com/scholar?q=Stephen+J+Hoch%2E+1985%2E+Counterfactual+reasoning+and+accuracy+in+predicting+personal+events%2E+Journal+of+Experimental+Psychology%3A+Learning%2C+Memory%2C+and+Cognition%2C+11%284%29%3A719%2E)
15. Srinivasan Iyer, Sewon Min, Yashar Mehdad, and Wen-tau Yih. 2021. Reconsider: Improved re-ranking using span-focused cross-attention for open domain question answering. In Procs. of NAACL-HLT. [find](https://scholar.google.com/scholar?q=Srinivasan+Iyer%2C+Sewon+Min%2C+Yashar+Mehdad%2C+and+Wen%2Dtau+Yih%2E+2021%2E+Reconsider%3A+Improved+re%2Dranking+using+span%2Dfocused+cross%2Dattention+for+open+domain+question+answering%2E+In+Procs%2E+of+NAACL%2DHLT%2E)
16. Gautier Izacard and Edouard Grave. 2020. Distilling knowledge from reader to retriever for question answering. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Gautier+Izacard+and+Edouard+Grave%2E+2020%2E+Distilling+knowledge+from+reader+to+retriever+for+question+answering%2E+In+International+Conference+on+Learning+Representations%2E)
17. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In EACL, pages 874– 880. [find](https://scholar.google.com/scholar?q=Gautier+Izacard+and+Edouard+Grave%2E+2021%2E+Leveraging+passage+retrieval+with+generative+models+for+open+domain+question+answering%2E+In+EACL%2C+pages+874%E2%80%93+880%2E)
18. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299. [↗](https://arxiv.org/abs/2208.03299)
19. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, pages 1601–1611. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+S+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+Triviaqa%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+ACL%2C+pages+1601%E2%80%931611%2E)
20. Mingxuan Ju, Wenhao Yu, Tong Zhao, Chuxu Zhang, and Yanfang Ye. 2022. Grape: Knowledge graph enhanced passage reader for open-domain question answering. In Findings of Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Mingxuan+Ju%2C+Wenhao+Yu%2C+Tong+Zhao%2C+Chuxu+Zhang%2C+and+Yanfang+Ye%2E+2022%2E+Grape%3A+Knowledge+graph+enhanced+passage+reader+for+open%2Ddomain+question+answering%2E+In+Findings+of+Empirical+Methods+in+Natural+Lan)
21. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. [find](https://scholar.google.com/scholar?q=Vladimir+Karpukhin%2C+Barlas+Oguz%2C+Sewon+Min%2C+Patrick+Lewis%2C+Ledell+Wu%2C+Sergey+Edunov%2C+Danqi+Chen%2C+and+Wen%2Dtau+Yih%2E+2020%2E+Dense+passage+retrieval+for+open%2Ddomain+question+answering%2E+In+Proceedings+of+th)
22. Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2022. Realtime qa: What’s the answer right now? arXiv preprint arXiv:2207.13332. [↗](https://arxiv.org/abs/2207.13332)
23. Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Divyansh+Kaushik%2C+Eduard+Hovy%2C+and+Zachary+Lipton%2E+2019%2E+Learning+the+difference+that+makes+a+difference+with+counterfactually%2Daugmented+data%2E+In+International+Conference+on+Learning+Representations%2E)
24. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference for Learning Representation (ICLR). [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+International+Conference+for+Learning+Representation+%28ICLR%29%2E)
25. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: A benchmark for question answering research. TACL, pages 452– 466. [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Red%2Dfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Jacob+Devlin%2C+Kenton+Lee%2C+et+al%2E+2019%2E+Natural+questions%3A+A+be)
26. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ethan+Perez%2C+Aleksandra+Piktus%2C+Fabio+Petroni%2C+Vladimir+Karpukhin%2C+Naman+Goyal%2C+Heinrich+K%C3%BCttler%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Tim+Rockt%C3%A4schel%2C+et+al%2E+2020%2E+Retrieval%2Daugmented+generation+f)
27. Haonan Li, Martin Tomko, Maria Vasardani, and Timothy Baldwin. 2022. Multispanqa: A dataset for multi-span question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1250–1260. [find](https://scholar.google.com/scholar?q=Haonan+Li%2C+Martin+Tomko%2C+Maria+Vasardani%2C+and+Timothy+Baldwin%2E+2022%2E+Multispanqa%3A+A+dataset+for+multi%2Dspan+question+answering%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+th)
28. Zujie Liang, Weitao Jiang, Haifeng Hu, and Jiaying Zhu. 2020. Learning to contrast the counterfactual samples for robust visual question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 3285–3292. [find](https://scholar.google.com/scholar?q=Zujie+Liang%2C+Weitao+Jiang%2C+Haifeng+Hu%2C+and+Jiaying+Zhu%2E+2020%2E+Learning+to+contrast+the+counterfactual+samples+for+robust+visual+question+answering%2E+In+Proceedings+of+the+2020+conference+on+empirical+m)
29. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. International Conference for Learning Representation (ICLR). [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+International+Conference+for+Learning+Representation+%28ICLR%29%2E)
30. Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. Reader-guided passage reranking for open-domain question answering. In Findings of ACL-IJCNLP. [find](https://scholar.google.com/scholar?q=Yuning+Mao%2C+Pengcheng+He%2C+Xiaodong+Liu%2C+Yelong+Shen%2C+Jianfeng+Gao%2C+Jiawei+Han%2C+and+Weizhu+Chen%2E+2021%2E+Reader%2Dguided+passage+reranking+for+open%2Ddomain+question+answering%2E+In+Findings+of+ACL%2DIJCNLP%2E)
31. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783– 5797. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2020%2E+Ambigqa%3A+Answering+ambiguous+open%2Ddomain+questions%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+La)
32. Sewon Min, Luke Zettlemoyer, Hannaneh Hajishirzi, et al. 2022. Crepe: Open-domain question answering with false presuppositions. arXiv e-prints, pages arXiv–2211. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Luke+Zettlemoyer%2C+Hannaneh+Hajishirzi%2C+et+al%2E+2022%2E+Crepe%3A+Open%2Ddomain+question+answering+with+false+presuppositions%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932211%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/yu-2023-crepe-open-domain-question-answering/)
33. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. [↗](https://arxiv.org/abs/2201.11903)
34. Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2022. Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering. arXiv preprint arXiv:2211.05655. [↗](https://arxiv.org/abs/2211.05655)
35. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with bertserini. In NAACL 2019 (demo). [find](https://scholar.google.com/scholar?q=Wei+Yang%2C+Yuqing+Xie%2C+Aileen+Lin%2C+Xingyu+Li%2C+Luchen+Tan%2C+Kun+Xiong%2C+Ming+Li%2C+and+Jimmy+Lin%2E+2019%2E+End%2Dto%2Dend+open%2Ddomain+question+answering+with+bertserini%2E+In+NAACL+2019+%28demo%29%2E)
36. Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. 2021. Counterfactual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12700– 12710. [find](https://scholar.google.com/scholar?q=Yulei+Niu%2C+Kaihua+Tang%2C+Hanwang+Zhang%2C+Zhiwu+Lu%2C+Xian%2DSheng+Hua%2C+and+Ji%2DRong+Wen%2E+2021%2E+Counterfactual+vqa%3A+A+cause%2Deffect+look+at+language+bias%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+)
37. Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5043–5053. [find](https://scholar.google.com/scholar?q=Lianhui+Qin%2C+Antoine+Bosselut%2C+Ari+Holtzman%2C+Chandra+Bhagavatula%2C+Elizabeth+Clark%2C+and+Yejin+Choi%2E+2019%2E+Counterfactual+story+reasoning+and+generation%2E+In+Proceedings+of+the+2019+Conference+on+Empiric)
38. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835–5847. [find](https://scholar.google.com/scholar?q=Yingqi+Qu%2C+Yuchen+Ding%2C+Jing+Liu%2C+Kai+Liu%2C+Ruiyang+Ren%2C+Wayne+Xin+Zhao%2C+Daxiang+Dong%2C+Hua+Wu%2C+and+Haifeng+Wang%2E+2021%2E+Rocketqa%3A+An+optimized+training+approach+to+dense+passage+retrieval+for+open%2Ddomai)
39. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+Peter+J+Liu%2C+et+al%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dt)
40. Samuel Joseph Amouyal Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. 2022. Qampari:: An open-domain question answering benchmark for questions with many answers from multiple paragraphs. arXiv preprint arXiv:2205.12665. [↗](https://arxiv.org/abs/2205.12665)
41. Devendra Singh Sachan, Mike Lewis, Dani Yogatama, Luke Zettlemoyer, Joelle Pineau, and Manzil Zaheer. 2022. Questions are all you need to train a dense passage retriever. arXiv preprint arXiv:2206.10658. [↗](https://arxiv.org/abs/2206.10658)
42. Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. 2019. Wiqa: A dataset for “what if...” reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6076–6085. [find](https://scholar.google.com/scholar?q=Niket+Tandon%2C+Bhavana+Dalvi%2C+Keisuke+Sakaguchi%2C+Peter+Clark%2C+and+Antoine+Bosselut%2E+2019%2E+Wiqa%3A+A+dataset+for+%E2%80%9Cwhat+if%2E%2E%2E%E2%80%9D+reasoning+over+procedural+text%2E+In+Proceedings+of+the+2019+Conference+on+Empir)
43. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. [find](https://scholar.google.com/scholar?q=Harsh+Trivedi%2C+Niranjan+Balasubramanian%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2022%2E+Musique%3A+Multi%2Dhop+questions+via+single%2Dhop+question+composition%2E+Transactions+of+the+Association+for+Computational+Li)
44. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D+Manning%2E+2018%2E+Hotpotqa%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+In)
45. Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. International Conference for Learning Representation (ICLR). [find](https://scholar.google.com/scholar?q=Wenhao+Yu%2C+Dan+Iter%2C+Shuohang+Wang%2C+Yichong+Xu%2C+Mingxuan+Ju%2C+Soumya+Sanyal%2C+Chenguang+Zhu%2C+Michael+Zeng%2C+and+Meng+Jiang%2E+2023%2E+Generate+rather+than+retrieve%3A+Large+language+models+are+strong+context+g)
46. Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A survey of knowledge-enhanced text generation. ACM Computing Surveys (CSUR). [find](https://scholar.google.com/scholar?q=Wenhao+Yu%2C+Chenguang+Zhu%2C+Zaitang+Li%2C+Zhiting+Hu%2C+Qingyun+Wang%2C+Heng+Ji%2C+and+Meng+Jiang%2E+2022%2E+A+survey+of+knowledge%2Denhanced+text+generation%2E+ACM+Computing+Surveys+%28CSUR%29%2E)
47. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021a. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 3277–3287. [find](https://scholar.google.com/scholar?q=Fengbin+Zhu%2C+Wenqiang+Lei%2C+Youcheng+Huang%2C+Chao+Wang%2C+Shuo+Zhang%2C+Jiancheng+Lv%2C+Fuli+Feng%2C+and+Tat%2DSeng+Chua%2E+2021a%2E+Tat%2Dqa%3A+A+question+answering+benchmark+on+a+hybrid+of+tabular+and+textual+content+i)
48. Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021b. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774. [↗](https://arxiv.org/abs/2101.00774)
49. Qingfu Zhu, Weinan Zhang, Ting Liu, and William Yang Wang. 2020. Counterfactual off-policy training for neural dialogue generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3438–3448. [find](https://scholar.google.com/scholar?q=Qingfu+Zhu%2C+Weinan+Zhang%2C+Ting+Liu%2C+and+William+Yang+Wang%2E+2020%2E+Counterfactual+off%2Dpolicy+training+for+neural+dialogue+generation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natura)