[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/sravanthi-2024-pub-pragmatics-understanding-benchmark-assessing/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities

# PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities

**Authors:** Settaluri Sravanthi, Meet Doshi, Pavan Tankala, Rudra Murthy, Raj Dabre, [Pushpak Bhattacharyya](https://md.mova.elamur.ai/free/en/docs/papers/authors/pushpak-bhattacharyya/) · **Year:** 2024 · **Venue:** Findings of ACL 2024 **Link:** [aclanthology.org/2024.findings-acl.719](https://aclanthology.org/2024.findings-acl.719/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (paper); the component datasets keep their own licences, so they are for tests only

[**22** citations](https://openalex.org/works?filter=cites:W4402670074) openalex · 2026-10-18[**53** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/sravanthi-2024-pub-pragmatics-understanding-benchmark-assessing/) of the 140673 papers whose bibliographies we read[Findings of the Association for Computational Linguistics: ACL 2024](https://aclanthology.org/volumes/2024.findings-acl/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2024.findings-acl.719/)
- [PDF](https://aclanthology.org/2024.findings-acl.719.pdf)
- [DOI 10.18653/v1/2024.findings-acl.719](https://doi.org/10.18653/v1/2024.findings-acl.719)
- [arXiv 2401.07078](https://arxiv.org/abs/2401.07078)
- [OpenAlex](https://openalex.org/W4402670074)

**Authors (6)**

1. [Settaluri Sravanthi](https://openalex.org/A5113392174) h 1 · 24 citations · 4 works
2. [Meet Doshi](https://openalex.org/A5111333814) h 2 · 27 citations · 11 works
3. [Pavan Tankala](https://openalex.org/A5107568518) h 2 · 25 citations · 5 works
4. [Rudra Murthy](https://openalex.org/A5000493089) [ORCID](https://orcid.org/0009-0003-8961-7866) · h 10 · 269 citations · 53 works
5. [Raj Dabre](https://openalex.org/A5077362985) [ORCID](https://orcid.org/0000-0003-0664-3421) · h 21 · 1749 citations · 199 works
6. [Pushpak Bhattacharyya](https://openalex.org/A5065100828) [ORCID](https://orcid.org/0000-0001-5319-5508) · h 52 · 14436 citations · 867 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

PUB gathers fourteen multiple-choice tasks over four phenomena. Ten are implicature tasks: direct vs indirect response, the meaning of an indirect answer with or without a hint (Circa and a new CircaPlus), GRICE dialogues, IMPPRES pairs, agreement vs sarcasm detection and figurative language with positive or contrastive hints (FLUTE, FigQA). Two are presupposition tasks: NOPE, and a new DialogAssumptions set of presuppositions in DailyDialog without trigger words. Deixis and reference are covered by a deixis task and a new metonymy QA set. There are 28k items, 6.1k of them newly annotated. Nine LLMs (T5, Flan-T5, Llama-2 sizes, GPT-3.5) are tested with cloze and multiple-choice prompting. The authors also check whether answers survive shuffling of the options (proportion of plurality agreement). Models trail humans widely, especially on presupposition and on sarcasm vs agreement. They are swayed by hints that share words with an answer, and they fail even on deixis, which needs no world knowledge. Instruction tuning helps small models.

## Ideas

- I1. Adopt the plurality-agreement check for Mova's student. Every school multiple-choice question in the courses should get the same answer under all orderings of its options. Run `world/src/knowledge.rs` on each test question with the options rotated and report any question whose answer changes. A deterministic scorer should pass this at 100%, and any failure is a bug (order-dependent tie-breaking). This is a cheap negative-control gate in the spirit of "run twice, compare".
- I2. The deixis task needs no world knowledge, which suits rules. Mova already maps «I/you» inside quotes to speaker and addressee (`prag::frames` Ctx). Extend it to «here/there», «now/then», «this/that» and «today/yesterday» in quoted speech, mapping them to the Say frame's place and time, and test on our own short dialogues in the PUB format.
- I3. The contrastive-hint effect (models pick the option that shares words with the hint) is a known risk for Mova's own scoring, which matches roles and entities. Add contrastive distractors to a few course questions — an option that repeats the lesson's words with the polarity or the role flipped — and see whether the student's negation and role checks hold. This links to the «a disagreeing negation weighs too little» finding in [Watching the SLM think: the diagnosis of its answers](https://md.mova.elamur.ai/free/en/docs/docs/mmm-diagnosis/).

## For Mova

[33. Implicature and presupposition](https://md.mova.elamur.ai/free/en/docs/languages/en/33-implicature-and-presupposition/) §13 (datasets), [38. Dialogue, discourse markers and reported speech](https://md.mova.elamur.ai/free/en/docs/languages/en/38-dialogue-and-discourse-markers/) §7.3 (deixis in quotes); `world/src/knowledge.rs`, `prag/src/frames.rs`; measures: option-order invariance (must be 100%), deixis accuracy on our own items, and student accuracy on the contrastive-distractor questions.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2024.findings-acl.719](https://doi.org/10.18653/v1/2024.findings-acl.719) · [arXiv 2401.07078](https://arxiv.org/abs/2401.07078)

```
@inproceedings{sravanthi2024pub,
  title = {{PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities}},
  author = {Sravanthi, Settaluri and Doshi, Meet and Tankala, Pavan and Murthy, Rudra and Dabre, Raj and Bhattacharyya, Pushpak},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.719},
  eprint = {2401.07078},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.719}
}
```

- [← previous Sentence Level Discourse Parsing using Syntactic and Lexical Information](https://md.mova.elamur.ai/free/en/docs/papers/soricut-2003-sentence-level-discourse-parsing-syntactic/)
- [next → Towards a Better Exploitation of the Brown 'Family' Corpora in Diachronic Studies of British and American English Language Varieties](https://md.mova.elamur.ai/free/en/docs/papers/stajner-2011-better-exploitation-brown-family-corpora/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

48 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jens Allwood. 1981. On the distinctions between semantics and pragmatics. In Crossing the Boundaries in Linguistics: Studies Presented to Manfred Bierwisch, pages 177–189. Springer. [find](https://scholar.google.com/scholar?q=Jens+Allwood%2E+1981%2E+On+the+distinctions+between+semantics+and+pragmatics%2E+In+Crossing+the+Boundaries+in+Linguistics%3A+Studies+Presented+to+Manfred+Bierwisch%2C+pages+177%E2%80%93189%2E+Springer%2E)
2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
3. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 7139–7159. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tuhin+Chakrabarty%2C+Arkadiy+Saakyan%2C+Debanjan+Ghosh%2C+and+Smaranda+Muresan%2E+2022%2E+FLUTE%3A+figurative+language+understanding+through+textual+explanations%2E+In+Proceedings+of+the+2022+Conference+on+Empirica)
4. Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311. [↗](https://arxiv.org/abs/2204.02311)
5. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. CoRR, abs/2210.11416. [↗](https://arxiv.org/abs/2210.11416)
6. Andre Cianflone, Yulan Feng, Jad Kabbara, and Jackie Chi Kit Cheung. 2018. Let’s do it" again": A first computational approach to detecting adverbial presupposition triggers. arXiv preprint arXiv:1806.04262. [↗](https://arxiv.org/abs/1806.04262) [our card](https://md.mova.elamur.ai/free/en/docs/papers/cianflone-2018-let-s-do-it-again/)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. CoRR, abs/2107.03374. [↗](https://arxiv.org/abs/2107.03374)
8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457. [↗](https://arxiv.org/abs/1803.05457)
9. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin-odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Trans. Assoc. Comput. Linguistics, 9:346–361. [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+Parker+Schuh%2C+Kensen+Shi%2C+Sasha+Tsvyashc)
10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168)
11. Lingjia Deng, Janyce Wiebe, and Yoonjung Choi. 2014. Joint inference and disambiguation of implicit sentiments via implicature constraints. In COLING 2014, 25th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, August 23-29, 2014, Dublin, Ireland, pages 79–88. ACL. [find](https://scholar.google.com/scholar?q=Lingjia+Deng%2C+Janyce+Wiebe%2C+and+Yoonjung+Choi%2E+2014%2E+Joint+inference+and+disambiguation+of+implicit+sentiments+via+implicature+constraints%2E+In+COLING+2014%2C+25th+International+Conference+on+Computation)
12. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. [find](https://scholar.google.com/scholar?q=Herbert+P+Grice%2E+1975%2E+Logic+and+conversation%2E+In+Speech+acts%2C+pages+41%E2%80%9358%2E+Brill%2E)
13. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938. [↗](https://arxiv.org/abs/2105.09938)
14. Laurence R Horn and Gregory L Ward. 2004. The handbook of pragmatics. Wiley Online Library. [find](https://scholar.google.com/scholar?q=Laurence+R+Horn+and+Gregory+L+Ward%2E+2004%2E+The+handbook+of+pragmatics%2E+Wiley+Online+Library%2E)
15. Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023. A fine-grained comparison of pragmatic language understanding in humans and language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 4194–4213. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Sammy+Floyd%2C+Olessia+Jouravlev%2C+Evelina+Fedorenko%2C+and+Edward+Gibson%2E+2023%2E+A+fine%2Dgrained+comparison+of+pragmatic+language+understanding+in+humans+and+language+models%2E+In+Proceedings+of+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/hu-2023-fine-grained-comparison-pragmatic-language/)
16. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models imppressive? learning implicature and presupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8690–8705. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Paloma+Jeretic%2C+Alex+Warstadt%2C+Suvrat+Bhooshan%2C+and+Adina+Williams%2E+2020%2E+Are+natural+language+inference+models+imppressive%3F+learning+implicature+and+presupposition%2E+In+Proceedings+of+the+58th+Annual+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/jeretic-2020-are-natural-language-inference-models/)
17. Jad Kabbara and Jackie Chi Kit Cheung. 2022. Investigating the performance of transformer-based NLI models on presuppositional inferences. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 779– 785. International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jad+Kabbara+and+Jackie+Chi+Kit+Cheung%2E+2022%2E+Investigating+the+performance+of+transformer%2Dbased+NLI+models+on+presuppositional+inferences%2E+In+Proceedings+of+the+29th+International+Conference+on+Comput)
18. Jad Kabbara and Jackie Chi Kit Cheung. 2023. Investigating the effect of pre-finetuning bert models on nli involving presuppositions. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10482–10494. [find](https://scholar.google.com/scholar?q=Jad+Kabbara+and+Jackie+Chi+Kit+Cheung%2E+2023%2E+Investigating+the+effect+of+pre%2Dfinetuning+bert+models+on+nli+involving+presuppositions%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNL)
19. Najoung Kim, Phu Mon Htut, Samuel R. Bowman, and Jackson Petty. 2022. (qa)2 : Question answering with questionable assumptions. CoRR, abs/2212.10003. [↗](https://arxiv.org/abs/2212.10003)
20. Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist invented the lightbulb? presupposition verification for question-answering. arXiv preprint arXiv:2101.00391. [↗](https://arxiv.org/abs/2101.00391) [our card](https://md.mova.elamur.ai/free/en/docs/papers/kim-2021-which-linguist-invented-lightbulb-presupposition/)
21. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466. [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Red%2Dfield%2C+Michael+Collins%2C+Ankur+P%2E+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+Toutanova%2C+Llion+Jones%2C+)
22. Shibamouli Lahiri. 2015. Squinky! A corpus of sentence-level formality, informativeness, and implicature. CoRR, abs/1506.02306. [↗](https://arxiv.org/abs/1506.02306)
23. George Lakoff and Mark Johnson. 2008. Metaphors we live by. University of Chicago press. [find](https://scholar.google.com/scholar?q=George+Lakoff+and+Mark+Johnson%2E+2008%2E+Metaphors+we+live+by%2E+University+of+Chicago+press%2E)
24. Elissa Li, Sebastian Schuster, and Judith Degen. 2021. Predicting scalar inferences from “or” to “not both” using neural sentence encoders. In Proceedings of the Society for Computation in Linguistics 2021, pages 446–450. [find](https://scholar.google.com/scholar?q=Elissa+Li%2C+Sebastian+Schuster%2C+and+Judith+Degen%2E+2021%2E+Predicting+scalar+inferences+from+%E2%80%9Cor%E2%80%9D+to+%E2%80%9Cnot+both%E2%80%9D+using+neural+sentence+encoders%2E+In+Proceedings+of+the+Society+for+Computation+in+Linguistics) [our card](https://md.mova.elamur.ai/free/en/docs/papers/li-2021-predicting-scalar-inferences-or-not/)
25. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 - December 1, 2017 - Volume 1: Long Papers, pages 986–995. Asian Federation of Natural Language Processing. [find](https://scholar.google.com/scholar?q=Yanran+Li%2C+Hui+Su%2C+Xiaoyu+Shen%2C+Wenjie+Li%2C+Ziqiang+Cao%2C+and+Shuzi+Niu%2E+2017%2E+Dailydialog%3A+A+manually+labelled+multi%2Dturn+dialogue+dataset%2E+In+Proceedings+of+the+Eighth+International+Joint+Conference+o)
26. Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022. Testing the ability of language models to interpret figurative language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 4437– 4452. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emmy+Liu%2C+Chenxuan+Cui%2C+Kenneth+Zheng%2C+and+Graham+Neubig%2E+2022%2E+Testing+the+ability+of+language+models+to+interpret+figurative+language%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Cha)
27. Annie Louis, Dan Roth, and Filip Radlinski. 2020. " i’d rather just go to bed": Understanding indirect answers. arXiv preprint arXiv:2010.03450. [↗](https://arxiv.org/abs/2010.03450) [our card](https://md.mova.elamur.ai/free/en/docs/papers/louis-2020-i-d-rather-just-go/)
28. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. ArXiv, abs/1609.07843. [↗](https://arxiv.org/abs/1609.07843)
29. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2381–2391. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+A+new+dataset+for+open+book+question+answering%2E+In+Proceedings+of+the+2018+Conference+on+)
30. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and R. Fernández. 2016. The lambada dataset: Word prediction requiring a broad discourse context. ArXiv, abs/1606.06031. [↗](https://arxiv.org/abs/1606.06031)
31. Alicia Parrish, Sebastian Schuster, Alex Warstadt, Omar Agha, Soo-Hwan Lee, Zhuoye Zhao, Samuel R. Bowman, and Tal Linzen. 2021. NOPE: A corpus of naturally-occurring presuppositions in english. In Proceedings of the 25th Conference on Computational Natural Language Learning, CoNLL 2021, Online, November 10-11, 2021, pages 349–366. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alicia+Parrish%2C+Sebastian+Schuster%2C+Alex+Warstadt%2C+Omar+Agha%2C+Soo%2DHwan+Lee%2C+Zhuoye+Zhao%2C+Samuel+R%2E+Bowman%2C+and+Tal+Linzen%2E+2021%2E+NOPE%3A+A+corpus+of+naturally%2Doccurring+presuppositions+in+english%2E+In+Pr) [our card](https://md.mova.elamur.ai/free/en/docs/papers/parrish-2021-nope-corpus-naturally-occurring-presuppositions/)
32. Peng Qi, Nina Du, Christopher D. Manning, and Jing Huang. 2023. Pragmaticqa: A dataset for pragmatic question answering in conversations. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 6175– 6191. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peng+Qi%2C+Nina+Du%2C+Christopher+D%2E+Manning%2C+and+Jing+Huang%2E+2023%2E+Pragmaticqa%3A+A+dataset+for+pragmatic+question+answering+in+conversations%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/qi-2023-pragmaticqa-dataset-pragmatic-question-answering/)
33. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
34. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 784–789. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+2018%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+squad%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguisti)
35. Joshua Robinson and David Wingate. 2023. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. [find](https://scholar.google.com/scholar?q=Joshua+Robinson+and+David+Wingate%2E+2023%2E+Leveraging+large+language+models+for+multiple+choice+question+answering%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+ICLR+2023%2C+Kigal)
36. Laura Eline Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2023. The goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicature resolution by llms. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Laura+Eline+Ruis%2C+Akbir+Khan%2C+Stella+Biderman%2C+Sara+Hooker%2C+Tim+Rockt%C3%A4schel%2C+and+Edward+Grefenstette%2E+2023%2E+The+goldilocks+of+pragmatic+understanding%3A+Fine%2Dtuning+strategy+matters+for+implicature+reso)
37. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. 2022. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100. [↗](https://arxiv.org/abs/2211.05100)
38. Sebastian Schuster, Yuxing Chen, and Judith Degen. 2019. Harnessing the linguistic signal to predict scalar inferences. arXiv preprint arXiv:1910.14254. [↗](https://arxiv.org/abs/1910.14254) [our card](https://md.mova.elamur.ai/free/en/docs/papers/schuster-2020-harnessing-linguistic-signal-predict-scalar/)
39. Damien Sileo, Philippe Muller, Tim Van de Cruys, and Camille Pradel. 2022. A pragmatics-centered evaluation framework for natural language understanding. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022, pages 2382–2394. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Damien+Sileo%2C+Philippe+Muller%2C+Tim+Van+de+Cruys%2C+and+Camille+Pradel%2E+2022%2E+A+pragmatics%2Dcentered+evaluation+framework+for+natural+language+understanding%2E+In+Proceedings+of+the+Thirteenth+Language+Reso)
40. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Her-rick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR, abs/2206.04615. [↗](https://arxiv.org/abs/2206.04615)
41. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Moly-bog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288. [↗](https://arxiv.org/abs/2307.09288)
42. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261–3275. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019a%2E+Superglue%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understa)
43. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019b%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+7th+Internatio)
44. Yizhong Wang, Swaroop Mishra, Pegah Alipoormo-labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An-jana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuz-nia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 5085–5109. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormo%2Dlabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Atharva+Naik%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+An%2Djana+Arunkumar%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karam)
45. Albert Webson and Ellie Pavlick. 2021. Do prompt-based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01247. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations. [↗](https://arxiv.org/abs/2109.01247)
46. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+C)
47. George Yule. 1996. Pragmatics. Oxford university press. [find](https://scholar.google.com/scholar?q=George+Yule%2E+1996%2E+Pragmatics%2E+Oxford+university+press%2E)
48. Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu, and Song-Chun Zhu. 2021. GRICE: A grammar-based dataset for recovering implicature and conversational reasoning. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 2074–2085. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zilong+Zheng%2C+Shuwen+Qiu%2C+Lifeng+Fan%2C+Yixin+Zhu%2C+and+Song%2DChun+Zhu%2E+2021%2E+GRICE%3A+A+grammar%2Dbased+dataset+for+recovering+implicature+and+conversational+reasoning%2E+In+Findings+of+the+Association+for+Com) [our card](https://md.mova.elamur.ai/free/en/docs/papers/zheng-2021-grice-grammar-based-dataset-recovering/)