[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/jiang-2021-he-thinks-he-knows-better/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics

# He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics

**Authors:** Nanjiang Jiang, [Marie-Catherine de Marneffe](https://md.mova.elamur.ai/free/en/docs/papers/authors/marie-catherine-de-marneffe/) · **Year:** 2021 · **Venue:** Transactions of the Association for Computational Linguistics 9, 1081–1097 **Link:** [aclanthology.org/2021.tacl-1.64](https://aclanthology.org/2021.tacl-1.64/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**8** citations](https://openalex.org/works?filter=cites:W3205597237) openalex · 2026-10-18[**13** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/jiang-2021-he-thinks-he-knows-better/) of the 140673 papers whose bibliographies we read[Transactions of the Association for Computational Linguistics](https://aclanthology.org/volumes/2021.tacl-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2021.tacl-1.64/)
- [PDF](https://aclanthology.org/2021.tacl-1.64.pdf)
- [DOI 10.1162/tacl_a_00414](https://doi.org/10.1162/tacl_a_00414)
- [arXiv 2107.00807](https://arxiv.org/abs/2107.00807)
- [OpenAlex](https://openalex.org/W3205597237)

**Authors (2)**

1. [Nanjiang Jiang](https://openalex.org/A5037009640) h 6 · 1130 citations · 13 works
2. [Marie-Catherine de Marneffe](https://openalex.org/A5112862689) h 31 · 8698 citations · 167 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The task is to predict, on a scale from −3 to +3, how factual an embedded event is. The authors fine-tune BERT on seven English datasets, among them FactBank, MEANTIME, UW, UDS-IH2, CommitmentBank (CB), Ross and Pavlick (RP) and MegaVeridicality. BERT does well on most of them, and sets new results on FactBank and UW, but it does so through surface patterns. An "expected inference" is computed from lexical signatures (the factive, implicative or neutral label of a verb in its frame). Where the gold label diverges from it, BERT errs: the slope of error against expected-inference error is positive for every dataset except FactBank. In CB, 55 of the 556 items with the largest errors were categorised by hand; the top category was "context suggests (non)factuality" (34 items, 61.8%). In RP, 250 items were categorised; the top category was lexical inference (88 items, 35.2%). The paper concludes that the model captures regularities, not pragmatics.

## Ideas

- I1. `prag/src/infer.rs`: build an expected-inference table from verb and frame signatures in the lexicon. For each embedded event, output the table's label next to the rule's label, and report the cases where they diverge. The table is the baseline a learned model must beat.
- I2. Use the paper's seven error categories (prior probability, context, QUD, tense/aspect, subject authority, subject–complement interaction, lexical inference) as a fixed checklist: after each Mova error on a factuality test, tag it with one category. This gives a measure of which reasoning step is missing.
- I3. Negative control: shuffle the preceding context of each item. If Mova's answers do not change, the context step is not used; if they change, measure whether the change is in the right direction.

## For Mova

The chapter is [23. Complementation](https://md.mova.elamur.ai/free/en/docs/languages/en/23-complementation/) (embedding frames) and [33. Implicature and presupposition](https://md.mova.elamur.ai/free/en/docs/languages/en/33-implicature-and-presupposition/). The code is `prag/src/infer.rs`, the verb and frame entries in `en/src/dict.rs`, and `prag/src/frames.rs`. Measures: agreement of the expected-inference baseline with gold labels; the category counts on a sample of 50 Mova errors; the effect of the shuffled-context control. Run twice. The paper's BERT results are a comparison point only and not something Mova uses.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.1162/tacl_a_00414](https://doi.org/10.1162/tacl_a_00414) · [arXiv 2107.00807](https://arxiv.org/abs/2107.00807)

```
@article{jiang2021he,
  title = {{He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics}},
  author = {Jiang, Nanjiang and Marneffe, Marie-Catherine de},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00414},
  eprint = {2107.00807},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00414}
}
```

- [← previous Adversarial Examples for Evaluating Reading Comprehension Systems](https://md.mova.elamur.ai/free/en/docs/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
- [next → Dependency-based Semantic Role Labeling of PropBank](https://md.mova.elamur.ai/free/en/docs/papers/johansson-2008-dependency-based-semantic-role-labeling/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

36 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Beaver. 2010. Have you noticed that your belly button lint colour is related to the colour of your clothing? In Rainer Bäuerle, Uwe Reyle, and Thomas Ede Zimmermann, editors, Presuppositions and Discourse: Essays Offered to Hans Kamp, pages 65–99. Leiden, The Netherlands: Brill. https://doi.org/10 .1163/9789004253162 004 [↗](https://doi.org/10)
2. Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English Web Treebank. Linguistic Data Consortium, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Ann+Bies%2C+Justin+Mott%2C+Colin+Warner%2C+and+Seth+Kulick%2E+2012%2E+English+Web+Treebank%2E+Linguistic+Data+Consortium%2C+Philadelphia%2C+PA%2E)
3. Rui P. Chaves. 2020. What don’t RNN language models learn about filler-gap dependencies? Proceedings of the Society for Computation in Linguistics, 3(1):20–30. [find](https://scholar.google.com/scholar?q=Rui+P%2E+Chaves%2E+2020%2E+What+don%E2%80%99t+RNN+language+models+learn+about+filler%2Dgap+dependencies%3F+Proceedings+of+the+Society+for+Computation+in+Linguistics%2C+3%281%29%3A20%E2%80%9330%2E)
4. Gennaro Chierchia and Sally McConnell-Ginet. 1990. Meaning and Grammar. MIT Press. tational Linguistics. https://doi.org/10 .18653/v1/D19-1630 [↗](https://aclanthology.org/D19-1630/)
5. Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2012. Did it happen? The pragmatic complexity of veridicality assessment. Computational Linguistics, 38(2):301–333. https://doi.org/10.1162/COLI a 00097 [↗](https://doi.org/10.1162/COLI)
6. Lauri Karttunen. 1971a. Implicative verbs. Language, 47(2):340–358. https://doi.org/10.2307/412084 [↗](https://doi.org/10.2307/412084)
7. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The CommitmentBank: Investigating projection in naturally occurring discourse. In Sinn und Bedeutung 23. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+de+Marneffe%2C+Mandy+Simons%2C+and+Judith+Tonhauser%2E+2019%2E+The+CommitmentBank%3A+Investigating+projection+in+naturally+occurring+discourse%2E+In+Sinn+und+Bedeutung+23%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/enprag-2019-marneffe-commitmentbank-investigating-projection-naturally/)
8. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6, Melbourne, Australia. Association for Computational Linguistics. https://doi.org/10.18653/v1/W18-2501 [↗](https://aclanthology.org/W18-2501/)
9. Paloma Jeretič, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IM-Plicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.768 [↗](https://doi.org/10.18653/v1/2020.acl-main.768) [our card](https://md.mova.elamur.ai/free/en/docs/papers/jeretic-2020-are-natural-language-inference-models/)
10. Nanjiang Jiang and Marie-Catherine de Marneffe. 2019a. Do you know that Florence is packed with visitors? Evaluating state-of-the-art models of speaker commitment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4208–4213, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1412 [↗](https://aclanthology.org/P19-1412/)
11. Nanjiang Jiang and Marie-Catherine de Marneffe. 2019b. Evaluating BERT for natural language inference: A case study on the CommitmentBank. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6086–6091, Hong Kong, China. Association for Compu-Lauri Karttunen. 1971b. Some observations on factivity. Paper in Linguistics, 4(1):55–69. https://doi.org/10.1080/08351817109370248 [↗](https://doi.org/10.1080/08351817109370248)
12. Lauri Karttunen. 2012. Simple and phrasal implicatives. In Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, pages 124–131. [find](https://scholar.google.com/scholar?q=Lauri+Karttunen%2E+2012%2E+Simple+and+phrasal+implicatives%2E+In+Proceedings+of+the+First+Joint+Conference+on+Lexical+and+Computational+Semantics+%2D+Volume+1%3A+Proceedings+of+the+Main+Conference+and+the+Share)
13. Paul Kiparsky and Carol Kiparsky. 1970. Fact. In M. Bierwisch and K. E. Heidolph, editors, Progress in Linguistics, pages 143–173. Mouton, The Hague, Paris. [find](https://scholar.google.com/scholar?q=Paul+Kiparsky+and+Carol+Kiparsky%2E+1970%2E+Fact%2E+In+M%2E+Bierwisch+and+K%2E+E%2E+Heidolph%2C+editors%2C+Progress+in+Linguistics%2C+pages+143%E2%80%93173%2E+Mouton%2C+The+Hague%2C+Paris%2E)
14. Kenton Lee, Yoav Artzi, Yejin Choi, and Luke Zettlemoyer. 2015. Event detection and factuality assessment with non-expert supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1643–1648. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Yoav+Artzi%2C+Yejin+Choi%2C+and+Luke+Zettlemoyer%2E+2015%2E+Event+detection+and+factuality+assessment+with+non%2Dexpert+supervision%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Nat)
15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. ROBERTA: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692)
16. Anne-Lyse Myriam Minard, Manuela Speranza, Ruben Urizar, Begona Altuna, Marieke van Erp, Anneleen Schoen, and Chantal van Son. 2016. MEANTIME, the newsreader multilingual event and time corpus. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), pages 4417–4422. [find](https://scholar.google.com/scholar?q=Anne%2DLyse+Myriam+Minard%2C+Manuela+Speranza%2C+Ruben+Urizar%2C+Begona+Altuna%2C+Marieke+van+Erp%2C+Anneleen+Schoen%2C+and+Chantal+van+Son%2E+2016%2E+MEANTIME%2C+the+newsreader+multilingual+event+and+time+corpus%2E+In+Pro)
17. Rowan Nairn, Cleo Condoravdi, and Lauri Karttunen. 2006. Computing relative polarity for textual inference. In Proceedings of the Fifth International Workshop on Inference in Computational Semantics (ICoS-5). [find](https://scholar.google.com/scholar?q=Rowan+Nairn%2C+Cleo+Condoravdi%2C+and+Lauri+Karttunen%2E+2006%2E+Computing+relative+polarity+for+textual+inference%2E+In+Proceedings+of+the+Fifth+International+Workshop+on+Inference+in+Computational+Semantics+%28) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2006-nairn-computing-relative-polarity-textual-inference/)
18. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.441 [↗](https://doi.org/10.18653/v1/2020.acl-main.441)
19. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 67–81. Brussels, Belgium, Association for Computational Linguistics. https://doi.org/10.18653/v1/D18-1007 [↗](https://doi.org/10.18653/v1/d18-1007)
20. Amir Pouran Ben Veyseh, Thien Huu Nguyen, and Dejing Dou. 2019. Graph based neural networks for event factuality prediction using syntactic and semantic structures. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4393–4399, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1432 [↗](https://doi.org/10.18653/v1/p19-1432) [our card](https://md.mova.elamur.ai/free/en/docs/papers/veyseh-2019-graph-based-neural-networks-event/)
21. Craige Roberts. 2012. Information structure in discourse: Towards an integrated formal theory of pragmatics. Semantics and Pragmatics, 5(6):1–69. https://doi.org/10.3765/sp .5.6 [↗](https://doi.org/10.3765/sp) [our card](https://md.mova.elamur.ai/free/en/docs/papers/roberts-2012-information-structure-discourse-towards-integrated/)
22. Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2230–2240, Hong Kong, China. Association for Computational Linguistics. https:// doi.org/10.18653/v1/D19-1228 [↗](https://doi.org/10.18653/v1/d19-1228) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ross-2019-how-well-do-nli-models/)
23. Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, pages 731–744. https://doi.org/10.18653/v1/N18-1067 [↗](https://aclanthology.org/N18-1067/) [our card](https://md.mova.elamur.ai/free/en/docs/papers/rudinger-2018-neural-models-factuality/)
24. Roser Saurı́. 2008. FactBank 1.0. annotation guidelines. [find](https://scholar.google.com/scholar?q=Roser+Saur%C4%B1%CC%81%2E+2008%2E+FactBank+1%2E0%2E+annotation+guidelines%2E)
25. Roser Saurı́ and James Pustejovsky. 2009. FactBank: A corpus annotated with event factuality. Language Resources and Evaluation, 43(3):227. https://doi.org/10.1007/s10579 -009-9089-9 [↗](https://doi.org/10.1007/s10579)
26. Philippe Schlenker. 2010. Local contexts and local meanings. Philosophical Studies, 151(1):115–142. https://doi.org/10 .1007/s11098-010-9586-0 [↗](https://doi.org/10)
27. Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the richness of the linguistic signal in predicting pragmatic inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.479 [↗](https://aclanthology.org/2020.acl-main.479/) [our card](https://md.mova.elamur.ai/free/en/docs/papers/schuster-2020-harnessing-linguistic-signal-predict-scalar/)
28. Mandy Simons, Judith Tonhauser, David Beaver, and Craige Roberts. 2010. What projects and why. In Proceedings of Semantics and Linguistic Theory 20. CLC Publications. https:// doi.org/10.3765/salt.v20i0.2584 [↗](https://doi.org/10.3765/salt.v20i0.2584)
29. E Allyn Smith and Kathleen Currie Hall. 2014. The relationship between projection and embedding environment. In Proceedings of the 48th Meeting of the Chicago Linguistics Society. Citeseer. [find](https://scholar.google.com/scholar?q=E+Allyn+Smith+and+Kathleen+Currie+Hall%2E+2014%2E+The+relationship+between+projection+and+embedding+environment%2E+In+Proceedings+of+the+48th+Meeting+of+the+Chicago+Linguistics+Society%2E+Citeseer%2E)
30. Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan, and Iryna Gurevych. 2017. Integrating deep linguistic features in factuality prediction over unified datasets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 352–357. https://doi.org/10.18653/v1/P17 -2056 [↗](https://doi.org/10.18653/v1/p17)
31. Judith Tonhauser, David I. Beaver, and Judith Degen. 2018. How projective is projective content? Gradience in projectivity and at-issueness. Journal of Semantics, 35(3):495–542. https:// doi.org/10.1093/jos/ffy007 Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Phil Yeres, Jason Phang, Haokun Liu, Phu Mon Htut, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Edouard Grave, Najoung Kim, Thibault Févry, Berlin Chen, Nikita Nangia, Anhad Mohananey, Katharina Kann, Shikha Bordia, Nicolas Patry, David Benton, Ellie Pavlick, and Samuel R. Bowman. 2019. jiant 1.3: A software toolkit for research on general-purpose text understanding models. http://jiant.info/ [↗](https://doi.org/10.1093/jos/ffy007)
32. Aaron Steven White, Drew Reisinger, Keisuke Sakaguchi, Tim Vieira, Sheng Zhang, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2016. Universal decompositional semantics on Universal Dependencies. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1713–1723, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aaron+Steven+White%2C+Drew+Reisinger%2C+Keisuke+Sakaguchi%2C+Tim+Vieira%2C+Sheng+Zhang%2C+Rachel+Rudinger%2C+Kyle+Rawlins%2C+and+Benjamin+Van+Durme%2E+2016%2E+Universal+decompositional+semantics+on+Universal+Dependenci) [our card](https://md.mova.elamur.ai/free/en/docs/papers/white-2016-universal-decompositional-semantics-universal-dependencies/)
33. Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. [find](https://scholar.google.com/scholar?q=Aaron+Steven+White%2C+Rachel+Rudinger%2C+Kyle+Rawlins%2C+and+Benjamin+Van+Durme%2E+2018%2E+Lexicosyntactic+inference+in+neural+models%2E)
34. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4717–4724, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+4717%E2%80%934724%2C+Brussels%2C+Belgium%2E+Association+for+Computational+Linguistics%2E)
35. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-1101 [↗](https://doi.org/10.18653/v1/n18-1101)
36. Adina Williams, Tristan Thrush, and Douwe Kiela. 2020. ANLIzing the adversarial natural language inference dataset. CoRR, abs/2010.12729. [↗](https://arxiv.org/abs/2010.12729)