[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

# Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

**Authors:** [R. Thomas McCoy](https://md.mova.elamur.ai/free/en/docs/papers/authors/r-thomas-mccoy/), [Ellie Pavlick](https://md.mova.elamur.ai/free/en/docs/papers/authors/ellie-pavlick/), [Tal Linzen](https://md.mova.elamur.ai/free/en/docs/papers/authors/tal-linzen/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics **Link:** [aclanthology.org/P19-1334](https://aclanthology.org/P19-1334/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology)

[**943** citations](https://openalex.org/works?filter=cites:W2951286828) openalex · 2026-10-18[**699** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/) of the 140673 papers whose bibliographies we read[Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics](https://aclanthology.org/volumes/P19-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/P19-1334/)
- [PDF](https://aclanthology.org/P19-1334.pdf)
- [DOI 10.18653/v1/p19-1334](https://doi.org/10.18653/v1/p19-1334)
- [arXiv 1902.01007](https://arxiv.org/abs/1902.01007)
- [OpenAlex](https://openalex.org/W2951286828)

**Authors (3)**

1. [R. Thomas McCoy](https://openalex.org/A5081894510) h 4 · 1093 citations · 10 works
2. [Ellie Pavlick](https://openalex.org/A5053850863) [ORCID](https://orcid.org/0000-0002-7155-5420) · h 33 · 6871 citations · 194 works
3. [Tal Linzen](https://openalex.org/A5081824828) [ORCID](https://orcid.org/0000-0003-0435-6912) · h 39 · 6472 citations · 242 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

Statistical NLI models can score high on MNLI while relying on three syntactic shortcuts: lexical overlap (a premise entails any hypothesis built from its words), the subsequence heuristic (it entails any contiguous word span), and the constituent heuristic (it entails any complete subtree). The authors build HANS, a template-generated test set of 30 subcases with 10,000 examples per heuristic, where the shortcuts give the wrong label on half of the items. Four models (DA, ESIM, SPINN, BERT) trained on MNLI score well on MNLI but almost always predict entailment on HANS, so accuracy on the non-entailment cases is near 0%; BERT gets 0% on subject-object swaps. Adding 30,000 HANS-like training examples (about 8% of MNLI) lifts HANS accuracy strongly and changes MNLI test accuracy only a little (BERT 84.1 to 84.4, DA 72.4 to 66.0). Human accuracy on HANS is 76% on crowd workers and 97% for experts.

## Ideas

- I1. Generate a HANS-style set for Mova from the templates in Tables 4 to 6 of the paper: a deterministic generator with BTreeMap-ordered vocabularies (plausible nouns and verbs only), and check each generated pair with the frames reader in `prag/src/frames.rs`. Measure per-subcase accuracy, subject-object swap first.
- I2. Negative control for the frame reader: for every sentence in the set, swap subject and object (or move a clause under "if", "unless", "because") and require the frames to differ. A reader that only collects bags of words fails this, and the failure count is a direct measure of order blindness.
- I3. Count the supporting and contradicting cases in Mova's own lesson data (`world/src/knowledge.rs`) per subcase, as the paper does for MNLI (2,158 vs 261 for lexical overlap). Where the contradicting count is near zero, add examples before any rule is tuned on it.
- I4. Treat embedded clauses under conditionals and "unless" as non-asserted. A rule in `prag` that refuses to add the clause under "if" to the asserted facts is a deterministic counterpart of the constituent heuristic cases; measure it on the 12 constituent subcases.

## For Mova

The grammar chapters are [24. Coordination and subordination](https://md.mova.elamur.ai/free/en/docs/languages/en/24-coordination-subordination/) (conjunctions, subsequence cases), [25. Relative clauses](https://md.mova.elamur.ai/free/en/docs/languages/en/25-relative-clauses/) (relative clauses on subject and object) and [21. The simple clause](https://md.mova.elamur.ai/free/en/docs/languages/en/21-clause-structure/) (embedded clauses). The code is `prag/src/frames.rs` and `prag/src/roles.rs` for who-did-what-to-whom, `en/src/parse.rs` for trees, and `world/src/babi.rs` for the test harness. The measure is HANS-style accuracy per subcase, run twice to confirm it is identical, with the swapped-role control reported next to it.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/p19-1334](https://doi.org/10.18653/v1/p19-1334) · [arXiv 1902.01007](https://arxiv.org/abs/1902.01007)

```
@inproceedings{mccoy2019right,
  title = {{Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference}},
  author = {McCoy, R. Thomas and Pavlick, Ellie and Linzen, Tal},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1334},
  eprint = {1902.01007},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1334}
}
```

- [← previous English Grammar, including the principles of grammatical analysis](https://md.mova.elamur.ai/free/en/docs/papers/mason-1877-english-grammar-including-principles/)
- [next → Abstraction and Generalisation in Semantic Role Labels: PropBank, VerbNet or both?](https://md.mova.elamur.ai/free/en/docs/papers/merlo-2009-abstraction-generalisation-semantic-role-labels/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

46 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yossi+Adi%2C+Einat+Kermany%2C+Yonatan+Belinkov%2C+Ofer+Lavi%2C+and+Yoav+Goldberg%2E+2017%2E+Fine%2Dgrained+analysis+of+sentence+embeddings+using+auxiliary+prediction+tasks%2E+In+International+Conference+on+Learning+R)
2. Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. Analyzing the behavior of visual question answering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1955–1960. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aishwarya+Agrawal%2C+Dhruv+Batra%2C+and+Devi+Parikh%2E+2016%2E+Analyzing+the+behavior+of+visual+question+answering+models%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Proces)
3. Thomas G. Bever. 1970. The cognitive basis for linguistic structures. [find](https://scholar.google.com/scholar?q=Thomas+G%2E+Bever%2E+1970%2E+The+cognitive+basis+for+linguistic+structures%2E)
4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
5. Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1466–1477. Association for Computational Linguistics. ternational Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW’05, pages 177–190, Berlin, Heidelberg. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Jon+Gauthier%2C+Abhinav+Rastogi%2C+Raghav+Gupta%2C+Christopher+D%2E+Manning%2C+and+Christopher+Potts%2E+2016%2E+A+fast+unified+model+for+parsing+and+sentence+understanding%2E+In+Proceedings+of+the+5)
6. Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J. Gershman, and Noah D. Goodman. 2018. Evaluating compositionality in sentence embeddings. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 1596– 1601, Madison, WI. [find](https://scholar.google.com/scholar?q=Ishita+Dasgupta%2C+Demi+Guo%2C+Andreas+Stuhlmu%CC%88ller%2C+Samuel+J%2E+Gershman%2C+and+Noah+D%2E+Goodman%2E+2018%2E+Evaluating+compositionality+in+sentence+embeddings%2E+In+Proceedings+of+the+40th+Annual+Conference+of+the+)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
8. Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sentence vector representations. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1790–1801. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2C+Ahmed+Elgohary%2C+Colin+Phillips%2C+and+Philip+Resnik%2E+2018%2E+Assessing+composition+in+sentence+vector+representations%2E+In+Proceedings+of+the+27th+International+Conference+on+Computationa)
9. Lyn Frazier and Keith Rayner. 1982. Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14(2):178–210. [find](https://scholar.google.com/scholar?q=Lyn+Frazier+and+Keith+Rayner%2E+1982%2E+Making+and+correcting+errors+during+sentence+comprehension%3A+Eye+movements+in+the+analysis+of+structurally+ambiguous+sentences%2E+Cognitive+Psychology%2C+14%282%29%3A178%E2%80%93210%2E)
10. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+LSTM+for+natural+language+inference%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computa)
11. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. AllenNLP: A Deep Semantic Natural Language Processing Platform. In Proceedings of the Workshop for NLP Open Source Software (NLP-OSS). [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Joel+Grus%2C+Mark+Neumann%2C+Oyvind+Tafjord%2C+Pradeep+Dasigi%2C+Nelson+F%2E+Liu%2C+Matthew+Peters%2C+Michael+Schmitz%2C+and+Luke+S%2E+Zettlemoyer%2E+2017%2E+AllenNLP%3A+A+Deep+Semantic+Natural+Language+Process)
12. Kiel Christianson, Andrew Hollingworth, John F Hal-liwell, and Fernanda Ferreira. 2001. Thematic roles assigned along the garden path linger. Cognitive Psychology, 42(4):368–407. [find](https://scholar.google.com/scholar?q=Kiel+Christianson%2C+Andrew+Hollingworth%2C+John+F+Hal%2Dliwell%2C+and+Fernanda+Ferreira%2E+2001%2E+Thematic+roles+assigned+along+the+garden+path+linger%2E+Cognitive+Psychology%2C+42%284%29%3A368%E2%80%93407%2E)
13. Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. 2018. Stress-testing neural models of natural language inference with multiply-quantified sentences. arXiv preprint arXiv:1810.13033. [↗](https://arxiv.org/abs/1810.13033)
14. Cleo Condoravdi, Dick Crouch, Valeria de Paiva, Reinhard Stolle, and Daniel G. Bobrow. 2003. Entailment, intensionality and text understanding. In Proceedings of the HLT-NAACL 2003 Workshop on Text Meaning. [find](https://scholar.google.com/scholar?q=Cleo+Condoravdi%2C+Dick+Crouch%2C+Valeria+de+Paiva%2C+Reinhard+Stolle%2C+and+Daniel+G%2E+Bobrow%2E+2003%2E+Entailment%2C+intensionality+and+text+understanding%2E+In+Proceedings+of+the+HLT%2DNAACL+2003+Workshop+on+Text+Me)
15. Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126–2136. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Germa%CC%81n+Kruszewski%2C+Guillaume+Lample%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Marco+Baroni%2E+2018%2E+What+you+can+cram+into+a+single+vector%3A+Probing+sentence+embeddings+for+linguistic+properties%2E+In+Proceedi)
16. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. In Proceedings of the First In-Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI Systems with Sentences that Require Simple Lexical Inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+PASCAL+Recognising+Textual+Entailment+Challenge%2E+In+Proceedings+of+the+First+In%2DMax+Glockner%2C+Vered+Shwartz%2C+and+Yoav+Goldberg%2E+2018%2E+Breaking) [our card](https://md.mova.elamur.ai/free/en/docs/papers/glockner-2018-breaking-nli-systems-sentences-that/)
17. Yoav Goldberg. 2019. Assessing BERT’s syntactic abilities. arXiv preprint arXiv:1901.05287. [↗](https://arxiv.org/abs/1901.05287)
18. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1195–1205. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kristina+Gulordava%2C+Piotr+Bojanowski%2C+Edouard+Grave%2C+Tal+Linzen%2C+and+Marco+Baroni%2E+2018%2E+Colorless+green+recurrent+networks+dream+hierarchically%2E+In+Proceedings+of+the+2018+Conference+of+the+North+Ame)
19. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+Bowman%2C+and+Noah+A%2E+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+Proceedings+of+the+2018+Conference+o)
20. Juho Kim, Christopher Malon, and Asim Kadav. 2018. Teaching syntax by adversarial distraction. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 79–84. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Juho+Kim%2C+Christopher+Malon%2C+and+Asim+Kadav%2E+2018%2E+Teaching+syntax+by+adversarial+distraction%2E+In+Proceedings+of+the+First+Workshop+on+Fact+Extraction+and+VERification+%28FEVER%29%2C+pages+79%E2%80%9384%2E+Associatio)
21. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dan+Klein+and+Christopher+D%2E+Manning%2E+2003%2E+Accurate+unlexicalized+parsing%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2003-klein-accurate-unlexicalized-parsing/)
22. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. [find](https://scholar.google.com/scholar?q=Tal+Linzen%2C+Emmanuel+Dupoux%2C+and+Yoav+Goldberg%2E+2016%2E+Assessing+the+ability+of+LSTMs+to+learn+syntax%2Dsensitive+dependencies%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+4%3A521%E2%80%93+535%2E)
23. Bill MacCartney and Christopher D Manning. 2009. Natural language inference. Ph.D. thesis, Stanford University. [find](https://scholar.google.com/scholar?q=Bill+MacCartney+and+Christopher+D+Manning%2E+2009%2E+Natural+language+inference%2E+Ph%2ED%2E+thesis%2C+Stanford+University%2E)
24. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1192%E2%80%931202%2E+Assoc)
25. R. Thomas McCoy, Robert Frank, and Tal Linzen. 2018. Revisiting the poverty of the stimulus: Hierarchical generalization without a hierarchical bias in recurrent neural networks. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 2093–2098, Madison, WI. [find](https://scholar.google.com/scholar?q=R%2E+Thomas+McCoy%2C+Robert+Frank%2C+and+Tal+Linzen%2E+2018%2E+Revisiting+the+poverty+of+the+stimulus%3A+Hierarchical+generalization+without+a+hierarchical+bias+in+recurrent+neural+networks%2E+In+Proceedings+of+the)
26. R. Thomas McCoy and Tal Linzen. 2019. Non-entailed subsequences as a challenge for natural language inference. In Proceedings of the Society for Computation in Linguistics, volume 2. [find](https://scholar.google.com/scholar?q=R%2E+Thomas+McCoy+and+Tal+Linzen%2E+2019%2E+Non%2Dentailed+subsequences+as+a+challenge+for+natural+language+inference%2E+In+Proceedings+of+the+Society+for+Computation+in+Linguistics%2C+volume+2%2E)
27. R. Thomas McCoy, Tal Linzen, Ewan Dunbar, and Paul Smolensky. 2019. RNNs implicitly implement tensor-product representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=R%2E+Thomas+McCoy%2C+Tal+Linzen%2C+Ewan+Dunbar%2C+and+Paul+Smolensky%2E+2019%2E+RNNs+implicitly+implement+tensor%2Dproduct+representations%2E+In+International+Conference+on+Learning+Representations%2E)
28. Yashar Mehdad, Alessandro Moschitti, and Fabio Massimo Zanzotto. 2010. Syntactic/semantic structures for textual entailment recognition. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 1020–1028. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yashar+Mehdad%2C+Alessandro+Moschitti%2C+and+Fabio+Massimo+Zanzotto%2E+2010%2E+Syntactic%2Fsemantic+structures+for+textual+entailment+recognition%2E+In+Human+Language+Technologies%3A+The+2010+Annual+Conference+of+t)
29. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aakanksha+Naik%2C+Abhilasha+Ravichander%2C+Norman+Sadeh%2C+Carolyn+Rose%2C+and+Graham+Neubig%2E+2018%2E+Stress+test+evaluation+for+natural+language+inference%2E+In+Proceedings+of+the+27th+International+Conference+o)
30. Nikita Nangia and Samuel R. Bowman. 2019. Human vs. muppet: A conservative estimate of human performance on the GLUE benchmark. [find](https://scholar.google.com/scholar?q=Nikita+Nangia+and+Samuel+R%2E+Bowman%2E+2019%2E+Human+vs%2E+muppet%3A+A+conservative+estimate+of+human+performance+on+the+GLUE+benchmark%2E)
31. Yixin Nie, Yicheng Wang, and Mohit Bansal. 2018. Analyzing compositionality-sensitivity of NLI models. arXiv preprint arXiv:1811.07033. [↗](https://arxiv.org/abs/1811.07033)
32. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249–2255. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ankur+Parikh%2C+Oscar+Ta%CC%88ckstro%CC%88m%2C+Dipanjan+Das%2C+and+Jakob+Uszkoreit%2E+2016%2E+A+decomposable+attention+model+for+natural+language+inference%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+N)
33. Ellie Pavlick and Chris Callison-Burch. 2016. Tense manages to predict implicative behavior in verbs. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2225–2229. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ellie+Pavlick+and+Chris+Callison%2DBurch%2E+2016%2E+Tense+manages+to+predict+implicative+behavior+in+verbs%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+2)
34. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018a. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 67–81. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+J%2E+Edward+Hu%2C+Ellie+Pavlick%2C+Aaron+Steven+White%2C+and+Benjamin+Van+Durme%2E+2018a%2E+Collecting+diverse+natural+language+inference+problems+for+sentence+repr)
35. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018b. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Jason+Naradowsky%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+and+Benjamin+Van+Durme%2E+2018b%2E+Hypothesis+only+baselines+in+natural+language+inference%2E+In+Proceedings+of+the+Seventh+Joint+Conference)
36. Laura Rimell and Stephen Clark. 2010. Cambridge: Parser evaluation using textual entailment by grammatical relation comparison. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 268–271. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Laura+Rimell+and+Stephen+Clark%2E+2010%2E+Cambridge%3A+Parser+evaluation+using+textual+entailment+by+grammatical+relation+comparison%2E+In+Proceedings+of+the+5th+International+Workshop+on+Semantic+Evaluation%2C)
37. Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 731–744. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Aaron+Steven+White%2C+and+Benjamin+Van+Durme%2E+2018%2E+Neural+models+of+factuality%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational) [our card](https://md.mova.elamur.ai/free/en/docs/papers/rudinger-2018-neural-models-factuality/)
38. Ivan Sanchez, Jeff Mitchell, and Sebastian Riedel. 2018. Behavior analysis of NLI models: Uncovering the influence of three factors on robustness. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1975–1985. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ivan+Sanchez%2C+Jeff+Mitchell%2C+and+Sebastian+Riedel%2E+2018%2E+Behavior+analysis+of+NLI+models%3A+Uncovering+the+influence+of+three+factors+on+robustness%2E+In+Proceedings+of+the+2018+Conference+of+the+North+Am)
39. Whitney Tabor, Bruno Galantucci, and Daniel Richardson. 2004. Effects of merely local syntactic coherence on sentence processing. Journal of Memory and Language, 50(4):355–370. [find](https://scholar.google.com/scholar?q=Whitney+Tabor%2C+Bruno+Galantucci%2C+and+Daniel+Richardson%2E+2004%2E+Effects+of+merely+local+syntactic+coherence+on+sentence+processing%2E+Journal+of+Memory+and+Language%2C+50%284%29%3A355%E2%80%93370%2E)
40. Jianyu Wang, Zhishuai Zhang, Cihang Xie, Yuyin Zhou, Vittal Premachandran, Jun Zhu, Lingxi Xie, and Alan Yuille. 2018. Visual concepts and compositional voting. Annals of Mathematical Sciences and Applications, 3(1):151–188. [find](https://scholar.google.com/scholar?q=Jianyu+Wang%2C+Zhishuai+Zhang%2C+Cihang+Xie%2C+Yuyin+Zhou%2C+Vittal+Premachandran%2C+Jun+Zhu%2C+Lingxi+Xie%2C+and+Alan+Yuille%2E+2018%2E+Visual+concepts+and+compositional+voting%2E+Annals+of+Mathematical+Sciences+and+App)
41. Noah Weber, Leena Shekhar, and Niranjan Balasubramanian. 2018. The fine line between linguistic generalization and failure in seq2seq-attention models. In Proceedings of the Workshop on Generalization in the Age of Deep Learning, pages 24–27. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Noah+Weber%2C+Leena+Shekhar%2C+and+Niranjan+Balasubramanian%2E+2018%2E+The+fine+line+between+linguistic+generalization+and+failure+in+seq2seq%2Dattention+models%2E+In+Proceedings+of+the+Workshop+on+Generalization)
42. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 996–1005. Asian Federation of Natural Language Processing. [find](https://scholar.google.com/scholar?q=Aaron+Steven+White%2C+Pushpendre+Rastogi%2C+Kevin+Duh%2C+and+Benjamin+Van+Durme%2E+2017%2E+Inference+is+everything%3A+Recasting+semantic+resources+into+a+unified+evaluation+framework%2E+In+Proceedings+of+the+Eighth) [our card](https://md.mova.elamur.ai/free/en/docs/papers/white-2017-inference-is-everything-recasting-semantic/)
43. Aaron Steven White and Kyle Rawlins. 2018. The role of veridicality and factivity in clause selection. In Proceedings of the 48th Annual Meeting of the North East Linguistic Society. [find](https://scholar.google.com/scholar?q=Aaron+Steven+White+and+Kyle+Rawlins%2E+2018%2E+The+role+of+veridicality+and+factivity+in+clause+selection%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+North+East+Linguistic+Society%2E)
44. Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4717–4724. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aaron+Steven+White%2C+Rachel+Rudinger%2C+Kyle+Rawlins%2C+and+Benjamin+Van+Durme%2E+2018%2E+Lexicosyntactic+inference+in+neural+models%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Langu)
45. Adina Williams, Andrew Drozdov, and Samuel R. Bowman. 2018a. Do latent tree learning models identify meaningful structure in sentences? Transactions of the Association of Computational Linguistics, 6:253–267. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Andrew+Drozdov%2C+and+Samuel+R%2E+Bowman%2E+2018a%2E+Do+latent+tree+learning+models+identify+meaningful+structure+in+sentences%3F+Transactions+of+the+Association+of+Computational+Linguistics%2C+6%3A)
46. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018b. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018b%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Cha)