[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/gardner-2020-evaluating-models-local-decision-boundaries/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / Evaluating Models' Local Decision Boundaries via Contrast Sets

# Evaluating Models' Local Decision Boundaries via Contrast Sets

**Authors:** [Matt Gardner](https://md.mova.elamur.ai/free/en/docs/papers/authors/matt-gardner/), [Yoav Artzi](https://md.mova.elamur.ai/free/en/docs/papers/authors/yoav-artzi/), Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, [Pradeep Dasigi](https://md.mova.elamur.ai/free/en/docs/papers/authors/pradeep-dasigi/), [Dheeru Dua](https://md.mova.elamur.ai/free/en/docs/papers/authors/dheeru-dua/), Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, [Hannaneh Hajishirzi](https://md.mova.elamur.ai/free/en/docs/papers/authors/hannaneh-hajishirzi/), Gabriel Ilharco, [Daniel Khashabi](https://md.mova.elamur.ai/free/en/docs/papers/authors/daniel-khashabi/), Kevin Lin, [Jiangming Liu](https://md.mova.elamur.ai/free/en/docs/papers/authors/jiangming-liu/), [Nelson F. Liu](https://md.mova.elamur.ai/free/en/docs/papers/authors/nelson-f-liu/), Phoebe Mulcaire, [Qiang Ning](https://md.mova.elamur.ai/free/en/docs/papers/authors/qiang-ning/), [Sameer Singh](https://md.mova.elamur.ai/free/en/docs/papers/authors/sameer-singh/), [Noah A. Smith](https://md.mova.elamur.ai/free/en/docs/papers/authors/noah-a-smith/), Sanjay Subramanian, [Reut Tsarfaty](https://md.mova.elamur.ai/free/en/docs/papers/authors/reut-tsarfaty/), Eric Wallace, [Ally Zhang](https://md.mova.elamur.ai/free/en/docs/papers/authors/ally-zhang/), Ben Zhou · **Year:** 2020 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2020 **Link:** [aclanthology.org/2020.findings-emnlp.117](https://aclanthology.org/2020.findings-emnlp.117/) (ACL Anthology 2020.findings-emnlp.117; DOI [10.18653/v1/2020.findings-emnlp.117](https://doi.org/10.18653/v1/2020.findings-emnlp.117)) **License of the paper:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — [creativecommons.org/licenses/by/4.0](https://creativecommons.org/licenses/by/4.0/)

[**260** citations](https://openalex.org/works?filter=cites:W3105928338) openalex · 2026-10-19[**264** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/gardner-2020-evaluating-models-local-decision-boundaries/) of the 140673 papers whose bibliographies we read[Findings of the Association for Computational Linguistics: EMNLP 2020](https://aclanthology.org/volumes/2020.findings-emnlp/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2020.findings-emnlp.117/)
- [PDF](https://aclanthology.org/2020.findings-emnlp.117.pdf)
- [DOI 10.18653/v1/2020.findings-emnlp.117](https://doi.org/10.18653/v1/2020.findings-emnlp.117)
- [arXiv 2004.02709](https://arxiv.org/abs/2004.02709)
- [OpenAlex](https://openalex.org/W3105928338)

**Authors (26)**

1. [Matt Gardner](https://openalex.org/A5035088083) [ORCID](https://orcid.org/0000-0001-8458-1727) · [Semantic Scholar](https://www.semanticscholar.org/author/40642935) · h 42 · 21027 citations · 136 works
2. [Yoav Artzi](https://openalex.org/A5047026141) [ORCID](https://orcid.org/0000-0002-4605-6144) · [Semantic Scholar](https://www.semanticscholar.org/author/3167681) · h 34 · 8344 citations · 119 works
3. [Victoria Basmov](https://openalex.org/A5015245341) [Semantic Scholar](https://www.semanticscholar.org/author/134827613) · h 3 · 268 citations · 6 works
4. [Jonathan Berant](https://openalex.org/A5045872048) [Semantic Scholar](https://www.semanticscholar.org/author/1750652) · h 48 · 11485 citations · 219 works
5. [Ben Bogin](https://openalex.org/A5051169643) [Semantic Scholar](https://www.semanticscholar.org/author/50757607) · h 17 · 1523 citations · 47 works
6. [Sihao Chen](https://openalex.org/A5101823158) [ORCID](https://orcid.org/0000-0001-6401-9366) · [Semantic Scholar](https://www.semanticscholar.org/author/2087205) · h 16 · 1540 citations · 89 works
7. [Pradeep Dasigi](https://openalex.org/A5029074038) [ORCID](https://orcid.org/0000-0001-7127-1316) · [Semantic Scholar](https://www.semanticscholar.org/author/2697425) · h 20 · 3446 citations · 90 works
8. [Dheeru Dua](https://openalex.org/A5076846819) [Semantic Scholar](https://www.semanticscholar.org/author/33546336) · h 12 · 1697 citations · 40 works
9. [Yanai Elazar](https://openalex.org/A5081462507) [Semantic Scholar](https://www.semanticscholar.org/author/51131518) · h 22 · 2422 citations · 94 works
10. [Ananth Gottumukkala](https://openalex.org/A5044422816) [Semantic Scholar](https://www.semanticscholar.org/author/1471885977) · h 6 · 392 citations · 6 works
11. [Nitish Gupta](https://openalex.org/A5102347449) [Semantic Scholar](https://www.semanticscholar.org/author/2285178) · h 12 · 904 citations · 21 works
12. [Hannaneh Hajishirzi](https://openalex.org/A5082305994) [ORCID](https://orcid.org/0000-0002-1055-6657) · [Semantic Scholar](https://www.semanticscholar.org/author/2548384) · h 63 · 18822 citations · 287 works
13. [Gabriel Ilharco](https://openalex.org/A5068360032) [Semantic Scholar](https://www.semanticscholar.org/author/2123694087) · h 22 · 3585 citations · 39 works
14. [Daniel Khashabi](https://openalex.org/A5043628255) [ORCID](https://orcid.org/0009-0009-7664-2230) · [Semantic Scholar](https://www.semanticscholar.org/author/1783281) · h 36 · 5868 citations · 223 works
15. [Kevin Lin](https://openalex.org/A5074764224) [ORCID](https://orcid.org/0000-0002-1236-9847) · [Semantic Scholar](https://www.semanticscholar.org/author/2154184644) · h 76 · 27619 citations · 738 works
16. [Jiangming Liu](https://openalex.org/A5014019150) [ORCID](https://orcid.org/0000-0002-2178-2385) · [Semantic Scholar](https://www.semanticscholar.org/author/2262374) · h 12 · 952 citations · 40 works
17. [Nelson F. Liu](https://openalex.org/A5031981513) [Semantic Scholar](https://www.semanticscholar.org/author/22243769) · h 14 · 4651 citations · 55 works
18. [Phoebe Mulcaire](https://openalex.org/A5019045595) [Semantic Scholar](https://www.semanticscholar.org/author/46244238) · h 8 · 609 citations · 19 works
19. [Qiang Ning](https://openalex.org/A5032597148) [ORCID](https://orcid.org/0000-0001-8321-866X) · [Semantic Scholar](https://www.semanticscholar.org/author/3333257) · h 24 · 2056 citations · 119 works
20. [Sameer Singh](https://openalex.org/A5005779128) [ORCID](https://orcid.org/0000-0003-0621-6323) · [Semantic Scholar](https://www.semanticscholar.org/author/34650964) · h 57 · 40857 citations · 304 works
21. [Noah A. Smith](https://openalex.org/A5088517824) [ORCID](https://orcid.org/0000-0002-2310-6380) · [Semantic Scholar](https://www.semanticscholar.org/author/144365875) · h 94 · 40232 citations · 486 works
22. [Sanjay Subramanian](https://openalex.org/A5020117260) [ORCID](https://orcid.org/0009-0001-1399-9920) · [Semantic Scholar](https://www.semanticscholar.org/author/17097887) · h 18 · 1824 citations · 70 works
23. [Reut Tsarfaty](https://openalex.org/A5063283689) [ORCID](https://orcid.org/0000-0001-6406-6568) · [Semantic Scholar](https://www.semanticscholar.org/author/2799181) · h 24 · 2886 citations · 217 works
24. [Eric Wallace](https://openalex.org/A5110564361) [Semantic Scholar](https://www.semanticscholar.org/author/145217343) · h 26 · 5801 citations · 48 works
25. [Ally Zhang](https://openalex.org/A5019162204) [ORCID](https://orcid.org/0000-0003-0753-4972) · [Semantic Scholar](https://www.semanticscholar.org/author/2153658440) · h 4 · 812 citations · 11 works
26. [Ben Zhou](https://openalex.org/A5020349386) [ORCID](https://orcid.org/0009-0009-1684-3719) · [Semantic Scholar](https://www.semanticscholar.org/author/145360756) · h 12 · 896 citations · 34 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## Summary

Standard test sets come from the same distribution as the training data, so a dataset with systematic gaps or annotation artifacts lets models score well with shallow heuristics. The authors propose that dataset creators add a step after building a dataset: take test instances and manually make small, meaningful edits that usually change the gold label. The group of perturbed variants around one original instance is a "contrast set". It probes whether the model's decision boundary is correct in the immediate neighbourhood of real test data. Edits are written without consulting any model, so they are not tuned to one system's weaknesses. Besides accuracy, a consistency metric records whether a model gets every variant of an instance right. Contrast sets were built for ten NLP datasets, among them NLVR2, DROP, QUOREF, MC-TACO, IMDb, MATRES, PERSPECTRUM and UD English parsing. Although not designed to be adversarial, they caused large drops for strong models, in some cases up to about 25%. The approach is a fairly cheap way to check what a model actually learned.

## How Mova uses it

- `world/src/babi.rs` (`CONTRAST` table and `contrast()`; command `world babi-contrast`): the bAbI story reader is tested on contrast variants of the bAbI tasks. These keep the same stories but swap in new names, places and objects, plus synonym verbs. Some synonyms are already in Mova's first-level verb classes, and some (hurried, abandoned) deliberately are not.
- The goal, as stated in the code, is to show that the reader applies a principle rather than a template, and to expose gaps honestly. In Mova the method serves as a robustness test for a rule-based reader, not as an evaluation of a trained neural model.

## Effectiveness in Mova

Measured (from the architecture status table): on the contrast variants the bAbI reader first scored 74.6%. This exposed parser defects ("attic" tagged ADJ, "Taras" lemmatised as "tara", "Following" and "is" read as verbs) and two vocabulary gaps (hurry, abandon). After these were fixed, the contrast set reached 99.7% with all 20 tasks at 95% or higher, and the original test did not drop (99.8%). The method found real bugs that the in-distribution test (already 99.8%) could not show.

---

👨‍🔬💥

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2020.findings-emnlp.117](https://doi.org/10.18653/v1/2020.findings-emnlp.117) · [arXiv 2004.02709](https://arxiv.org/abs/2004.02709)

```
@inproceedings{gardner2020evaluating,
  title = {{Evaluating Models' Local Decision Boundaries via Contrast Sets}},
  author = {Gardner, Matt and Artzi, Yoav and Basmov, Victoria and Berant, Jonathan and Bogin, Ben and Chen, Sihao and Dasigi, Pradeep and Dua, Dheeru and Elazar, Yanai and Gottumukkala, Ananth and Gupta, Nitish and Hajishirzi, Hannaneh and Ilharco, Gabriel and Khashabi, Daniel and Lin, Kevin and Liu, Jiangming and Liu, Nelson F. and Mulcaire, Phoebe and Ning, Qiang and Singh, Sameer and Smith, Noah A. and Subramanian, Sanjay and Tsarfaty, Reut and Wallace, Eric and Zhang, Ally and Zhou, Ben},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.117},
  eprint = {2004.02709},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.117}
}
```

- [← previous Assessing the Agreement Competence of Large Language Models](https://md.mova.elamur.ai/free/en/docs/papers/garcia-2025-assessing-agreement-competence-large-language/)
- [next → LLM Dependency Parsing with In-Context Rules](https://md.mova.elamur.ai/free/en/docs/papers/ginn-2025-llm-dependency-parsing-context-rules/)
- [↑ back to papers](https://md.mova.elamur.ai/free/en/docs/papers/)

## References

77 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lars Ahrenberg. 2007. LinES: an English-Swedish parallel treebank. In NODALIDA. [find](https://scholar.google.com/scholar?q=Lars+Ahrenberg%2E+2007%2E+LinES%3A+an+English%2DSwedish+parallel+treebank%2E+In+NODALIDA%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-2007-ahrenberg-lines-english-swedish-parallel-treebank/)
2. Jacob Andreas. 2020. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7556–7566, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jacob+Andreas%2E+2020%2E+Good%2Denough+compositional+data+augmentation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+7556%E2%80%937566%2C+Online%2E+Association+for+C)
3. Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine Learning. [find](https://scholar.google.com/scholar?q=Shai+Ben%2DDavid%2C+John+Blitzer%2C+Koby+Crammer%2C+Alex+Kulesza%2C+Fernando+Pereira%2C+and+Jennifer+Wortman+Vaughan%2E+2010%2E+A+theory+of+learning+from+different+domains%2E+Machine+Learning%2E)
4. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+EMNLP%2E)
5. Franck Burlot, Yves Scherrer, Vinit Ravishankar, Ondřej Bojar, Stig-Arne Grönroos, Maarit Koponen, Tommi Nieminen, and François Yvon. 2018. The WMT’18 morpheval test suites for English-Czech, English-German, English-Finnish and Turkish-English. In Proceedings of the Third Conference on Machine Tr… [find](https://scholar.google.com/scholar?q=Franck+Burlot%2C+Yves+Scherrer%2C+Vinit+Ravishankar%2C+Ondr%CC%8Cej+Bojar%2C+Stig%2DArne+Gro%CC%88nroos%2C+Maarit+Koponen%2C+Tommi+Nieminen%2C+and+Franc%CC%A7ois+Yvon%2E+2018%2E+The+WMT%E2%80%9918+morpheval+test+suites+for+English%2DCzech%2C+Engli)
6. Franck Burlot and François Yvon. 2017. Evaluating the morphological competence of machine translation systems. In Proceedings of the Second Conference on Machine Translation, pages 43–55, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Franck+Burlot+and+Franc%CC%A7ois+Yvon%2E+2017%2E+Evaluating+the+morphological+competence+of+machine+translation+systems%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+pages+43%E2%80%9355%2C+Copenhagen%2C)
7. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/Daily Mail reading comprehension task. In ACL. [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Jason+Bolton%2C+and+Christopher+D%2E+Manning%2E+2016%2E+A+thorough+examination+of+the+CNN%2FDaily+Mail+reading+comprehension+task%2E+In+ACL%2E)
8. Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. Seeing things from a different angle: Discovering diverse perspectives about claims. In NAACL. [find](https://scholar.google.com/scholar?q=Sihao+Chen%2C+Daniel+Khashabi%2C+Wenpeng+Yin%2C+Chris+Callison%2DBurch%2C+and+Dan+Roth%2E+2019%2E+Seeing+things+from+a+different+angle%3A+Discovering+diverse+perspectives+about+claims%2E+In+NAACL%2E)
9. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural Yes/No questions. In NAACL. [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+2019%2E+BoolQ%3A+Exploring+the+surprising+difficulty+of+natural+Yes%2FNo+questions%2E+In+NAACL%2E)
10. Michael Collins and James Brooks. 1995. Prepositional phrase attachment through a backed-off model. In Third Workshop on Very Large Corpora. [find](https://scholar.google.com/scholar?q=Michael+Collins+and+James+Brooks%2E+1995%2E+Prepositional+phrase+attachment+through+a+backed%2Doff+model%2E+In+Third+Workshop+on+Very+Large+Corpora%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-1995-collins-prepositional-phrase-attachment-through-backed/)
11. Pradeep Dasigi, Nelson F Liu, Ana Marasovic, Noah A Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In EMNLP. [find](https://scholar.google.com/scholar?q=Pradeep+Dasigi%2C+Nelson+F+Liu%2C+Ana+Marasovic%2C+Noah+A+Smith%2C+and+Matt+Gardner%2E+2019%2E+Quoref%3A+A+reading+comprehension+dataset+with+questions+requiring+coreferential+reasoning%2E+In+EMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/dasigi-2019-quoref-reading-comprehension-dataset-questions/)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2E)
13. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In ACM AIES. [find](https://scholar.google.com/scholar?q=Lucas+Dixon%2C+John+Li%2C+Jeffrey+Sorensen%2C+Nithum+Thain%2C+and+Lucy+Vasserman%2E+2018%2E+Measuring+and+mitigating+unintended+bias+in+text+classification%2E+In+ACM+AIES%2E)
14. Timothy Dozat and Christopher D Manning. 2017. Deep biaffine attention for neural dependency parsing. In ICLR. [find](https://scholar.google.com/scholar?q=Timothy+Dozat+and+Christopher+D+Manning%2E+2017%2E+Deep+biaffine+attention+for+neural+dependency+parsing%2E+In+ICLR%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/missing-2016-dozat-deep-biaffine-attention-neural-dependency/)
15. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL. [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Yizhong+Wang%2C+Pradeep+Dasigi%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2019%2E+DROP%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+In+NAACL%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
16. Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In EMNLP. [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+Alvin+Grissom+II%2C+Mohit+Iyyer%2C+Pedro+Rodriguez%2C+and+Jordan+Boyd%2DGraber%2E+2018%2E+Pathologies+of+neural+models+make+interpretations+difficult%2E+In+EMNLP%2E)
17. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets. In EMNLP. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Yoav+Goldberg%2C+and+Jonathan+Berant%2E+2019%2E+Are+we+modeling+the+task+or+the+annotator%3F+An+investigation+of+annotator+bias+in+natural+language+understanding+datasets%2E+In+EMNLP%2E)
18. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In ACL. [find](https://scholar.google.com/scholar?q=Max+Glockner%2C+Vered+Shwartz%2C+and+Yoav+Goldberg%2E+2018%2E+Breaking+NLI+systems+with+sentences+that+require+simple+lexical+inferences%2E+In+ACL%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/glockner-2018-breaking-nli-systems-sentences-that/)
19. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In NAACL. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+R%2E+Bowman%2C+and+Noah+A%2E+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+NAACL%2E)
20. Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. 2019. A multi-type multi-span network for reading comprehension that requires discrete reasoning. In EMNLP. [find](https://scholar.google.com/scholar?q=Minghao+Hu%2C+Yuxing+Peng%2C+Zhen+Huang%2C+and+Dongsheng+Li%2E+2019%2E+A+multi%2Dtype+multi%2Dspan+network+for+reading+comprehension+that+requires+discrete+reasoning%2E+In+EMNLP%2E)
21. Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Pierre+Isabelle%2C+Colin+Cherry%2C+and+George+Foster%2E+2017%2E+A+challenge+set+approach+to+evaluating+machine+translation%2E+In+EMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/isabelle-2017-challenge-set-approach-evaluating-machine/)
22. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In EMNLP. [find](https://scholar.google.com/scholar?q=Robin+Jia+and+Percy+Liang%2E+2017%2E+Adversarial+examples+for+evaluating+reading+comprehension+systems%2E+In+EMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
23. Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. 2019. Certified robustness to adversarial word substitutions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL… [find](https://scholar.google.com/scholar?q=Robin+Jia%2C+Aditi+Raghunathan%2C+Kerem+Go%CC%88ksel%2C+and+Percy+Liang%2E+2019%2E+Certified+robustness+to+adversarial+word+substitutions%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Langua)
24. Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In ICLR. [find](https://scholar.google.com/scholar?q=Divyansh+Kaushik%2C+Eduard+Hovy%2C+and+Zachary+C+Lipton%2E+2020%2E+Learning+the+difference+that+makes+a+difference+with+counterfactually%2Daugmented+data%2E+In+ICLR%2E)
25. Daniel Khashabi, Tushar Khot, and Ashish Sabhwawal. 2020. More bang for your buck: Natural perturbation for robust question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Tushar+Khot%2C+and+Ashish+Sabhwawal%2E+2020%2E+More+bang+for+your+buck%3A+Natural+perturbation+for+robust+question+answering%2E+In+EMNLP%2E)
26. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale reading comprehension dataset from examinations. In EMNLP. [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2017%2E+RACE%3A+Large%2Dscale+reading+comprehension+dataset+from+examinations%2E+In+EMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lai-2017-race-large-scale-reading-comprehension/)
27. Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M. Bender. 2017. Towards linguistically generalizable NLP systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, Copenhagen, Denmark. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2C+Sudha+Rao%2C+Hal+Daume%CC%81+III%2C+and+Emily+M%2E+Bender%2E+2017%2E+Towards+linguistically+generalizable+NLP+systems%3A+A+workshop+and+shared+task%2E+In+Proceedings+of+the+First+Workshop+on+Building+L)
28. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2011. The winograd schema challenge. In KR. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2011%2E+The+winograd+schema+challenge%2E+In+KR%2E)
29. Shi Feng, Eric Wallace, and Jordan Boyd-Graber. 2019. Misleading failures of partial-input baselines. In ACL. [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+and+Jordan+Boyd%2DGraber%2E+2019%2E+Misleading+failures+of+partial%2Dinput+baselines%2E+In+ACL%2E)
30. Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. 2018. Detecting and correcting for label shift with black box predictors. In ICML. [find](https://scholar.google.com/scholar?q=Zachary+Lipton%2C+Yu%2DXiang+Wang%2C+and+Alexander+Smola%2E+2018%2E+Detecting+and+correcting+for+label+shift+with+black+box+predictors%2E+In+ICML%2E)
31. Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019. Reasoning over paragraph effects in situations. In EMNLP MRQA Workshop. [find](https://scholar.google.com/scholar?q=Kevin+Lin%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Matt+Gardner%2E+2019%2E+Reasoning+over+paragraph+effects+in+situations%2E+In+EMNLP+MRQA+Workshop%2E)
32. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
33. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Aman-charla, and Anupam Datta. 2018. Gender bias in neural natural language processing. arXiv preprint arXiv:1807.11714. [find](https://scholar.google.com/scholar?q=Kaiji+Lu%2C+Piotr+Mardziel%2C+Fangjing+Wu%2C+Preetam+Aman%2Dcharla%2C+and+Anupam+Datta%2E+2018%2E+Gender+bias+in+neural+natural+language+processing%2E+arXiv+preprint+arXiv%3A1807%2E11714%2E)
34. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In ACL. [find](https://scholar.google.com/scholar?q=Andrew+L%2E+Maas%2C+Raymond+E%2E+Daly%2C+Peter+T%2E+Pham%2C+Dan+Huang%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2011%2E+Learning+word+vectors+for+sentiment+analysis%2E+In+ACL%2E)
35. Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The Penn treebank. In Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mitchell+Marcus%2C+Beatrice+Santorini%2C+and+Mary+Ann+Marcinkiewicz%2E+1993%2E+Building+a+large+annotated+corpus+of+english%3A+The+Penn+treebank%2E+In+Computational+Linguistics%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
36. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In EMNLP. [find](https://scholar.google.com/scholar?q=Rebecca+Marvin+and+Tal+Linzen%2E+2018%2E+Targeted+syntactic+evaluation+of+language+models%2E+In+EMNLP%2E)
37. Paul Michel, Xian Li, Graham Neubig, and Juan Miguel Pino. 2019. On evaluation of adversarial perturbations for sequence-to-sequence models. In NAACL. [find](https://scholar.google.com/scholar?q=Paul+Michel%2C+Xian+Li%2C+Graham+Neubig%2C+and+Juan+Miguel+Pino%2E+2019%2E+On+evaluation+of+adversarial+perturbations+for+sequence%2Dto%2Dsequence+models%2E+In+NAACL%2E)
38. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In ACL. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Eric+Wallace%2C+Sameer+Singh%2C+Matt+Gardner%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2019%2E+Compositional+questions+do+not+necessitate+multi%2Dhop+reasoning%2E+In+ACL%2E)
39. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In COLING. [find](https://scholar.google.com/scholar?q=Aakanksha+Naik%2C+Abhilasha+Ravichander%2C+Norman+Sadeh%2C+Carolyn+Rose%2C+and+Graham+Neubig%2E+2018%2E+Stress+test+evaluation+for+natural+language+inference%2E+In+COLING%2E)
40. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial NLI: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599. [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2019%2E+Adversarial+NLI%3A+A+new+benchmark+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1910%2E14599%2E)
41. Qiang Ning, Sanjay Subramanian, and Dan Roth. 2019. An Improved Neural Baseline for Temporal Relation Extraction. In EMNLP. [find](https://scholar.google.com/scholar?q=Qiang+Ning%2C+Sanjay+Subramanian%2C+and+Dan+Roth%2E+2019%2E+An+Improved+Neural+Baseline+for+Temporal+Relation+Extraction%2E+In+EMNLP%2E)
42. Qiang Ning, Hao Wu, and Dan Roth. 2018. A Multi-Axis Annotation Scheme for Event Temporal Relations. In ACL. [find](https://scholar.google.com/scholar?q=Qiang+Ning%2C+Hao+Wu%2C+and+Dan+Roth%2E+2018%2E+A+Multi%2DAxis+Annotation+Scheme+for+Event+Temporal+Relations%2E+In+ACL%2E)
43. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Timothy+Niven+and+Hung%2DYu+Kao%2E+2019%2E+Probing+neural+network+comprehension+of+natural+language+arguments%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+Flo)
44. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In LREC. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Marie%2DCatherine+de+Marneffe%2C+Filip+Ginter%2C+Yoav+Goldberg%2C+Jan+Hajic%CC%8C%2C+Christopher+D%2E+Manning%2C+Ryan+McDonald%2C+Slav+Petrov%2C+Sampo+Pyysalo%2C+Natalia+Silveira%2C+Reut+Tsarfaty%2C+and+Daniel+Zeman) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2016-nivre-universal-dependencies-v1-multilingual-treebank/)
45. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+NAACL%2E)
46. K.L. Pike. 1946. Phonemics: A Technique for Reducing Languages to Writing. Number v. 2 in Phonemics: A Technique for Reducing Languages to Writing. Summer Institute of Linguistics. [find](https://scholar.google.com/scholar?q=K%2EL%2E+Pike%2E+1946%2E+Phonemics%3A+A+Technique+for+Reducing+Languages+to+Writing%2E+Number+v%2E+2+in+Phonemics%3A+A+Technique+for+Reducing+Languages+to+Writing%2E+Summer+Institute+of+Linguistics%2E)
47. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In *SEM. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Jason+Naradowsky%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+and+Benjamin+Van+Durme%2E+2018%2E+Hypothesis+only+baselines+in+natural+language+inference%2E+In+%2ASEM%2E)
48. Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do ImageNet classifiers generalize to ImageNet? In ICML. [find](https://scholar.google.com/scholar?q=Benjamin+Recht%2C+Rebecca+Roelofs%2C+Ludwig+Schmidt%2C+and+Vaishaal+Shankar%2E+2019%2E+Do+ImageNet+classifiers+generalize+to+ImageNet%3F+In+ICML%2E)
49. Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are red roses red? Evaluating consistency of question-answering models. In ACL. [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Carlos+Guestrin%2C+and+Sameer+Singh%2E+2019%2E+Are+red+roses+red%3F+Evaluating+consistency+of+question%2Danswering+models%2E+In+ACL%2E)
50. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018a. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856–865, Melbourne, Australia. Association for… [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2018a%2E+Semantically+equivalent+adversarial+rules+for+debugging+NLP+models%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Comp)
51. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018b. Semantically equivalent adversarial rules for debugging NLP models. In ACL. [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2018b%2E+Semantically+equivalent+adversarial+rules+for+debugging+NLP+models%2E+In+ACL%2E)
52. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In NAACL. [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Jason+Naradowsky%2C+Brian+Leonard%2C+and+Benjamin+Van+Durme%2E+2018%2E+Gender+bias+in+coreference+resolution%2E+In+NAACL%2E)
53. Manuela Sanguinetti and Cristina Bosco. 2015. Part-TUT: The Turin university parallel treebank. In Harmonization and Development of Resources and Tools for Italian Natural Language Processing within the PARLI Project. [find](https://scholar.google.com/scholar?q=Manuela+Sanguinetti+and+Cristina+Bosco%2E+2015%2E+Part%2DTUT%3A+The+Turin+university+parallel+treebank%2E+In+Harmonization+and+Development+of+Resources+and+Tools+for+Italian+Natural+Language+Processing+within+t)
54. Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, and David L. Dill. 2019. Learning a SAT solver from single-bit supervision. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Daniel+Selsam%2C+Matthew+Lamm%2C+Benedikt+Bu%CC%88nz%2C+Percy+Liang%2C+Leonardo+de+Moura%2C+and+David+L%2E+Dill%2E+2019%2E+Learning+a+SAT+solver+from+single%2Dbit+supervision%2E+In+International+Conference+on+Learning+Represe)
55. Rico Sennrich. 2017. How grammatical is character-level neural machine translation? Assessing MT quality with contrastive translation pairs. In EACL. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2E+2017%2E+How+grammatical+is+character%2Dlevel+neural+machine+translation%3F+Assessing+MT+quality+with+contrastive+translation+pairs%2E+In+EACL%2E)
56. Deven Shah, H. Andrew Schwartz, and Dirk Hovy. 2020. Predictive biases in natural language processing models: A conceptual framework and overview. In ACL. [find](https://scholar.google.com/scholar?q=Deven+Shah%2C+H%2E+Andrew+Schwartz%2C+and+Dirk+Hovy%2E+2020%2E+Predictive+biases+in+natural+language+processing+models%3A+A+conceptual+framework+and+overview%2E+In+ACL%2E)
57. Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. 2017. Foil it! Find One mismatch between image and language caption. In ACL. [find](https://scholar.google.com/scholar?q=Ravi+Shekhar%2C+Sandro+Pezzelle%2C+Yauhen+Klimovich%2C+Aure%CC%81lie+Herbelot%2C+Moin+Nabi%2C+Enver+Sangineto%2C+and+Raffaella+Bernardi%2E+2017%2E+Foil+it%21+Find+One+mismatch+between+image+and+language+caption%2E+In+ACL%2E)
58. Hidetoshi Shimodaira. 2000. Improving predictive inference under covariate shift by weighting the log-likelihood function. In Journal of Statistical Planning and Inference. [find](https://scholar.google.com/scholar?q=Hidetoshi+Shimodaira%2E+2000%2E+Improving+predictive+inference+under+covariate+shift+by+weighting+the+log%2Dlikelihood+function%2E+In+Journal+of+Statistical+Planning+and+Inference%2E)
59. Avi Shmidman, Joshua Guedalia, Shaltiel Shmidman, Moshe Koppel, and Reut Tsarfaty. A novel challenge set for hebrew morphological disambiguation and diacritics restoration. In Findings of EMNLP. [find](https://scholar.google.com/scholar?q=Avi+Shmidman%2C+Joshua+Guedalia%2C+Shaltiel+Shmidman%2C+Moshe+Koppel%2C+and+Reut+Tsarfaty%2E+A+novel+challenge+set+for+hebrew+morphological+disambiguation+and+diacritics+restoration%2E+In+Findings+of+EMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/findings-2020-shmidman-novel-challenge-set-hebrew-morphological/)
60. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014. A gold standard dependency corpus for English. In LREC. [find](https://scholar.google.com/scholar?q=Natalia+Silveira%2C+Timothy+Dozat%2C+Marie%2DCatherine+de+Marneffe%2C+Samuel+Bowman%2C+Miriam+Connor%2C+John+Bauer%2C+and+Chris+Manning%2E+2014%2E+A+gold+standard+dependency+corpus+for+English%2E+In+LREC%2E)
61. Alane Suhr and Yoav Artzi. 2019. NLVR2 visual bias analysis. arXiv preprint arXiv:1909.10411. — [arXiv:1909.10411](https://arxiv.org/abs/1909.10411) [find](https://scholar.google.com/scholar?q=Alane+Suhr+and+Yoav+Artzi%2E+2019%2E+NLVR2+visual+bias+analysis%2E+arXiv+preprint+arXiv%3A1909%2E10411%2E+%E2%80%94+%5BarXiv%3A1909%2E10411%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1909%2E10411%29)
62. Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In ACL. [find](https://scholar.google.com/scholar?q=Alane+Suhr%2C+Stephanie+Zhou%2C+Ally+Zhang%2C+Iris+Zhang%2C+Huajun+Bai%2C+and+Yoav+Artzi%2E+2019%2E+A+corpus+for+reasoning+about+natural+language+grounded+in+photographs%2E+In+ACL%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/suhr-2019-corpus-reasoning-natural-language-grounded/)
63. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In ICLR. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wojciech+Zaremba%2C+Ilya+Sutskever%2C+Joan+Bruna%2C+Dumitru+Erhan%2C+Ian+J%2E+Goodfellow%2C+and+Rob+Fergus%2E+2014%2E+Intriguing+properties+of+neural+networks%2E+In+ICLR%2E)
64. Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. QuaRTz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (E… [find](https://scholar.google.com/scholar?q=Oyvind+Tafjord%2C+Matt+Gardner%2C+Kevin+Lin%2C+and+Peter+Clark%2E+2019%2E+QuaRTz%3A+An+open%2Ddomain+dataset+of+qualitative+relationship+questions%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natu)
65. Chhavi Yadav and Léon Bottou. 2019. Cold case: The lost MNIST digits. In NeurIPS. [find](https://scholar.google.com/scholar?q=Chhavi+Yadav+and+Le%CC%81on+Bottou%2E+2019%2E+Cold+case%3A+The+lost+MNIST+digits%2E+In+NeurIPS%2E)
66. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Ruslan+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+XLNet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+NeurIPS%2E)
67. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. In LREC. [find](https://scholar.google.com/scholar?q=Amir+Zeldes%2E+2017%2E+The+GUM+corpus%3A+Creating+multilayer+resources+in+the+classroom%2E+In+LREC%2E)
68. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In EMNLP. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+2018%2E+SWAG%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+In+EMNLP%2E)
69. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In ACL. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+ACL%2E)
70. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In NAACL. [find](https://scholar.google.com/scholar?q=Jieyu+Zhao%2C+Tianlu+Wang%2C+Mark+Yatskar%2C+Vicente+Ordonez%2C+and+Kai%2DWei+Chang%2E+2018%2E+Gender+bias+in+coreference+resolution%3A+Evaluation+and+debiasing+methods%2E+In+NAACL%2E)
71. Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. “Going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding. In EMNLP. [find](https://scholar.google.com/scholar?q=Ben+Zhou%2C+Daniel+Khashabi%2C+Qiang+Ning%2C+and+Dan+Roth%2E+2019%2E+%E2%80%9CGoing+on+a+vacation%E2%80%9D+takes+longer+than+%E2%80%9Cgoing+for+a+walk%E2%80%9D%3A+A+study+of+temporal+commonsense+understanding%2E+In+EMNLP%2E)
72. Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In EMNLP. [find](https://scholar.google.com/scholar?q=Hao+Tan+and+Mohit+Bansal%2E+2019%2E+LXMERT%3A+Learning+cross%2Dmodality+encoder+representations+from+transformers%2E+In+EMNLP%2E)
73. Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. SemEval-2013 Task 1: TempEval-3: Evaluating time expressions, events, and temporal relations. In *SEM. [find](https://scholar.google.com/scholar?q=Naushad+UzZaman%2C+Hector+Llorens%2C+Leon+Derczynski%2C+James+Allen%2C+Marc+Verhagen%2C+and+James+Pustejovsky%2E+2013%2E+SemEval%2D2013+Task+1%3A+TempEval%2D3%3A+Evaluating+time+expressions%2C+events%2C+and+temporal+relations%2E)
74. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019a. Universal adversarial triggers for attacking and analyzing NLP. In EMNLP. [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Shi+Feng%2C+Nikhil+Kandpal%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+2019a%2E+Universal+adversarial+triggers+for+attacking+and+analyzing+NLP%2E+In+EMNLP%2E)
75. Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019b. Trick me if you can: Human-in-the-loop generation of adversarial question answering examples. In TACL. [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Pedro+Rodriguez%2C+Shi+Feng%2C+Ikuya+Yamada%2C+and+Jordan+Boyd%2DGraber%2E+2019b%2E+Trick+me+if+you+can%3A+Human%2Din%2Dthe%2Dloop+generation+of+adversarial+question+answering+examples%2E+In+TACL%2E)
76. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2019. BLiMP: A benchmark of linguistic minimal pairs for english. arXiv preprint arXiv:1912.00582. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Alicia+Parrish%2C+Haokun+Liu%2C+Anhad+Mohananey%2C+Wei+Peng%2C+Sheng%2DFu+Wang%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+BLiMP%3A+A+benchmark+of+linguistic+minimal+pairs+for+english%2E+arXiv+preprint+arXiv%3A1912%2E0)
77. Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2019. Errudite: Scalable, reproducible, and testable error analysis. In ACL. [find](https://scholar.google.com/scholar?q=Tongshuang+Wu%2C+Marco+Tulio+Ribeiro%2C+Jeffrey+Heer%2C+and+Daniel+Weld%2E+2019%2E+Errudite%3A+Scalable%2C+reproducible%2C+and+testable+error+analysis%2E+In+ACL%2E)