[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/schmitt-2019-sherliic-typed-event-focused-lexical/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / SherLIiC: A Typed Event-Focused Lexical Inference Benchmark for Evaluating Natural Language Inference

# SherLIiC: A Typed Event-Focused Lexical Inference Benchmark for Evaluating Natural Language Inference

**Authors:** Martin Schmitt and [Hinrich Schütze](https://md.mova.elamur.ai/free/en/docs/papers/authors/hinrich-schutze/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics **Link:** [aclanthology.org/P19-1086](https://aclanthology.org/P19-1086/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology). The relations come from ClueWeb09 and Freebase, whose terms were not checked here; use for evaluation only until checked

[**18** citations](https://openalex.org/works?filter=cites:W2948691361) openalex-snapshot · 2026-10-14[**10** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/schmitt-2019-sherliic-typed-event-focused-lexical/) of the 140673 papers whose bibliographies we read[Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics](https://aclanthology.org/volumes/P19-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/P19-1086/)
- [PDF](https://aclanthology.org/P19-1086.pdf)
- [DOI 10.18653/v1/p19-1086](https://doi.org/10.18653/v1/p19-1086)
- [OpenAlex](https://openalex.org/W2948691361)

**Authors (2)**

1. Martin Schmitt
2. [Hinrich Schütze](https://openalex.org/A5071144367) [ORCID](https://orcid.org/0000-0001-9514-7934) · h 63 · 51036 citations · 501 works

Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart.

## What it is about

SherLIiC is a benchmark for lexical inference in context. Each item is a candidate rule, "A is granting to B" implies "A is giving to B", where A and B are typed argument slots such as ORGANIZATION FOUNDER and EMPLOYER. The relations come from verbal paths between Freebase entities in the ClueWeb09 web corpus, grouped by the most frequent Freebase types in each argument slot. The authors kept candidates with strong distributional evidence, so that distribution alone is a weak guide. There are 3,985 manually annotated candidates, labelled yes or no by Mechanical Turk workers (at least five per item; workers with trust below 0.8 were replaced). Of the candidates, 33% are yes and 67% no. Workers agreed unanimously on 53%, and the majority label is the gold label. The data are split 25:75 into dev and test. Rule bases such as PPDB, Patty and Schoenmackers have recall from 0.119 to 0.308, and their union reaches 0.483 on dev and 0.493 on test, so most valid candidates are missing from existing rule bases. The supervised NLI model ESIM, trained on MultiNLI, hardly improves on predicting "entailment" for every item. The paper's best-performing baseline uses types.

## Ideas

- I1. `prag/src/roles.rs` and `world/src/events.rs`: give each event argument a type signature, as the paper does with the most frequent types per slot, and let a rule A implies B fire only when the two type signatures overlap. The paper's example, "run" with PERSON/COMPANY versus COMPUTER/SOFTWARE, is a test case.
- I2. Sherlock-style scores as counts in BTreeMap order: relevance P(B given A) over P(B), a significance statistic, and an entity-support ratio that penalises rules whose overlap comes from a few frequent entities.
- I3. Build an item set where the "always yes" and distributional baselines are near chance, and require a Mova rule to beat "always yes" on held-out items. A rule that only matches the overlap of arguments should fail, which is a useful negative control.
- I4. Simultaneity: the paper's yes means "true at the same time as the premise". Mova's `world/src/qa.rs` should mark whether a rule's hypothesis is simultaneous, earlier or later, so that a yes is not given for a temporal shift.

## For Mova

The chapters are [19. The lexicon: vocabulary, multiword expressions and register](https://md.mova.elamur.ai/free/en/docs/languages/en/19-lexicon-mwe-register/) (lexical relations) and [21. The simple clause](https://md.mova.elamur.ai/free/en/docs/languages/en/21-clause-structure/) (predicate-argument structure). The code is `prag/src/roles.rs` for argument roles, `world/src/events.rs` for event records and `en/src/dict.rs` for types. The measure: precision and recall on held-out candidates against the "always yes" baseline, run twice, with the typed and untyped variants as the ablation. The data are for evaluation only; the Freebase and ClueWeb licences must be checked before any release.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/p19-1086](https://doi.org/10.18653/v1/p19-1086)

```
@inproceedings{schmitt2019sherliic,
  title = {{SherLIiC: A Typed Event-Focused Lexical Inference Benchmark for Evaluating Natural Language Inference}},
  author = {Schmitt, Martin and Schütze, Hinrich},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1086},
  url = {https://doi.org/10.18653/v1/p19-1086}
}
```

- [← previous The semantic transparency of English compound nouns](https://md.mova.elamur.ai/free/en/docs/papers/schafer-2018-semantic-transparency-english-compound/)
- [next → Adposition and Case Supersenses v2.6: Guidelines for English](https://md.mova.elamur.ai/free/en/docs/papers/schneider-2017-adposition-case-supersenses-v2-6/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

54 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Moustafa+Alzantot%2C+Yash+Sharma%2C+Ahmed+Elgohary%2C+Bo%2DJhang+Ho%2C+Mani+Srivastava%2C+and+Kai%2DWei+Chang%2E+2018%2E+Generating+natural+language+adversarial+examples%2E+In+Proceedings+of+the+2018+Conference+on+Empiri)
2. Michele Banko, Michael J Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Procs. of IJCAI, volume 7, pages 2670–2676. [find](https://scholar.google.com/scholar?q=Michele+Banko%2C+Michael+J+Cafarella%2C+Stephen+Soderland%2C+Matthew+Broadhead%2C+and+Oren+Etzioni%2E+2007%2E+Open+information+extraction+from+the+web%2E+In+Procs%2E+of+IJCAI%2C+volume+7%2C+pages+2670%E2%80%932676%2E)
3. Jonathan Berant. 2012. Global Learning of Textual Entailment Graphs. Ph.D. thesis, The Blavatnik School of Computer Science, Raymond and Beverly Sackler Faculty of Exact Sciences, Tel Aviv University. [find](https://scholar.google.com/scholar?q=Jonathan+Berant%2E+2012%2E+Global+Learning+of+Textual+Entailment+Graphs%2E+Ph%2ED%2E+thesis%2C+The+Blavatnik+School+of+Computer+Science%2C+Raymond+and+Beverly+Sackler+Faculty+of+Exact+Sciences%2C+Tel+Aviv+University%2E)
4. Jonathan Berant, Ido Dagan, and Jacob Goldberger. 2011. Global learning of typed entailment rules. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 610– 619, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jonathan+Berant%2C+Ido+Dagan%2C+and+Jacob+Goldberger%2E+2011%2E+Global+learning+of+typed+entailment+rules%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Lan)
5. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, pages 1247–1250, New York, NY, USA. [find](https://scholar.google.com/scholar?q=Kurt+Bollacker%2C+Colin+Evans%2C+Praveen+Paritosh%2C+Tim+Sturge%2C+and+Jamie+Taylor%2E+2008%2E+Freebase%3A+A+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+Proceedings+of+the+2008+ACM+SI)
6. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 2787–2795. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Antoine+Bordes%2C+Nicolas+Usunier%2C+Alberto+Garcia%2DDuran%2C+Jason+Weston%2C+and+Oksana+Yakhnenko%2E+2013%2E+Translating+embeddings+for+modeling+multi%2Drelational+data%2E+In+C%2E+J%2E+C%2E+Burges%2C+L%2E+Bottou%2C+M%2E+Welling%2C+Z)
7. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1508.05326)
8. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+lstm+for+natural+language+inference%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computa)
9. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Morgan & Claypool Publishers. [↗](https://doi.org/10.2200/s00509ed1v01y201305hlt023)
10. Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545, Edinburgh, Scotland, UK. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anthony+Fader%2C+Stephen+Soderland%2C+and+Oren+Etzioni%2E+2011%2E+Identifying+relations+for+open+information+extraction%2E+In+Proceedings+of+the+2011+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
11. Christiane Fellbaum. 2005. Wordnet and wordnets. In Keith Brown et al., editor, Encyclopedia of Language and Linguistics, second edition, pages 665– 670. Elsevier, Oxford. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2E+2005%2E+Wordnet+and+wordnets%2E+In+Keith+Brown+et+al%2E%2C+editor%2C+Encyclopedia+of+Language+and+Linguistics%2C+second+edition%2C+pages+665%E2%80%93+670%2E+Elsevier%2C+Oxford%2E)
12. Benoı̂t Frénay and Michel Verleysen. 2014. Classification in the Presence of Label Noise: A Survey. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 25(5):845–869. [↗](https://doi.org/10.1109/tnnls.2013.2292894)
13. Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. 2013. FACC1: Freebase annotation of ClueWeb corpora, Version 1 (Release date 2013-06-26, Format version 1, Correction level 0). [find](https://scholar.google.com/scholar?q=Evgeniy+Gabrilovich%2C+Michael+Ringgaard%2C+and+Amarnag+Subramanya%2E+2013%2E+FACC1%3A+Freebase+annotation+of+ClueWeb+corpora%2C+Version+1+%28Release+date+2013%2D06%2D26%2C+Format+version+1%2C+Correction+level+0%29%2E)
14. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 758–764, Atlanta, Georgia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Juri+Ganitkevitch%2C+Benjamin+Van+Durme%2C+and+Chris+Callison%2DBurch%2E+2013%2E+PPDB%3A+The+paraphrase+database%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Association+for+Computa) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ganitkevitch-2013-ppdb-paraphrase-database/)
15. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In The 56th Annual Meeting of the Association for Computational Linguistics (ACL), Melbourne, Australia. [find](https://scholar.google.com/scholar?q=Max+Glockner%2C+Vered+Shwartz%2C+and+Yoav+Goldberg%2E+2018%2E+Breaking+NLI+systems+with+sentences+that+require+simple+lexical+inferences%2E+In+The+56th+Annual+Meeting+of+the+Association+for+Computational+Lingui) [our card](https://md.mova.elamur.ai/free/en/docs/papers/glockner-2018-breaking-nli-systems-sentences-that/)
16. Edouard Grave, Tomas Mikolov, Armand Joulin, and Piotr Bojanowski. 2017. Bag of tricks for efficient text classification. In EACL (2), pages 427–431. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Edouard+Grave%2C+Tomas+Mikolov%2C+Armand+Joulin%2C+and+Piotr+Bojanowski%2E+2017%2E+Bag+of+tricks+for+efficient+text+classification%2E+In+EACL+%282%29%2C+pages+427%E2%80%93431%2E+Association+for+Computational+Linguistics%2E)
17. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+Bowman%2C+and+Noah+A%2E+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+Proceedings+of+the+2018+Conference+o)
18. Xu Han, Shulin Cao, Lv Xin, Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. 2018. Openke: An open toolkit for knowledge embedding. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Shulin+Cao%2C+Lv+Xin%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+and+Juanzi+Li%2E+2018%2E+Openke%3A+An+open+toolkit+for+knowledge+embedding%2E+In+Proceedings+of+EMNLP%2E)
19. Marti A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings of the 14th Conference on Computational Linguistics - Volume 2, COLING ’92, pages 539–545, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marti+A%2E+Hearst%2E+1992%2E+Automatic+acquisition+of+hyponyms+from+large+text+corpora%2E+In+Proceedings+of+the+14th+Conference+on+Computational+Linguistics+%2D+Volume+2%2C+COLING+%E2%80%9992%2C+pages+539%E2%80%93545%2C+Stroudsburg%2C)
20. Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. 2018. Using trusted data to train deep networks on labels corrupted by severe noise. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 10477–10486. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.1802.05300)
21. Lili Kotlerman, Ido Dagan, Idan Szpektor, and Maayan Zhitomirsky-geffet. 2010. Directional distributional similarity for lexical inference. Nat. Lang. Eng., 16(4):359–389. [↗](https://doi.org/10.1017/s1351324910000124)
22. John Lalor, Hao Wu, and hong yu. 2016. Building an evaluation scale using item response theory. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 648– 657, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=John+Lalor%2C+Hao+Wu%2C+and+hong+yu%2E+2016%2E+Building+an+evaluation+scale+using+item+response+theory%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+648%E2%80%93+65)
23. Alessandro Lenci and Giulia Benotto. 2012. Identifying hypernyms in distributional semantic spaces. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 75–79, Montréal, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alessandro+Lenci+and+Giulia+Benotto%2E+2012%2E+Identifying+hypernyms+in+distributional+semantic+spaces%2E+In+%2ASEM+2012%3A+The+First+Joint+Conference+on+Lexical+and+Computational+Semantics+%E2%80%93+Volume+1%3A+Proceedi)
24. Omer Levy and Ido Dagan. 2016. Annotating relation inference in context via question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Ido+Dagan%2E+2016%2E+Annotating+relation+inference+in+context+via+question+answering%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+ACL+2016%2C+Au)
25. Mike Lewis and Mark Steedman. 2013. Combining distributional and logical semantics. Transactions of the Association for Computational Linguistics, 1:179–192. [find](https://scholar.google.com/scholar?q=Mike+Lewis+and+Mark+Steedman%2E+2013%2E+Combining+distributional+and+logical+semantics%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+1%3A179%E2%80%93192%2E)
26. Dekang Lin and Patrick Pantel. 2001. DIRT: Discovery of Inference Rules from Text. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’01), pages 323–328, New York, NY, USA. ACM Press. [find](https://scholar.google.com/scholar?q=Dekang+Lin+and+Patrick+Pantel%2E+2001%2E+DIRT%3A+Discovery+of+Inference+Rules+from+Text%2E+In+Proceedings+of+the+Seventh+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining+%28KDD%E2%80%9901%29%2C+pa)
27. Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. In In Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics. Philadelphia: Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.cs/0205028)
28. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. [↗](https://arxiv.org/abs/1301.3781)
29. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.1310.4546)
30. George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41. [↗](https://doi.org/10.1145/219717.219748) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lex-1994-miller-wordnet/)
31. Saif Mohammad, Bonnie Dorr, and Graeme Hirst. 2008. Computing word-pair antonymy. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 982–991, Honolulu, Hawaii. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Saif+Mohammad%2C+Bonnie+Dorr%2C+and+Graeme+Hirst%2E+2008%2E+Computing+word%2Dpair+antonymy%2E+In+Proceedings+of+the+2008+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+982%E2%80%93991%2C+Honolulu%2C+Ha)
32. Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 142–148, San Diego, California. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nikola+Mrks%CC%8Cic%CC%81%2C+Diarmuid+O%CC%81+Se%CC%81aghdha%2C+Blaise+Thomson%2C+Milica+Gas%CC%8Cic%CC%81%2C+Lina+M%2E+Rojas%2DBarahona%2C+Pei%2DHao+Su%2C+David+Vandyke%2C+Tsung%2DHsien+Wen%2C+and+Steve+Young%2E+2016%2E+Counter%2Dfitting+word+vectors+to+lingu)
33. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1806.00692)
34. Ndapandula Nakashole, Gerhard Weikum, and Fabian Suchanek. 2012. Patty: A taxonomy of relational patterns with semantic types. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1135–1145, Jeju Island, Korea. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ndapandula+Nakashole%2C+Gerhard+Weikum%2C+and+Fabian+Suchanek%2E+2012%2E+Patty%3A+A+taxonomy+of+relational+patterns+with+semantic+types%2E+In+Proceedings+of+the+2012+Joint+Conference+on+Empirical+Methods+in+Natur)
35. Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gülşen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007. Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(2):95–135. [↗](https://doi.org/10.1017/s1351324906004505)
36. Ellie Pavlick and Chris Callison-Burch. 2016. Tense manages to predict implicative behavior in verbs. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2225–2229, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ellie+Pavlick+and+Chris+Callison%2DBurch%2E+2016%2E+Tense+manages+to+predict+implicative+behavior+in+verbs%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+2)
37. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. Ppdb 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 425–430, Beijing, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ellie+Pavlick%2C+Pushpendre+Rastogi%2C+Juri+Ganitkevitch%2C+Benjamin+Van+Durme%2C+and+Chris+Callison%2DBurch%2E+2015%2E+Ppdb+2%2E0%3A+Better+paraphrase+ranking%2C+fine%2Dgrained+entailment+relations%2C+word+embeddings%2C+and+s) [our card](https://md.mova.elamur.ai/free/en/docs/papers/pavlick-2015-ppdb-2-0-better-paraphrase/)
38. Stephen Roller, Douwe Kiela, and Maximilian Nickel. 2018. Hearst patterns revisited: Automatic hypernym detection from large text corpora. In Proceedings of the 56th Annual Meeting of the Association 2018. Jack the Reader – A Machine Reading Framework. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL) System Demonstrations. for Computational Linguistics (Volume 2: Short Papers), pages 358–363, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Stephen+Roller%2C+Douwe+Kiela%2C+and+Maximilian+Nickel%2E+2018%2E+Hearst+patterns+revisited%3A+Automatic+hypernym+detection+from+large+text+corpora%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+)
39. Wesley C Salmon. 1971. Statistical explanation and statistical relevance, volume 69. University of Pittsburgh Pre. [find](https://scholar.google.com/scholar?q=Wesley+C+Salmon%2E+1971%2E+Statistical+explanation+and+statistical+relevance%2C+volume+69%2E+University+of+Pittsburgh+Pre%2E)
40. Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte im Walde. 2014. Chasing hypernyms in vector spaces with entropy. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 38–42, Gothenburg, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Enrico+Santus%2C+Alessandro+Lenci%2C+Qin+Lu%2C+and+Sabine+Schulte+im+Walde%2E+2014%2E+Chasing+hypernyms+in+vector+spaces+with+entropy%2E+In+Proceedings+of+the+14th+Conference+of+the+European+Chapter+of+the+Associ)
41. Stefan Schoenmackers, Jesse Davis, Oren Etzioni, and Daniel Weld. 2010. Learning first-order horn clauses from web text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing: 2010, pages 1088–1098. [find](https://scholar.google.com/scholar?q=Stefan+Schoenmackers%2C+Jesse+Davis%2C+Oren+Etzioni%2C+and+Daniel+Weld%2E+2010%2E+Learning+first%2Dorder+horn+clauses+from+web+text%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
42. Satoshi Sekine. 2005. Automatic paraphrase discovery based on context and keywords between NE pairs. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). [find](https://scholar.google.com/scholar?q=Satoshi+Sekine%2E+2005%2E+Automatic+paraphrase+discovery+based+on+context+and+keywords+between+NE+pairs%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphrasing+%28IWP2005%29%2E)
43. Vered Shwartz, Enrico Santus, and Dominik Schlechtweg. 2017a. Hypernyms under siege: Linguistically-motivated artillery for hypernymy detection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 65–75, Valencia, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vered+Shwartz%2C+Enrico+Santus%2C+and+Dominik+Schlechtweg%2E+2017a%2E+Hypernyms+under+siege%3A+Linguistically%2Dmotivated+artillery+for+hypernymy+detection%2E+In+Proceedings+of+the+15th+Conference+of+the+European+C)
44. Vered Shwartz, Gabriel Stanovsky, and Ido Dagan. 2017b. Acquiring predicate paraphrases from news tweets. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 155–160, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vered+Shwartz%2C+Gabriel+Stanovsky%2C+and+Ido+Dagan%2E+2017b%2E+Acquiring+predicate+paraphrases+from+news+tweets%2E+In+Proceedings+of+the+6th+Joint+Conference+on+Lexical+and+Computational+Semantics+%28%2ASEM+2017%29%2C)
45. Idan Szpektor, Hristo Tanev, Ido Dagan, and Bonaventura Coppola. 2004. Scaling web-based acquisition of entailment relations. In Proceedings of EMNLP 2004, pages 41–48, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Idan+Szpektor%2C+Hristo+Tanev%2C+Ido+Dagan%2C+and+Bonaventura+Coppola%2E+2004%2E+Scaling+web%2Dbased+acquisition+of+entailment+relations%2E+In+Proceedings+of+EMNLP+2004%2C+pages+41%E2%80%9348%2C+Barcelona%2C+Spain%2E+Association+f)
46. Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, pages 2071–2080. [↗](https://doi.org/10.48550/arxiv.1606.06357)
47. Julie Weeds, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional similarity. In Proceedings of Coling 2004, pages 1015–1021, Geneva, Switzerland. COLING. Dirk Weissenborn, Pasquale Minervini, Tim Dettmers, Isabelle Augenstein, Johannes Welbl, Tim Rocktäschel, Matko Bošnjak, Jeff Mitchell, Thomas Demeester, Pontus Stenetorp, and Sebastian Riedel. [find](https://scholar.google.com/scholar?q=Julie+Weeds%2C+David+Weir%2C+and+Diana+McCarthy%2E+2004%2E+Characterising+measures+of+lexical+distributional+similarity%2E+In+Proceedings+of+Coling+2004%2C+pages+1015%E2%80%931021%2C+Geneva%2C+Switzerland%2E+COLING%2E+Dirk+Weiss)
48. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
49. XTAG Research Group. 2001. A lexicalized tree adjoining grammar for english. Technical Report IRCS-01-03, IRCS, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=XTAG+Research+Group%2E+2001%2E+A+lexicalized+tree+adjoining+grammar+for+english%2E+Technical+Report+IRCS%2D01%2D03%2C+IRCS%2C+University+of+Pennsylvania%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/group-1998-lexicalized-tree-adjoining-grammar-english/)
50. Wei Xu, Alan Ritter, Chris Callison-Burch, William B. Dolan, and Yangfeng Ji. 2014. Extracting lexically divergent paraphrases from twitter. Transactions of the Association for Computational Linguistics, 2:435–448. [↗](https://doi.org/10.1162/tacl_a_00194)
51. Limin Yao, Sebastian Riedel, and Andrew McCallum. 2012. Unsupervised relation discovery with sense disambiguation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1, ACL ’12, pages 712–720, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Limin+Yao%2C+Sebastian+Riedel%2C+and+Andrew+McCallum%2E+2012%2E+Unsupervised+relation+discovery+with+sense+disambiguation%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+Lingui)
52. Naomi Zeichner, Jonathan Berant, and Ido Dagan. 2012. Crowdsourcing inference-rule evaluation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 156–160, Jeju Island, Korea. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Naomi+Zeichner%2C+Jonathan+Berant%2C+and+Ido+Dagan%2E+2012%2E+Crowdsourcing+inference%2Drule+evaluation%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Sho)
53. Congle Zhang, Stephen Soderland, and Daniel S. Weld. 2015. Exploiting parallel news streams for unsupervised event extraction. Transactions of the Association for Computational Linguistics, 3:117–129. [↗](https://doi.org/10.1162/tacl_a_00127)
54. Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1710.11342)