[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/ud-2018-fares-2018-shared-task-extrinsic-parser/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency Parsers

# The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency Parsers

**Authors:** Murhaf Fares, [Stephan Oepen](https://md.mova.elamur.ai/free/en/docs/papers/authors/stephan-oepen/), Lilja Øvrelid, Jari Björne, [Richard Johansson](https://md.mova.elamur.ai/free/en/docs/papers/authors/richard-johansson/) · **Year:** 2018 · **Venue:** Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies **Link:** [aclanthology.org/K18-2002](https://aclanthology.org/K18-2002/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**14** citations](https://openalex.org/works?filter=cites:W2898722612) openalex · 2026-10-19[**13** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/ud-2018-fares-2018-shared-task-extrinsic-parser/) of the 140673 papers whose bibliographies we read[Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies](https://aclanthology.org/volumes/K18-2/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/K18-2002/)
- [PDF](https://aclanthology.org/K18-2002.pdf)
- [DOI 10.18653/v1/k18-2002](https://doi.org/10.18653/v1/k18-2002)
- [OpenAlex](https://openalex.org/W2898722612)

**Authors (5)**

1. [Murhaf Fares](https://openalex.org/A5058116024) [Semantic Scholar](https://www.semanticscholar.org/author/152951667) · h 6 · 206 citations · 13 works
2. [Stephan Oepen](https://openalex.org/A5058412010) [Semantic Scholar](https://www.semanticscholar.org/author/2949607) · h 36 · 3610 citations · 234 works
3. [Lilja Øvrelid](https://openalex.org/A5080614776) [ORCID](https://orcid.org/0000-0002-8880-8404) · [Semantic Scholar](https://www.semanticscholar.org/author/2732223) · h 25 · 1989 citations · 190 works
4. [Jari Björne](https://openalex.org/A5049143500) [ORCID](https://orcid.org/0000-0003-1722-6404) · [Semantic Scholar](https://www.semanticscholar.org/author/1761599) · h 21 · 3988 citations · 62 works
5. [Richard Johansson](https://openalex.org/A5079881269) [ORCID](https://orcid.org/0000-0002-9429-4884) · [Semantic Scholar](https://www.semanticscholar.org/author/145341661) · h 23 · 2708 citations · 140 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

The Second Extrinsic Parser Evaluation (EPE 2018) runs the basic UD trees of the English parsers of seventeen participating teams through three downstream systems: biological event extraction (TEES), negation resolution (Sherlock) and opinion analysis. The question is how far intrinsic scores predict downstream utility. The best 2018 system (UDPipe-Future) reaches downstream F1 of 49.66, 58.45 and 56.19, below the 2017 best system, which used enhanced UD on about 1.7 million tokens of training text, so the two campaigns are not strictly comparable. The correlation analysis is the main result: the average downstream rank correlates most with intrinsic UPOS and MLAS (0.88), event extraction with UPOS (0.71), negation with XPOS (0.75), and opinion analysis with MLAS (0.83). Parsers that predict no fine-grained tags (Uppsala-18) do well on events and opinion but poorly on negation.

## Ideas

- I1. `en/src/ud.rs` and `en/src/ptag.rs`: for every change to the tagger, report UPOS and MLAS on EWT test next to LAS, since these predict downstream rank best; metric: the UPOS-to-MLAS pair on EWT test, one line per release.
- I2. Keep the fine PTB tag as well as the UPOS in the `en` output for negation-type use: metric: negation-scope F on the EPE negation set with the `en` parses, UPOS input versus PTB input (test only if the set is research-only).
- I3. A downstream harness in `prag`/`world`: a small event and negation check on `en` parses of GENIA-like sentences, reported beside LAS; metric: event and negation F per release, treated as a second axis, not a training target.

## For Mova

The chapters are [20. Syntax: the Universal Dependencies frame](https://md.mova.elamur.ai/free/en/docs/languages/en/20-syntax-overview/) and `37-register-and-variation.md` for the claim that intrinsic scores are not enough. The code is `en/src/ud.rs` (the UPOS and feature tables), `en/src/ptag.rs`, and the `prag`/`world` layer as the natural downstream check. Measure UPOS and MLAS first, because they are the cheap predictors.

Topics: [Dependency parsing](https://md.mova.elamur.ai/free/en/docs/papers/topics/dependency-parsing/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/k18-2002](https://doi.org/10.18653/v1/k18-2002)

```
@inproceedings{fares20182018,
  title = {{The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency Parsers}},
  author = {Fares, Murhaf and Oepen, Stephan and Øvrelid, Lilja and Björne, Jari and Johansson, Richard},
  booktitle = {Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies},
  year = {2018},
  doi = {10.18653/v1/k18-2002},
  url = {https://doi.org/10.18653/v1/k18-2002}
}
```

- [← previous Mind the Gap: Data Enrichment in Dependency Parsing of Elliptical Constructions](https://md.mova.elamur.ai/free/en/docs/papers/ud-2018-droganova-mind-gap-data-enrichment-dependency/)
- [next → Investigating NP-Chunking with Universal Dependencies for English](https://md.mova.elamur.ai/free/en/docs/papers/ud-2018-lacroix-investigating-np-chunking-universal-dependencies/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Miguel Ballesteros, Bernd Bohnet, Simon Mille, and Leo Wanner. 2015. Data-driven deep-syntactic dependency parsing. Natural Language Engineering 22:1 – 36. [find](https://scholar.google.com/scholar?q=Miguel+Ballesteros%2C+Bernd+Bohnet%2C+Simon+Mille%2C+and+Leo+Wanner%2E+2015%2E+Data%2Ddriven+deep%2Dsyntactic+dependency+parsing%2E+Natural+Language+Engineering+22%3A1+%E2%80%93+36%2E)
2. Thorsten Joachims. 1999. Making large-scale SVM learning practical. In Bernhard Schölkopf, Christopher J. C. Burges, and Alexander J. Smola, editors, Advances in Kernel Methods. Support Vector Learning, MIT Press, Cambridge, MA, USA, page 41 – 56. [find](https://scholar.google.com/scholar?q=Thorsten+Joachims%2E+1999%2E+Making+large%2Dscale+SVM+learning+practical%2E+In+Bernhard+Sch%C3%B6lkopf%2C+Christopher+J%2E+C%2E+Burges%2C+and+Alexander+J%2E+Smola%2C+editors%2C+Advances+in+Kernel+Methods%2E+Support+Vector+Learnin)
3. Richard Johansson. 2017. EPE 2017: The Trento– Gothenburg opinion extraction system. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 31 – 39. [find](https://scholar.google.com/scholar?q=Richard+Johansson%2E+2017%2E+EPE+2017%3A+The+Trento%E2%80%93+Gothenburg+opinion+extraction+system%2E+In+Proceedings+of+the+2017+Shared+Task+on+Extrinsic+Parser+Evaluation+at+the+Fourth+International+Conference+on+Dep)
4. Richard Johansson and Alessandro Moschitti. 2013. Relational features in fine-grained opinion analysis. Computational Linguistics 39(3):473 – 509. [find](https://scholar.google.com/scholar?q=Richard+Johansson+and+Alessandro+Moschitti%2E+2013%2E+Relational+features+in+fine%2Dgrained+opinion+analysis%2E+Computational+Linguistics+39%283%29%3A473+%E2%80%93+509%2E)
5. Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshi-nobu Kano, and Jun’ichi Tsujii. 2009. Overview of BioNLP’09 Shared Task on event extraction. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing. Shared Task. Boulder, CO, USA, page 1 – 9. [find](https://scholar.google.com/scholar?q=Jin%2DDong+Kim%2C+Tomoko+Ohta%2C+Sampo+Pyysalo%2C+Yoshi%2Dnobu+Kano%2C+and+Jun%E2%80%99ichi+Tsujii%2E+2009%2E+Overview+of+BioNLP%E2%80%9909+Shared+Task+on+event+extraction%2E+In+Proceedings+of+the+Workshop+on+Current+Trends+in+Biomedi)
6. Marco Kuhlmann and Stephan Oepen. 2016. Towards a catalogue of linguistic graph banks. Computational Linguistics 42(4):819 – 827. [find](https://scholar.google.com/scholar?q=Marco+Kuhlmann+and+Stephan+Oepen%2E+2016%2E+Towards+a+catalogue+of+linguistic+graph+banks%2E+Computational+Linguistics+42%284%29%3A819+%E2%80%93+827%2E)
7. Emanuele Lapponi, Stephan Oepen, and Lilja Øvrelid. 2017. EPE 2017: The Sherlock negation resolution downstream application. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 25 – 30. [find](https://scholar.google.com/scholar?q=Emanuele+Lapponi%2C+Stephan+Oepen%2C+and+Lilja+%C3%98vrelid%2E+2017%2E+EPE+2017%3A+The+Sherlock+negation+resolution+downstream+application%2E+In+Proceedings+of+the+2017+Shared+Task+on+Extrinsic+Parser+Evaluation+at+th)
8. Jari Björne. 2014. Biomedical Event Extraction with Machine Learning. Ph.D. thesis, University of Turku, Turku, Finland. [find](https://scholar.google.com/scholar?q=Jari+Bj%C3%B6rne%2E+2014%2E+Biomedical+Event+Extraction+with+Machine+Learning%2E+Ph%2ED%2E+thesis%2C+University+of+Turku%2C+Turku%2C+Finland%2E)
9. Yufei Chen, Junjie Cao, Weiwei Sun, and Xiaojun Wan. 2017. Peking at EPE 2017: A comparison of tree approximation, transition-based, and maximum subgraph models for semantic dependency analysis. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 60 – 64. [find](https://scholar.google.com/scholar?q=Yufei+Chen%2C+Junjie+Cao%2C+Weiwei+Sun%2C+and+Xiaojun+Wan%2E+2017%2E+Peking+at+EPE+2017%3A+A+comparison+of+tree+approximation%2C+transition%2Dbased%2C+and+maximum+subgraph+models+for+semantic+dependency+analysis%2E+In+Pr)
10. Emanuele Lapponi, Erik Velldal, Lilja Øvrelid, and Jonathon Read. 2012. UiO2. Sequence-labeling negation using dependency features. In Proceedings of the 1st Joint Conference on Lexical and Computational Semantics. Montréal, Canada, page 319 – 327. [find](https://scholar.google.com/scholar?q=Emanuele+Lapponi%2C+Erik+Velldal%2C+Lilja+%C3%98vrelid%2C+and+Jonathon+Read%2E+2012%2E+UiO2%2E+Sequence%2Dlabeling+negation+using+dependency+features%2E+In+Proceedings+of+the+1st+Joint+Conference+on+Lexical+and+Computatio)
11. Thomas Lavergne, Olivier Cappé, and François Yvon. 2010. Practical very large scale CRFs. In Proceedings of the 48th Meeting of the Association for Computational Linguistics. Uppsala, Sweden, page 504 – 513. [find](https://scholar.google.com/scholar?q=Thomas+Lavergne%2C+Olivier+Capp%C3%A9%2C+and+Fran%C3%A7ois+Yvon%2E+2010%2E+Practical+very+large+scale+CRFs%2E+In+Proceedings+of+the+48th+Meeting+of+the+Association+for+Computational+Linguistics%2E+Uppsala%2C+Sweden%2C+page+504)
12. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th International Conference on Language Resources and Evaluation. Genoa, Italy, page 449 – 454. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+de+Marneffe%2C+Bill+MacCartney%2C+and+Christopher+D%2E+Manning%2E+2006%2E+Generating+typed+dependency+parses+from+phrase+structure+parses%2E+In+Proceedings+of+the+5th+International+Conference+on+L) [our card](https://md.mova.elamur.ai/free/en/docs/papers/map-2006-marneffe-generating-typed-dependency-parses-phrase/)
13. Angelina Ivanova, Stephan Oepen, Lilja Øvrelid, and Dan Flickinger. 2012. Who did what to whom? A contrastive study of syntacto-semantic dependencies. In Proceedings of the 6th Linguistic Annotation Workshop. Jeju, Republic of Korea, page 2 – 11. [find](https://scholar.google.com/scholar?q=Angelina+Ivanova%2C+Stephan+Oepen%2C+Lilja+%C3%98vrelid%2C+and+Dan+Flickinger%2E+2012%2E+Who+did+what+to+whom%3F+A+contrastive+study+of+syntacto%2Dsemantic+dependencies%2E+In+Proceedings+of+the+6th+Linguistic+Annotation+W)
14. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, and Oscar Täckström. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the 51th Meeting of the Association for Computational Linguistics. Sofia, Bulgaria, page 92 – 97. [find](https://scholar.google.com/scholar?q=Ryan+McDonald%2C+Joakim+Nivre%2C+Yvonne+Quirmbach%2DBrundage%2C+Yoav+Goldberg%2C+Dipanjan+Das%2C+Kuzman+Ganchev%2C+Keith+Hall%2C+Slav+Petrov%2C+Hao+Zhang%2C+and+Oscar+T%C3%A4ckstr%C3%B6m%2E+2013%2E+Universal+dependency+annotation+for+)
15. Tao Ji, Yuekun Yao, Qi Zheng, Yuanbin Wu, and Man Lan. 2017. ECNU at EPE 2017: Universal dependencies representations parser. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Simon Mille, Roberto Carlini, Ivan Latorre, and Leo Wanner. 2017. UPF at EPE 2017: Transduction-based deep analysis. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Milan Straka, Jana Straková, and Jan Hajič. 2017. Prague at EPE 2017: The UDPipe system. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 65 – 74. Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 76 – 84. [find](https://scholar.google.com/scholar?q=Tao+Ji%2C+Yuekun+Yao%2C+Qi+Zheng%2C+Yuanbin+Wu%2C+and+Man+Lan%2E+2017%2E+ECNU+at+EPE+2017%3A+Universal+dependencies+representations+parser%2E+In+Proceedings+of+the+2017+Shared+Task+on+Extrinsic+Parser+Evaluation+at+t)
16. Yusuke Miyao. 2006. From Linguistic Theory to Syntactic Analysis. Corpus-Oriented Grammar Development and Feature Forest Model. Doctoral dissertation, University of Tokyo, Tokyo, Japan. [find](https://scholar.google.com/scholar?q=Yusuke+Miyao%2E+2006%2E+From+Linguistic+Theory+to+Syntactic+Analysis%2E+Corpus%2DOriented+Grammar+Development+and+Feature+Forest+Model%2E+Doctoral+dissertation%2C+University+of+Tokyo%2C+Tokyo%2C+Japan%2E)
17. Roser Morante and Eduardo Blanco. 2012. *SEM 2012 Shared Task. Resolving the scope and focus of negation. In Proceedings of the 1st Joint Conference on Lexical and Computational Semantics. Montréal, Canada, page 265 – 274. [find](https://scholar.google.com/scholar?q=Roser+Morante+and+Eduardo+Blanco%2E+2012%2E+%2ASEM+2012+Shared+Task%2E+Resolving+the+scope+and+focus+of+negation%2E+In+Proceedings+of+the+1st+Joint+Conference+on+Lexical+and+Computational+Semantics%2E+Montr%C3%A9al%2C+C)
18. Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluís Màrquez, and Joakim Nivre. 2008. [find](https://scholar.google.com/scholar?q=Mihai+Surdeanu%2C+Richard+Johansson%2C+Adam+Meyers%2C+Llu%C3%ADs+M%C3%A0rquez%2C+and+Joakim+Nivre%2E+2008%2E)
19. The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies. In Proceedings of the 12th Conference on Natural Language Learning. Manchester, UK, page 159 – 177. [find](https://scholar.google.com/scholar?q=The+CoNLL+2008+Shared+Task+on+Joint+Parsing+of+Syntactic+and+Semantic+Dependencies%2E+In+Proceedings+of+the+12th+Conference+on+Natural+Language+Learning%2E+Manchester%2C+UK%2C+page+159+%E2%80%93+177%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/surdeanu-2008-conll-2008-shared-task-joint/)
20. Roser Morante and Walter Daelemans. 2012. ConanDoyle-neg. Annotation of negation in Conan Doyle stories. In Proceedings of the 8th International Conference on Language Resources and Evaluation. Istanbul, Turkey, page 1563 – 1568. [find](https://scholar.google.com/scholar?q=Roser+Morante+and+Walter+Daelemans%2E+2012%2E+ConanDoyle%2Dneg%2E+Annotation+of+negation+in+Conan+Doyle+stories%2E+In+Proceedings+of+the+8th+International+Conference+on+Language+Resources+and+Evaluation%2E+Istanb)
21. Zsolt Szántó and Richárd Farkas. 2017. Szeged at EPE 2017: First experiments in a generalized syntactic parsing framework. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 75 – 79. [find](https://scholar.google.com/scholar?q=Zsolt+Sz%C3%A1nt%C3%B3+and+Rich%C3%A1rd+Farkas%2E+2017%2E+Szeged+at+EPE+2017%3A+First+experiments+in+a+generalized+syntactic+parsing+framework%2E+In+Proceedings+of+the+2017+Shared+Task+on+Extrinsic+Parser+Evaluation+at+the+)
22. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1. A multilingual treebank collection. In Proceedings of the 10th International Conference on Language Resources and Evaluation. Portorož, Slovenia, page 1659 – 1666. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Marie%2DCatherine+de+Marneffe%2C+Filip+Ginter%2C+Yoav+Goldberg%2C+Jan+Hajic%CC%8C%2C+Christopher+D%2E+Manning%2C+Ryan+McDonald%2C+Slav+Petrov%2C+Sampo+Pyysalo%2C+Natalia+Silveira%2C+Reut+Tsarfaty%2C+and+Daniel+Zeman) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2016-nivre-universal-dependencies-v1-multilingual-treebank/)
23. Yuka Tateisi, Akane Yakushiji, Tomoko Ohta, and Jun’ichi Tsujii. 2005. Syntax annotation for the GE-NIA corpus. In Proceedings of the 2nd International Joint Conference on Natural Language Processing. Companion Volume. Jeju, Korea. [find](https://scholar.google.com/scholar?q=Yuka+Tateisi%2C+Akane+Yakushiji%2C+Tomoko+Ohta%2C+and+Jun%E2%80%99ichi+Tsujii%2E+2005%2E+Syntax+annotation+for+the+GE%2DNIA+corpus%2E+In+Proceedings+of+the+2nd+International+Joint+Conference+on+Natural+Language+Processing%2E)
24. Stephan Oepen, Lilja Øvrelid, Jari Björne, Richard Johansson, Emanuele Lapponi, Filip Ginter, and Erik Velldal. 2017. The 2017 Shared Task on Extrinsic Parser Evaluation. Towards a reusable community infrastructure. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 1 – 16. [find](https://scholar.google.com/scholar?q=Stephan+Oepen%2C+Lilja+%C3%98vrelid%2C+Jari+Bj%C3%B6rne%2C+Richard+Johansson%2C+Emanuele+Lapponi%2C+Filip+Ginter%2C+and+Erik+Velldal%2E+2017%2E+The+2017+Shared+Task+on+Extrinsic+Parser+Evaluation%2E+Towards+a+reusable+community+)
25. Erik Velldal, Lilja Øvrelid, Jonathon Read, and Stephan Oepen. 2012. Speculation and negation: Rules, rankers, and the role of syntax. Computational Linguistics 38(2):369 – 410. [find](https://scholar.google.com/scholar?q=Erik+Velldal%2C+Lilja+%C3%98vrelid%2C+Jonathon+Read%2C+and+Stephan+Oepen%2E+2012%2E+Speculation+and+negation%3A+Rules%2C+rankers%2C+and+the+role+of+syntax%2E+Computational+Linguistics+38%282%29%3A369+%E2%80%93+410%2E)
26. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation 39(2-3):165 – 210. [find](https://scholar.google.com/scholar?q=Janyce+Wiebe%2C+Theresa+Wilson%2C+and+Claire+Cardie%2E+2005%2E+Annotating+expressions+of+opinions+and+emotions+in+language%2E+Language+Resources+and+Evaluation+39%282%2D3%29%3A165+%E2%80%93+210%2E)
27. Hao Peng, Sam Thomson, and Noah A. Smith. 2017. Deep multitask learning for semantic dependency parsing. In Proceedings of the 55th Meeting of the Association for Computational Linguistics. Vancouver, Canada, page 2037 – 2048. [find](https://scholar.google.com/scholar?q=Hao+Peng%2C+Sam+Thomson%2C+and+Noah+A%2E+Smith%2E+2017%2E+Deep+multitask+learning+for+semantic+dependency+parsing%2E+In+Proceedings+of+the+55th+Meeting+of+the+Association+for+Computational+Linguistics%2E+Vancouver%2C)
28. Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 Shared Task. Multilingual Parsing from Raw Text to Universal Dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Brussels, Belgium, page 1 – 20. [find](https://scholar.google.com/scholar?q=Daniel+Zeman%2C+Jan+Hajic%CC%8C%2C+Martin+Popel%2C+Martin+Potthast%2C+Milan+Straka%2C+Filip+Ginter%2C+Joakim+Nivre%2C+and+Slav+Petrov%2E+2018%2E+CoNLL+2018+Shared+Task%2E+Multilingual+Parsing+from+Raw+Text+to+Universal+Depend) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2018-zeman-conll-2018-shared-task-multilingual/)
29. Martin Potthast, Tim Gollub, Francisco Rangel, Paolo Rosso, Efstathios Stamatatos, and Benno Stein. 2014. Improving the reproducibility of PAN’s shared tasks. Plagiarism detection, author identification, and author profiling. In Evangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sanderson, Mark Hall, Allan Hanbury, and Elaine Toms, editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. 5th International Conference of the CLEF Initiative. Springer, Berlin, Germany, page 268 – 299. [find](https://scholar.google.com/scholar?q=Martin+Potthast%2C+Tim+Gollub%2C+Francisco+Rangel%2C+Paolo+Rosso%2C+Efstathios+Stamatatos%2C+and+Benno+Stein%2E+2014%2E+Improving+the+reproducibility+of+PAN%E2%80%99s+shared+tasks%2E+Plagiarism+detection%2C+author+identificati)
30. Sebastian Schuster, Eric De La Clergerie, Marie Candito, Benoît Sagot, Christopher D. Manning, and Djamé Seddah. 2017. Paris and Stanford at EPE 2017: Downstream evaluation of graph-based dependency representations. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 47 – 59. [find](https://scholar.google.com/scholar?q=Sebastian+Schuster%2C+Eric+De+La+Clergerie%2C+Marie+Candito%2C+Beno%C3%AEt+Sagot%2C+Christopher+D%2E+Manning%2C+and+Djam%C3%A9+Seddah%2E+2017%2E+Paris+and+Stanford+at+EPE+2017%3A+Downstream+evaluation+of+graph%2Dbased+dependency+r)