[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/scholman-2022-discogem-crowdsourced-corpus-genre-mixed/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations

# DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations

**Authors:** Merel Scholman, Tianai Dong, [Frances Yung](https://md.mova.elamur.ai/free/en/docs/papers/authors/frances-yung/), [Vera Demberg](https://md.mova.elamur.ai/free/en/docs/papers/authors/vera-demberg/) · **Year:** 2022 · **Venue:** LREC 2022 **Link:** [aclanthology.org/2022.lrec-1.351](https://aclanthology.org/2022.lrec-1.351/) · **Licence:** [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) as printed on the paper — summary only; the corpus has its own licence, tests only

[**23** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/scholman-2022-discogem-crowdsourced-corpus-genre-mixed/) of the 140673 papers whose bibliographies we read[Proceedings of the Thirteenth Language Resources and Evaluation Conference](https://aclanthology.org/volumes/2022.lrec-1/)[licence](https://creativecommons.org/licenses/by-nc/4.0/)

- [Publisher's page](https://aclanthology.org/2022.lrec-1.351/)
- [PDF](https://aclanthology.org/2022.lrec-1.351.pdf)

**Authors (4)**

1. Merel Scholman
2. Tianai Dong
3. Frances Yung
4. Vera Demberg

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

DiscoGeM holds 6,505 implicit relations between consecutive sentences (no connective among the first five words of the second) from political speeches (Europarl), 20 novels and Wikipedia summaries, each labelled by 10 crowd workers (65,863 judgements) with PDTB 3.0 senses. Workers first type the connective that fits the gap, then choose among up to ten unambiguous connectives offered for what they typed, from a bank of over 2,000 connectives, variants and alternative lexicalisations mapped to senses; quality is controlled by a selection task with feedback. Many relations are genuinely ambiguous (a Wikipedia sentence may be both detail and result), so probability-distribution labels describe them better than one label; trained experts agreed only 60% (kappa 0.45) on level-3 senses. Genre strongly shapes the distribution: conjunction dominates Wikipedia, result Europarl, and novels carry more temporal relations; dialogue pairs in novels («He said X. She said Y.») were excluded as uninformative.

## Ideas

- I1. Get teacher silver for implicit links by connective insertion, the paper's two-step method: for a pair of adjacent tale or lesson sentences, the teacher writes the connective that fits (or none), and Mova's own connective table maps it to `cause`/`next`/contrast/none deterministically. The label is then checkable and explainable through the inserted word, and the mapping stays in Mova. Measure agreement of the resulting labels with the frame bench gold on the bench lines before using them for rules.
- I2. Treat consecutive quotes by different speakers as their own relation (a reply, see the dialogue cards), not as `next` between events, as the paper excludes them from temporal counts. Today they would count in Mova's sentence-order `next`; separating them should clean up `next` precision on the tales. Measure frame bench next F1 with and without.
- I3. Keep link statistics per genre (tales vs lessons vs everyday dialogue): the paper shows temporal relations are typical of narrative and conjunction/detail of encyclopedic text, so a default like «unmarked consecutive actions → next» is right for tales and wrong for lessons. A genre switch on the default, measured on the course groups separately.

## For Mova

[36. Discourse relations and coherence](https://md.mova.elamur.ai/free/en/docs/languages/en/36-discourse-relations/) §7 (implicit relations), §8 (time and narrative), §11.1–11.2. Code: `prag/src/frames.rs` (`set_next` defaults by genre, quote pairs), `prag/src/silver.rs` (connective insertion prompt), the connective table. Measures: frame bench next F1; agreement of insertion labels with the bench gold; course results by genre.

Topics: [Discourse analysis](https://md.mova.elamur.ai/free/en/docs/papers/topics/discourse-analysis/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{scholman2022discogem,
  title = {{DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations}},
  author = {Scholman, Merel and Dong, Tianai and Yung, Frances and Demberg, Vera},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  url = {https://aclanthology.org/2022.lrec-1.351/}
}
```

- [← previous Categories of coherence relations in discourse annotation](https://md.mova.elamur.ai/free/en/docs/papers/scholman-2016-categories-coherence-relations-discourse-annotation/)
- [next → Picking Reference Events from Tense Trees: A Formal, Implementable Theory of English Tense-Aspect Semantics](https://md.mova.elamur.ai/free/en/docs/papers/schubert-1990-picking-reference-events-tense-formal/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

40 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aroyo, L. and Welty, C. (2013). Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. WebSci2013 ACM, 2013(2013). [find](https://scholar.google.com/scholar?q=Aroyo%2C+L%2E+and+Welty%2C+C%2E+%282013%29%2E+Crowd+truth%3A+Harnessing+disagreement+in+crowdsourcing+a+relation+extraction+gold+standard%2E+WebSci2013+ACM%2C+2013%282013%29%2E)
2. Carlson, L., Marcu, D., and Okurowski, M. E. (2003). Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory. In Current and new directions in discourse and dialogue, pages 85–112. Springer. [find](https://scholar.google.com/scholar?q=Carlson%2C+L%2E%2C+Marcu%2C+D%2E%2C+and+Okurowski%2C+M%2E+E%2E+%282003%29%2E+Building+a+discourse%2Dtagged+corpus+in+the+framework+of+Rhetorical+Structure+Theory%2E+In+Current+and+new+directions+in+discourse+and+dialogue%2C+pages+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/carlson-2001-building-discourse-tagged-corpus-framework/)
3. Cartoni, B. and Meyer, T. (2012). Extracting directional and comparable corpora from a multilingual corpus for translation studies. In Proceedings 8th International Conference on Language Resources and Evaluation (LREC), Istanbul, Turkey. [find](https://scholar.google.com/scholar?q=Cartoni%2C+B%2E+and+Meyer%2C+T%2E+%282012%29%2E+Extracting+directional+and+comparable+corpora+from+a+multilingual+corpus+for+translation+studies%2E+In+Proceedings+8th+International+Conference+on+Language+Resources+an)
4. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Cohen%2C+J%2E+%281960%29%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+Psychological+Measurement%2C+20%281%29%3A37%E2%80%9346%2E)
5. Demberg, V., Scholman, M. C. J., and Asr, F. T. (2019). How compatible are our discourse annotation frameworks? Insights from mapping RST-DT and PDTB annotations. Dialogue & Discourse, pages 87–135. [find](https://scholar.google.com/scholar?q=Demberg%2C+V%2E%2C+Scholman%2C+M%2E+C%2E+J%2E%2C+and+Asr%2C+F%2E+T%2E+%282019%29%2E+How+compatible+are+our+discourse+annotation+frameworks%3F+Insights+from+mapping+RST%2DDT+and+PDTB+annotations%2E+Dialogue+%26+Discourse%2C+pages+87%E2%80%93135%2E)
6. Dumitrache, A., Inel, O., Aroyo, L., Timmermans, B., and Welty, C. (2018). CrowdTruth 2.0: Quality metrics for crowdsourcing with disagreement. In 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing, and Short Paper 1st Workshop on Disentangling the Relation Between Crowdsourcing and Bias Management, SAD+ CrowdBias 2018, pages 11–18. CEUR-WS. [find](https://scholar.google.com/scholar?q=Dumitrache%2C+A%2E%2C+Inel%2C+O%2E%2C+Aroyo%2C+L%2E%2C+Timmermans%2C+B%2E%2C+and+Welty%2C+C%2E+%282018%29%2E+CrowdTruth+2%2E0%3A+Quality+metrics+for+crowdsourcing+with+disagreement%2E+In+1st+Workshop+on+Subjectivity%2C+Ambiguity+and+Disagreem)
7. Hobbs, J. R. (1979). Coherence and coreference. Cognitive Science, 3(1):67–90. [find](https://scholar.google.com/scholar?q=Hobbs%2C+J%2E+R%2E+%281979%29%2E+Coherence+and+coreference%2E+Cognitive+Science%2C+3%281%29%3A67%E2%80%9390%2E)
8. Hoek, J., Scholman, M. C. J., and Sanders, T. J. (2021). Is there less annotator agreement when the discourse relation is underspecified? In Integrating Perspectives on Discourse Annotation (DiscAnn), pages 1– 6. [find](https://scholar.google.com/scholar?q=Hoek%2C+J%2E%2C+Scholman%2C+M%2E+C%2E+J%2E%2C+and+Sanders%2C+T%2E+J%2E+%282021%29%2E+Is+there+less+annotator+agreement+when+the+discourse+relation+is+underspecified%3F+In+Integrating+Perspectives+on+Discourse+Annotation+%28DiscAnn%29%2C)
9. Ji, Y., Haffari, G., and Eisenstein, J. (2016). A latent variable recurrent neural network for discourse-driven language models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 332–342, San Diego, California, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ji%2C+Y%2E%2C+Haffari%2C+G%2E%2C+and+Eisenstein%2C+J%2E+%282016%29%2E+A+latent+variable+recurrent+neural+network+for+discourse%2Ddriven+language+models%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+)
10. Kawahara, D., Machida, Y., Shibata, T., Kurohashi, S., Kobayashi, H., and Sassano, M. (2014). Rapid development of a corpus with discourse annotations using two-stage crowdsourcing. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 269–278, Dublin, Ireland. [find](https://scholar.google.com/scholar?q=Kawahara%2C+D%2E%2C+Machida%2C+Y%2E%2C+Shibata%2C+T%2E%2C+Kurohashi%2C+S%2E%2C+Kobayashi%2C+H%2E%2C+and+Sassano%2C+M%2E+%282014%29%2E+Rapid+development+of+a+corpus+with+discourse+annotations+using+two%2Dstage+crowdsourcing%2E+In+Proceedings+of+)
11. Kishimoto, Y., Sawada, S., Murawaki, Y., Kawahara, D., and Kurohashi, S. (2018). Improving crowdsourcing-based annotation of Japanese discourse relations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). [find](https://scholar.google.com/scholar?q=Kishimoto%2C+Y%2E%2C+Sawada%2C+S%2E%2C+Murawaki%2C+Y%2E%2C+Kawahara%2C+D%2E%2C+and+Kurohashi%2C+S%2E+%282018%29%2E+Improving+crowdsourcing%2Dbased+annotation+of+Japanese+discourse+relations%2E+In+Proceedings+of+the+Eleventh+International+)
12. Knaebel, R. and Stede, M. (2020). Contextualized embeddings for connective disambiguation in shallow discourse parsing. In Proceedings of the First Workshop on Computational Approaches to Discourse (CoDi 2020), pages 65–75, Online, November. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Knaebel%2C+R%2E+and+Stede%2C+M%2E+%282020%29%2E+Contextualized+embeddings+for+connective+disambiguation+in+shallow+discourse+parsing%2E+In+Proceedings+of+the+First+Workshop+on+Computational+Approaches+to+Discourse+%28C)
13. Knott, A. (1996). A data-driven methodology for motivating a set of coherence relations. Ph.D. thesis, The University of Edinburgh: College of Science and Engineering. [find](https://scholar.google.com/scholar?q=Knott%2C+A%2E+%281996%29%2E+A+data%2Ddriven+methodology+for+motivating+a+set+of+coherence+relations%2E+Ph%2ED%2E+thesis%2C+The+University+of+Edinburgh%3A+College+of+Science+and+Engineering%2E)
14. Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. In Proceedings of MT Summit X, pages 79–86, Phuket, Thailand. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E+%282005%29%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+Proceedings+of+MT+Summit+X%2C+pages+79%E2%80%9386%2C+Phuket%2C+Thailand%2E)
15. Lan, M., Wang, J., Wu, Y., Niu, Z.-Y., and Wang, H. (2017). Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1299–1308, Copenhagen, Denmark, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lan%2C+M%2E%2C+Wang%2C+J%2E%2C+Wu%2C+Y%2E%2C+Niu%2C+Z%2E%2DY%2E%2C+and+Wang%2C+H%2E+%282017%29%2E+Multi%2Dtask+attention%2Dbased+neural+networks+for+implicit+discourse+relationship+representation+and+identification%2E+In+Proceedings+of+the+2017)
16. Liang, L., Zhao, Z., and Webber, B. (2020). Extending implicit discourse relation recognition to the PDTB-3. In Proceedings of the First Workshop on Computational Approaches to Discourse, pages 135–147. [find](https://scholar.google.com/scholar?q=Liang%2C+L%2E%2C+Zhao%2C+Z%2E%2C+and+Webber%2C+B%2E+%282020%29%2E+Extending+implicit+discourse+relation+recognition+to+the+PDTB%2D3%2E+In+Proceedings+of+the+First+Workshop+on+Computational+Approaches+to+Discourse%2C+pages+135%E2%80%9314)
17. Lin, Z., Ng, H. T., and Kan, M.-Y. (2014). A PDTB-styled end-to-end discourse parser. Natural Language Engineering, 20(2):151–184. [find](https://scholar.google.com/scholar?q=Lin%2C+Z%2E%2C+Ng%2C+H%2E+T%2E%2C+and+Kan%2C+M%2E%2DY%2E+%282014%29%2E+A+PDTB%2Dstyled+end%2Dto%2Dend+discourse+parser%2E+Natural+Language+Engineering%2C+20%282%29%3A151%E2%80%93184%2E)
18. Long, W., Webber, B., and Xiong, D. (2020). Ted-cdb: A large-scale chinese discourse relation dataset on ted talks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2793–2803. [find](https://scholar.google.com/scholar?q=Long%2C+W%2E%2C+Webber%2C+B%2E%2C+and+Xiong%2C+D%2E+%282020%29%2E+Ted%2Dcdb%3A+A+large%2Dscale+chinese+discourse+relation+dataset+on+ted+talks%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
19. Passonneau, R. J. and Carpenter, B. (2014). The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311–326. [find](https://scholar.google.com/scholar?q=Passonneau%2C+R%2E+J%2E+and+Carpenter%2C+B%2E+%282014%29%2E+The+benefits+of+a+model+of+annotation%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+2%3A311%E2%80%93326%2E)
20. Pitler, E. and Nenkova, A. (2009). Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 13–16, Suntec, Singapore, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pitler%2C+E%2E+and+Nenkova%2C+A%2E+%282009%29%2E+Using+syntax+to+disambiguate+explicit+discourse+connectives+in+text%2E+In+Proceedings+of+the+ACL%2DIJCNLP+2009+Conference+Short+Papers%2C+pages+13%E2%80%9316%2C+Suntec%2C+Singapore%2C+A) [our card](https://md.mova.elamur.ai/free/en/docs/papers/pitler-2009-syntax-disambiguate-explicit-discourse-connectives/)
21. Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A., and Webber, B. (2008). The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco, May. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Prasad%2C+R%2E%2C+Dinesh%2C+N%2E%2C+Lee%2C+A%2E%2C+Miltsakaki%2C+E%2E%2C+Robaldo%2C+L%2E%2C+Joshi%2C+A%2E%2C+and+Webber%2C+B%2E+%282008%29%2E+The+Penn+Discourse+TreeBank+2%2E0%2E+In+Proceedings+of+the+Sixth+International+Conference+on+Language+Resour) [our card](https://md.mova.elamur.ai/free/en/docs/papers/disc-2008-prasad-penn-discourse-treebank-2-0/)
22. Prasad, R., McRoy, S., Frid, N., Joshi, A., and Yu, H. (2011). The biomedical discourse relation bank. BMC bioinformatics, 12(1):1–18. [find](https://scholar.google.com/scholar?q=Prasad%2C+R%2E%2C+McRoy%2C+S%2E%2C+Frid%2C+N%2E%2C+Joshi%2C+A%2E%2C+and+Yu%2C+H%2E+%282011%29%2E+The+biomedical+discourse+relation+bank%2E+BMC+bioinformatics%2C+12%281%29%3A1%E2%80%9318%2E)
23. Pusse, F., Sayeed, A., and Demberg, V. (2016). Lingo-Turk: Managing crowdsourced tasks for psycholinguistics. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 57–61, San Diego, CA. [find](https://scholar.google.com/scholar?q=Pusse%2C+F%2E%2C+Sayeed%2C+A%2E%2C+and+Demberg%2C+V%2E+%282016%29%2E+Lingo%2DTurk%3A+Managing+crowdsourced+tasks+for+psycholinguistics%2E+In+Proceedings+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguist)
24. Pyatkin, V., Klein, A., Tsarfaty, R., and Dagan, I. (2020). QADiscourse-Discourse Relations as QA Pairs: Representation, crowdsourcing and baselines. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2804–2819. [find](https://scholar.google.com/scholar?q=Pyatkin%2C+V%2E%2C+Klein%2C+A%2E%2C+Tsarfaty%2C+R%2E%2C+and+Dagan%2C+I%2E+%282020%29%2E+QADiscourse%2DDiscourse+Relations+as+QA+Pairs%3A+Representation%2C+crowdsourcing+and+baselines%2E+In+Proceedings+of+the+2020+Conference+on+Empirical)
25. Rehbein, I., Scholman, M. C. J., and Demberg, V. (2016). Annotating discourse relations in spoken language: A comparison of the PDTB and CCR frameworks. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1039–1046, Portorož, Slovenia, May. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Rehbein%2C+I%2E%2C+Scholman%2C+M%2E+C%2E+J%2E%2C+and+Demberg%2C+V%2E+%282016%29%2E+Annotating+discourse+relations+in+spoken+language%3A+A+comparison+of+the+PDTB+and+CCR+frameworks%2E+In+Proceedings+of+the+Tenth+International+Confe)
26. Rohde, H., Dickinson, A., Schneider, N., Clark, C., Louis, A., and Webber, B. (2016). Filling in the blanks in understanding discourse adverbials: Consistency, conflict, and context-dependence in a crowdsourced elicitation task. In Proceedings of the 10th Linguistic Annotation Workshop (LAW X), pages 49–58, Berlin, Germany. [find](https://scholar.google.com/scholar?q=Rohde%2C+H%2E%2C+Dickinson%2C+A%2E%2C+Schneider%2C+N%2E%2C+Clark%2C+C%2E%2C+Louis%2C+A%2E%2C+and+Webber%2C+B%2E+%282016%29%2E+Filling+in+the+blanks+in+understanding+discourse+adverbials%3A+Consistency%2C+conflict%2C+and+context%2Ddependence+in+a+cr)
27. Sanders, T. J. M., Spooren, W. P. M. S., and Noordman, L. G. M. (1992). Toward a taxonomy of coherence relations. Discourse Processes, 15(1):1–35. [find](https://scholar.google.com/scholar?q=Sanders%2C+T%2E+J%2E+M%2E%2C+Spooren%2C+W%2E+P%2E+M%2E+S%2E%2C+and+Noordman%2C+L%2E+G%2E+M%2E+%281992%29%2E+Toward+a+taxonomy+of+coherence+relations%2E+Discourse+Processes%2C+15%281%29%3A1%E2%80%9335%2E)
28. Scholman, M. C. J. and Demberg, V. (2017a). Crowdsourcing discourse interpretations: On the influence of context and the reliability of a connective insertion task. In Proceedings of the 11th Linguistic Annotation Workshop (LAW), pages 24–33, Valencia, Spain. [find](https://scholar.google.com/scholar?q=Scholman%2C+M%2E+C%2E+J%2E+and+Demberg%2C+V%2E+%282017a%29%2E+Crowdsourcing+discourse+interpretations%3A+On+the+influence+of+context+and+the+reliability+of+a+connective+insertion+task%2E+In+Proceedings+of+the+11th+Linguist)
29. Scholman, M. C. J. and Demberg, V. (2017b). Examples and specifications that prove a point: Identifying elaborative and argumentative discourse relations. Dialogue & Discourse, 8(2):56–83. [find](https://scholar.google.com/scholar?q=Scholman%2C+M%2E+C%2E+J%2E+and+Demberg%2C+V%2E+%282017b%29%2E+Examples+and+specifications+that+prove+a+point%3A+Identifying+elaborative+and+argumentative+discourse+relations%2E+Dialogue+%26+Discourse%2C+8%282%29%3A56%E2%80%9383%2E)
30. Scholman, M. C. J., Dong, T., Yung, F., and Demberg, V. (2021). Comparison of methods for explicit discourse connective identification across various domains. In Proceedings of the First Workshop on Computational Approaches to Discourse (CoDi 2020). [find](https://scholar.google.com/scholar?q=Scholman%2C+M%2E+C%2E+J%2E%2C+Dong%2C+T%2E%2C+Yung%2C+F%2E%2C+and+Demberg%2C+V%2E+%282021%29%2E+Comparison+of+methods+for+explicit+discourse+connective+identification+across+various+domains%2E+In+Proceedings+of+the+First+Workshop+on+C)
31. Scholman, M. C. J., Pyatkin, V., Yung, F., Dagan, I., Tsarfaty, R., and Demberg, V. (2022). Design choices in crowdsourcing discourse relation annotations: The effect of worker selection and training. In Proceedings of the Thirteenth International Conference on Language Resources and Evaluation (LREC’22), Marseille, France. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Scholman%2C+M%2E+C%2E+J%2E%2C+Pyatkin%2C+V%2E%2C+Yung%2C+F%2E%2C+Dagan%2C+I%2E%2C+Tsarfaty%2C+R%2E%2C+and+Demberg%2C+V%2E+%282022%29%2E+Design+choices+in+crowdsourcing+discourse+relation+annotations%3A+The+effect+of+worker+selection+and+training%2E)
32. Shi, W. and Demberg, V. (2019a). Learning to explicitate connectives with Seq2Seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 188–199, Gothenburg, Sweden, May. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shi%2C+W%2E+and+Demberg%2C+V%2E+%282019a%29%2E+Learning+to+explicitate+connectives+with+Seq2Seq+network+for+implicit+discourse+relation+classification%2E+In+Proceedings+of+the+13th+International+Conference+on+Computa)
33. Shi, W. and Demberg, V. (2019b). Next sentence prediction helps implicit discourse relation classification within and across domains. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 5790–5796. [find](https://scholar.google.com/scholar?q=Shi%2C+W%2E+and+Demberg%2C+V%2E+%282019b%29%2E+Next+sentence+prediction+helps+implicit+discourse+relation+classification+within+and+across+domains%2E+In+Proceedings+of+the+2019+conference+on+empirical+methods+in+natu)
34. Spooren, W. P. M. S. and Degand, L. (2010). Coding coherence relations: Reliability and validity. Corpus Linguistics and Linguistic Theory, 6(2):241–266. [find](https://scholar.google.com/scholar?q=Spooren%2C+W%2E+P%2E+M%2E+S%2E+and+Degand%2C+L%2E+%282010%29%2E+Coding+coherence+relations%3A+Reliability+and+validity%2E+Corpus+Linguistics+and+Linguistic+Theory%2C+6%282%29%3A241%E2%80%93266%2E)
35. Webber, B., Prasad, R., Lee, A., and Joshi, A. (2019). The Penn Discourse Treebank 3.0 annotation manual. Philadelphia, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=Webber%2C+B%2E%2C+Prasad%2C+R%2E%2C+Lee%2C+A%2E%2C+and+Joshi%2C+A%2E+%282019%29%2E+The+Penn+Discourse+Treebank+3%2E0+annotation+manual%2E+Philadelphia%2C+University+of+Pennsylvania%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/miltsakaki-2004-penn-discourse-treebank/)
36. Webber, B. (2009). Genre distinctions for discourse in the Penn TreeBank. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 674– 682. [find](https://scholar.google.com/scholar?q=Webber%2C+B%2E+%282009%29%2E+Genre+distinctions+for+discourse+in+the+Penn+TreeBank%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+ACL+and+the+4th+International+Joint+Conference+on+Nat)
37. Xue, N. (2005). Annotating discourse connectives in the Chinese Treebank. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 84–91, Ann Arbor, MI. [find](https://scholar.google.com/scholar?q=Xue%2C+N%2E+%282005%29%2E+Annotating+discourse+connectives+in+the+Chinese+Treebank%2E+In+Proceedings+of+the+Workshop+on+Frontiers+in+Corpus+Annotations+II%3A+Pie+in+the+Sky%2C+pages+84%E2%80%9391%2C+Ann+Arbor%2C+MI%2E)
38. Yung, F., Demberg, V., and Scholman, M. C. J. (2019). Crowdsourcing discourse relation annotations by a two-step connective insertion task. In Proceedings of the 13th Linguistic Annotation Workshop, pages 16–25. [find](https://scholar.google.com/scholar?q=Yung%2C+F%2E%2C+Demberg%2C+V%2E%2C+and+Scholman%2C+M%2E+C%2E+J%2E+%282019%29%2E+Crowdsourcing+discourse+relation+annotations+by+a+two%2Dstep+connective+insertion+task%2E+In+Proceedings+of+the+13th+Linguistic+Annotation+Workshop%2C+p)
39. Zeldes, A. (2017). The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. [find](https://scholar.google.com/scholar?q=Zeldes%2C+A%2E+%282017%29%2E+The+GUM+corpus%3A+Creating+multilayer+resources+in+the+classroom%2E+Language+Resources+and+Evaluation%2C+51%283%29%3A581%E2%80%93612%2E)
40. Zeyrek, D., Mendes, A., Grishina, Y., Kurfalı, M., Gibbon, S., and Ogrodniczuk, M. (2019). TED Multilingual Discourse Bank (TED-MDB): a parallel corpus annotated in the PDTB style. Language Resources and Evaluation, pages 1–27. [find](https://scholar.google.com/scholar?q=Zeyrek%2C+D%2E%2C+Mendes%2C+A%2E%2C+Grishina%2C+Y%2E%2C+Kurfal%C4%B1%2C+M%2E%2C+Gibbon%2C+S%2E%2C+and+Ogrodniczuk%2C+M%2E+%282019%29%2E+TED+Multilingual+Discourse+Bank+%28TED%2DMDB%29%3A+a+parallel+corpus+annotated+in+the+PDTB+style%2E+Language+Resources+)