[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2016-zeldes-different-flavors-gum-evaluating-genre/) [home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Different Flavors of GUM: Evaluating Genre and Sentence Type Effects on Multilayer Corpus Annotation Quality # Different Flavors of GUM: Evaluating Genre and Sentence Type Effects on Multilayer Corpus Annotation Quality **Authors:** [Amir Zeldes](https://md.mova.elamur.ai/free/en/docs/papers/authors/amir-zeldes/), Dan Simonson · **Year:** 2016 · **Venue:** Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016) **Link:** [aclanthology.org/W16-1709](https://aclanthology.org/W16-1709/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) [**7** citations](https://openalex.org/works?filter=cites:W2513006088) openalex-snapshot · 2026-10-14[**10** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2016-zeldes-different-flavors-gum-evaluating-genre/) of the 140673 papers whose bibliographies we read[Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)](https://aclanthology.org/volumes/W16-17/)[licence](https://creativecommons.org/licenses/by/4.0/) - [Publisher's page](https://aclanthology.org/W16-1709/) - [PDF](https://aclanthology.org/W16-1709.pdf) - [DOI 10.18653/v1/w16-1709](https://doi.org/10.18653/v1/w16-1709) - [OpenAlex](https://openalex.org/W2513006088) **Authors (2)** 1. [Amir Zeldes](https://openalex.org/A5089212858) [ORCID](https://orcid.org/0000-0001-8016-6753) · h 16 · 1524 citations · 238 works 2. [Dan Simonson](https://openalex.org/A5017060959) h 4 · 28 citations · 8 works Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart. ## What it is about Tagging and parsing accuracy is usually reported on newswire, and genre is the usual explanation when it falls elsewhere. The authors ask whether sentence type (declarative, imperative, question, fragment, and so on) explains annotation quality better than genre. Using the GUM corpus (54 documents, about 44,000 tokens, four genres: news, interviews, travel guides, how-to guides), they compare uncorrected student annotations, the corrected gold, and three automatic taggers and one parser, with mixed-effects models that include document as a random effect and sentence length as a control. Sentence type predicts errors across the three tasks (POS tagging, dependency correction, coreference), and genre mostly disappears once document is in the model. Fragments and non-declarative types are the hardest for the taggers and the parser, and imperatives, which the EWT paper singles out, are not a strong problem. ## Ideas - I1. `en tag-eval` and `en ud-eval` broken down by sentence type (decl, imp, q, wh, frag, sub, inf) and by genre, with a mixed-effects check that sentence type stays significant when document is a random effect; measure UPOS and LAS per sentence type on GUM test (test only). - I2. `en/src/ptag.rs`: a sentence-type feature from the first word and the presence of a finite verb, to help the tagger on fragments and headings; measure UPOS on fragments and gerund headings in GUM test before and after. - I3. `annot` routing: send fragment and other-type sentences to a second teacher pass with a stricter gate, since the paper shows they carry most errors; measure gate violations per 1k silver sentences by sentence type with `treeagree check`. - I4. `en/src/ctx.rs`: document-level features (genre tag or a heading flag) only where they add to the sentence-type model; measure with the mixed-effects check, not by accuracy alone, since the paper finds genre adds nothing once sentence type is in. ## For Mova Touches `languages/en/37-register-and-variation`, `10-word-classes` and `38-dialogue-and-discourse-markers`, and the `en` crate (`ptag.rs`, `ctx.rs`, `parse.rs`), `annot` and `tools/treeagree`. GUM is non-commercial: test only, never in a shipped model. Measure first the per-sentence-type breakdown of the current `en` scores on GUM test, which costs nothing and shows whether fragments are the weak spot. Topics: [Part-of-speech tagging](https://md.mova.elamur.ai/free/en/docs/papers/topics/part-of-speech-tagging/) ## Cite The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/w16-1709](https://doi.org/10.18653/v1/w16-1709) ``` @inproceedings{zeldes2016different, title = {{Different Flavors of GUM: Evaluating Genre and Sentence Type Effects on Multilayer Corpus Annotation Quality}}, author = {Zeldes, Amir and Simonson, Dan}, booktitle = {Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)}, year = {2016}, doi = {10.18653/v1/w16-1709}, url = {https://doi.org/10.18653/v1/w16-1709} } ``` - [← previous Joint Transition-based Dependency Parsing and Disfluency Detection for Automatic Speech Recognition Texts](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2016-yoshikawa-joint-transition-based-dependency-parsing/) - [next → Learning attention for historical text normalization by learning to pronounce](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2017-bollmann-learning-attention-historical-text-normalization/) - [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Emilia Apostolova, Noriko Tomuro, Pattanasak Mongkolwat and Dina Demner-Fushman. 2012. Domain Adaptation of Coreference Resolution for Radiology Reports. In Proceedings of the 2012 Workshop on Biomedical Natural Language Processing (BioNLP 2012). Montreal, 118–121. R. Harald. [find](https://scholar.google.com/scholar?q=Emilia+Apostolova%2C+Noriko+Tomuro%2C+Pattanasak+Mongkolwat+and+Dina+Demner%2DFushman%2E+2012%2E+Domain+Adaptation+of+Coreference+Resolution+for+Radiology+Reports%2E+In+Proceedings+of+the+2012+Workshop+on+Biomedi) 2. Baayen. 2008. Analyzing Linguistic Data. A Practical Introduction to Statistics using R. Cambridge: Cambridge University Press. [↗](https://doi.org/10.1017/cbo9780511801686) 3. Douglas Biber. 2009. Multi-Dimensional Approaches. In Anke Lüdeling & Merja Kytö (eds.), Corpus [find](https://scholar.google.com/scholar?q=Douglas+Biber%2E+2009%2E+Multi%2DDimensional+Approaches%2E+In+Anke+L%C3%BCdeling+%26+Merja+Kyt%C3%B6+%28eds%2E%29%2C+Corpus)