[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/kubler-2011-singletons-coreference-resolution-evaluation/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Singletons and Coreference Resolution Evaluation

# Singletons and Coreference Resolution Evaluation

**Authors:** [Sandra Kübler](https://md.mova.elamur.ai/free/en/docs/papers/authors/sandra-kubler/), Desislava Zhekova · **Year:** 2011 · **Venue:** Proceedings of the International Conference Recent Advances in Natural Language Processing (RANLP) 2011 **Link:** [aclanthology.org/R11-1036](https://aclanthology.org/R11-1036/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, before 2016) — summary only

[**11** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/kubler-2011-singletons-coreference-resolution-evaluation/) of the 140673 papers whose bibliographies we read[Proceedings of the International Conference Recent Advances in Natural Language Processing 2011](https://aclanthology.org/volumes/R11-1/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/R11-1036/)
- [PDF](https://aclanthology.org/R11-1036.pdf)

**Authors (2)**

1. Sandra Kübler
2. Desislava Zhekova

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

The paper asks how singletons (mentions that are never coreferent with another mention) change coreference scores. It uses the English data of SemEval-2010 Task 1 and CoNLL-2011, which differ in whether singletons are annotated, and the two official scorers of those tasks. The system is UBIU, a memory-based mention-pair model. Four settings are compared: singletons kept or removed in the gold standard, and kept or removed in the system output. Singletons in the gold standard raise the scores, and the authors show that the metrics behave very differently. MUC ignores singletons; B3, CEAF and BLANC react to them, and the SemEval and CoNLL scorers differ in how they treat singletons in the system output. A trivial baseline that puts every mention in its own chain already reaches high CEAF and B3 scores on the SemEval English data (the paper's Table 1, singleton column: CEAF F1 71.2, B3 F1 83.2, MUC F1 0.0). The conclusion is that scores are not comparable unless the singleton policy is stated and matched.

## Ideas

- I1. Make the singleton policy an explicit setting in `coref/src/eval.rs`: report the four settings (gold with or without singletons, system with or without) side by side, and refuse to compare two runs whose settings differ.
- I2. Keep a trivial baseline in every evaluation table: all-singletons and all-in-one, as in the paper, so that a score is read against what a system that links nothing would get.
- I3. Twinless mentions: when the sieve proposes a mention that is not in the gold set, log it as a count, not as a silent drop, so that a gain from fewer mentions is visible.

## For Mova

Code: `coref/src/eval.rs` (scorer settings and the baseline rows), `coref/src/mention.rs` (mention extraction that over-generates NPs). Chapter: [35. Reference and coreference](https://md.mova.elamur.ai/free/en/docs/languages/en/35-reference-and-coreference/). Measure: the sieve's CoNLL-style scores under the four singleton settings, with the all-singletons row, run twice to check the output is identical.

Topics: [Coreference resolution](https://md.mova.elamur.ai/free/en/docs/papers/topics/coreference-resolution/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{kubler2011singletons,
  title = {{Singletons and Coreference Resolution Evaluation}},
  author = {Kübler, Sandra and Zhekova, Desislava},
  booktitle = {Proceedings of the International Conference Recent Advances in Natural Language Processing 2011},
  year = {2011},
  url = {https://aclanthology.org/R11-1036/}
}
```

- [← previous A Grammar of the Dialect of West Somerset, Descriptive and Historical](https://md.mova.elamur.ai/free/en/docs/papers/kruisinga-1905-grammar-dialect-west-somerset/)
- [next → Further Developments in Treebank Error Detection Using Derivation Trees](https://md.mova.elamur.ai/free/en/docs/papers/kulick-2012-further-developments-treebank-error-detection/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the LREC Workshop on Linguistic Coreference, Granada, Spain. [find](https://scholar.google.com/scholar?q=Amit+Bagga+and+Breck+Baldwin%2E+1998%2E+Algorithms+for+scoring+coreference+chains%2E+In+Proceedings+of+the+LREC+Workshop+on+Linguistic+Coreference%2C+Granada%2C+Spain%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bagga-1998-algorithms-scoring-coreference-chains/)
2. Walter Daelemans, Jakub Zavrel, Ko van der Sloot, and Antal van den Bosch. 2007. TiMBL: Tilburg memory based learner – version 6.1 – reference guide. Technical Report ILK 07-07, ILK-CL, Tilburg University. [find](https://scholar.google.com/scholar?q=Walter+Daelemans%2C+Jakub+Zavrel%2C+Ko+van+der+Sloot%2C+and+Antal+van+den+Bosch%2E+2007%2E+TiMBL%3A+Tilburg+memory+based+learner+%E2%80%93+version+6%2E1+%E2%80%93+reference+guide%2E+Technical+Report+ILK+07%2D07%2C+ILK%2DCL%2C+Tilburg+Univer)
3. Pascal Denis and Jason Baldridge. 2007. Joint determination of anaphoricity and coreference resolution using integer programming. In Proceedings of HLT-NAACL 2007, Rochester, NY. [find](https://scholar.google.com/scholar?q=Pascal+Denis+and+Jason+Baldridge%2E+2007%2E+Joint+determination+of+anaphoricity+and+coreference+resolution+using+integer+programming%2E+In+Proceedings+of+HLT%2DNAACL+2007%2C+Rochester%2C+NY%2E)
4. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90% solution. In Proceedings of HLT-NAACL, New York, NY. [find](https://scholar.google.com/scholar?q=Eduard+Hovy%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Lance+Ramshaw%2C+and+Ralph+Weischedel%2E+2006%2E+Ontonotes%3A+The+90%25+solution%2E+In+Proceedings+of+HLT%2DNAACL%2C+New+York%2C+NY%2E)
5. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proceedings of ACL, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Xiaoqiang+Luo%2C+Abe+Ittycheriah%2C+Hongyan+Jing%2C+Nanda+Kambhatla%2C+and+Salim+Roukos%2E+2004%2E+A+mention%2Dsynchronous+coreference+resolution+algorithm+based+on+the+bell+tree%2E+In+Proceedings+of+ACL%2C+Barcelona%2C+)
6. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of HLT-EMNLP, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Xiaoqiang+Luo%2E+2005%2E+On+coreference+resolution+performance+metrics%2E+In+Proceedings+of+HLT%2DEMNLP%2C+Vancouver%2C+Canada%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/luo-2005-coreference-resolution-performance-metrics/)
7. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of CoNLL 2011, Portland, OR. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Lance+Ramshaw%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Ralph+Weischedel%2C+and+Nianwen+Xue%2E+2011%2E+CoNLL%2D2011+shared+task%3A+Modeling+unrestricted+coreference+in+OntoNotes%2E+In+Proceedings+of+CoNLL+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/pradhan-2011-conll-2011-shared-task-modeling/)
8. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of EMNLP, Singapore. [find](https://scholar.google.com/scholar?q=Altaf+Rahman+and+Vincent+Ng%2E+2009%2E+Supervised+models+for+coreference+resolution%2E+In+Proceedings+of+EMNLP%2C+Singapore%2E)
9. Altaf Rahman and Vincent Ng. 2011. Narrowing the modeling gap: A cluster-ranking approach to coreference resolution. JAIR, 40:469–521. [find](https://scholar.google.com/scholar?q=Altaf+Rahman+and+Vincent+Ng%2E+2011%2E+Narrowing+the+modeling+gap%3A+A+cluster%2Dranking+approach+to+coreference+resolution%2E+JAIR%2C+40%3A469%E2%80%93521%2E)
10. Marta Recasens and Eduard Hovy. 2011. BLANC: Implementing the Rand Index for coreference evaluation. Natural Language Engineering. [find](https://scholar.google.com/scholar?q=Marta+Recasens+and+Eduard+Hovy%2E+2011%2E+BLANC%3A+Implementing+the+Rand+Index+for+coreference+evaluation%2E+Natural+Language+Engineering%2E)
11. Marta Recasens, Lluı̀s Márquez, Emili Sapena, M. Antònia Martı́, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. SemEval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of SemEval, Uppsala, Sweden. [find](https://scholar.google.com/scholar?q=Marta+Recasens%2C+Llu%C4%B1%CC%80s+Ma%CC%81rquez%2C+Emili+Sapena%2C+M%2E+Anto%CC%80nia+Mart%C4%B1%CC%81%2C+Mariona+Taule%CC%81%2C+Ve%CC%81ronique+Hoste%2C+Massimo+Poesio%2C+and+Yannick+Versley%2E+2010%2E+SemEval%2D2010+task+1%3A+Coreference+resolution+in+multiple+)
12. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL-AFNLP, Singapore. [find](https://scholar.google.com/scholar?q=Veselin+Stoyanov%2C+Nathan+Gilbert%2C+Claire+Cardie%2C+and+Ellen+Riloff%2E+2009%2E+Conundrums+in+noun+phrase+coreference+resolution%3A+Making+sense+of+the+state%2Dof%2Dthe%2Dart%2E+In+Proceedings+of+ACL%2DAFNLP%2C+Singapore%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/stoyanov-2009-conundrums-noun-phrase-coreference-resolution/)
13. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of MUC-6, Columbia, MD. [find](https://scholar.google.com/scholar?q=Marc+Vilain%2C+John+Burger%2C+John+Aberdeen%2C+Dennis+Connolly%2C+and+Lynette+Hirschman%2E+1995%2E+A+model%2Dtheoretic+coreference+scoring+scheme%2E+In+Proceedings+of+MUC%2D6%2C+Columbia%2C+MD%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/vilain-1995-model-theoretic-coreference-scoring-scheme/)
14. Holger Wunsch. 2009. Rule-Based and Memory-Based Pronoun Resolution for German: A Comparison and Assessment of Data Sources. Ph.D. thesis, Universität Tübingen. [find](https://scholar.google.com/scholar?q=Holger+Wunsch%2E+2009%2E+Rule%2DBased+and+Memory%2DBased+Pronoun+Resolution+for+German%3A+A+Comparison+and+Assessment+of+Data+Sources%2E+Ph%2ED%2E+thesis%2C+Universita%CC%88t+Tu%CC%88bingen%2E)
15. Desislava Zhekova and Sandra Kübler. 2010. UBIU: A language-independent system for coreference resolution. In Proceedings of SemEval, Uppsala, Sweden. [find](https://scholar.google.com/scholar?q=Desislava+Zhekova+and+Sandra+Ku%CC%88bler%2E+2010%2E+UBIU%3A+A+language%2Dindependent+system+for+coreference+resolution%2E+In+Proceedings+of+SemEval%2C+Uppsala%2C+Sweden%2E)
16. Desislava Zhekova and Sandra Kübler. 2011. UBIU: A Robust System for Resolving Unrestricted Coreference. In Proceedings of CoNLL: Shared Task, Portland, OR. [find](https://scholar.google.com/scholar?q=Desislava+Zhekova+and+Sandra+Ku%CC%88bler%2E+2011%2E+UBIU%3A+A+Robust+System+for+Resolving+Unrestricted+Coreference%2E+In+Proceedings+of+CoNLL%3A+Shared+Task%2C+Portland%2C+OR%2E)