[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/pradhan-2014-scoring-coreference-partitions-predicted-mentions/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation

# Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation

**Authors:** [Sameer Pradhan](https://md.mova.elamur.ai/free/en/docs/papers/authors/sameer-pradhan/), [Xiaoqiang Luo](https://md.mova.elamur.ai/free/en/docs/papers/authors/xiaoqiang-luo/), [Marta Recasens](https://md.mova.elamur.ai/free/en/docs/papers/authors/marta-recasens/), [Eduard Hovy](https://md.mova.elamur.ai/free/en/docs/papers/authors/eduard-hovy/), [Vincent Ng](https://md.mova.elamur.ai/free/en/docs/papers/authors/vincent-ng/), [Michael Strube](https://md.mova.elamur.ai/free/en/docs/papers/authors/michael-strube/) · **Year:** 2014 · **Venue:** Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) **Link:** [aclanthology.org/P14-2006](https://aclanthology.org/P14-2006/) (ACL Anthology P14-2006; DOI [10.3115/v1/P14-2006](https://doi.org/10.3115/v1/P14-2006)) **License of the paper:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, before 2016) — [creativecommons.org/licenses/by-nc-sa/3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/)

[**175** citations](https://openalex.org/works?filter=cites:W2250668331) openalex · 2026-10-18[**110** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/pradhan-2014-scoring-coreference-partitions-predicted-mentions/) of the 140673 papers whose bibliographies we read[Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://aclanthology.org/volumes/P14-2/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/P14-2006/)
- [PDF](https://aclanthology.org/P14-2006.pdf)
- [DOI 10.3115/v1/p14-2006](https://doi.org/10.3115/v1/p14-2006)
- [OpenAlex](https://openalex.org/W2250668331)

**Authors (6)**

1. [Sameer Pradhan](https://openalex.org/A5110598639) [ORCID](https://orcid.org/0000-0002-5537-2181) · h 31 · 5311 citations · 87 works
2. [Xiaoqiang Luo](https://openalex.org/A5111895886) h 15 · 1631 citations · 39 works
3. [Marta Recasens](https://openalex.org/A5103978745) h 19 · 2106 citations · 72 works
4. [Eduard Hovy](https://openalex.org/A5060225743) [ORCID](https://orcid.org/0000-0002-3270-7903) · h 94 · 45015 citations · 690 works
5. [Vincent Ng](https://openalex.org/A5102732054) [ORCID](https://orcid.org/0000-0001-8237-429X) · h 34 · 3982 citations · 136 works
6. [Michael Strube](https://openalex.org/A5020347604) [ORCID](https://orcid.org/0000-0002-4731-8142) · h 74 · 20807 citations · 485 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## Summary

Coreference metrics such as MUC, B-cubed and CEAF were originally defined for gold mentions, and different research groups adapted them to predicted mentions in incompatible ways, often by adding or removing mentions artificially so that the key and response partitions line up. This made published scores hard to compare. The paper states how each metric should be computed directly on partitions with predicted mentions, without such manipulations: a mention present in only one partition simply contributes nothing where it has no counterpart. It walks through worked examples and describes a reference implementation of the scorer that became the standard tool for CoNLL-style evaluation. The aim is a single, transparent definition so that results across systems and papers are comparable.

## How Mova uses it

- `coref/src/score.rs` — the MUC, B-cubed, CEAF-e and CEAF-m formulas for predicted mentions follow this paper: no mention manipulation, unmatched mentions just do not contribute; CoNLL F1 is the mean of MUC, B-cubed and CEAF-e, and multi-document totals sum numerators and denominators.
- The worked example from the paper's Section 4 (key {a,b,c}{d,e,f,g}, response {a,b}{c,d}{f,g,h,i}) is a unit test of the Rust scorer, checking MUC, B-cubed, CEAF-e and CEAF-m values. The test documents that the paper's printed B-cubed F1 of 0.46 comes from rounded precision and recall; the exact value is 0.4545.

## Effectiveness in Mova

Not an idea that changes scores, so not measured as a gain. Its role is correctness of the measuring instrument: the scorer reproduces the paper's worked example, and every reported coreference result (for example CoNLL F1 73.04 with singletons and 60.31 without on GUM test, gold trees) depends on it.

---

👨‍🔬💥

Topics: [Coreference resolution](https://md.mova.elamur.ai/free/en/docs/papers/topics/coreference-resolution/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/v1/p14-2006](https://doi.org/10.3115/v1/p14-2006)

```
@inproceedings{pradhan2014scoring,
  title = {{Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation}},
  author = {Pradhan, Sameer and Luo, Xiaoqiang and Recasens, Marta and Hovy, Eduard and Ng, Vincent and Strube, Michael},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-2006},
  url = {https://doi.org/10.3115/v1/p14-2006}
}
```

- [← previous The Characters of the English Verb (1921); The Infinitive, the Gerund and the Participles of the English Verb (1923)](https://md.mova.elamur.ai/free/en/docs/papers/poutsma-1921-characters-english-verb/)
- [next → Conjunct Lengths in English, Dependency Length Minimization, and Dependency Structure of Coordination](https://md.mova.elamur.ai/free/en/docs/papers/przepiorkowski-2023-conjunct-lengths-english-dependency-length/)
- [↑ back to papers](https://md.mova.elamur.ai/free/en/docs/papers/)

## References

24 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of LREC, pages 563–566. [find](https://scholar.google.com/scholar?q=Amit+Bagga+and+Breck+Baldwin%2E+1998%2E+Algorithms+for+scoring+coreference+chains%2E+In+Proceedings+of+LREC%2C+pages+563%E2%80%93566%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bagga-1998-algorithms-scoring-coreference-chains/)
2. Jie Cai and Michael Strube. 2010. Evaluation metrics for end-to-end coreference resolution systems. In Proceedings of SIGDIAL, pages 28–36. [find](https://scholar.google.com/scholar?q=Jie+Cai+and+Michael+Strube%2E+2010%2E+Evaluation+metrics+for+end%2Dto%2Dend+coreference+resolution+systems%2E+In+Proceedings+of+SIGDIAL%2C+pages+28%E2%80%9336%2E)
3. Chen Chen and Vincent Ng. 2013. Linguistically aware coreference evaluation metrics. In Proceedings of the Sixth IJCNLP, pages 1366–1374, Nagoya, Japan, October. [find](https://scholar.google.com/scholar?q=Chen+Chen+and+Vincent+Ng%2E+2013%2E+Linguistically+aware+coreference+evaluation+metrics%2E+In+Proceedings+of+the+Sixth+IJCNLP%2C+pages+1366%E2%80%931374%2C+Nagoya%2C+Japan%2C+October%2E)
4. Nancy Chinchor and Beth Sundheim. 2003. Message understanding conference (MUC) 6. In LDC2003T13. [find](https://scholar.google.com/scholar?q=Nancy+Chinchor+and+Beth+Sundheim%2E+2003%2E+Message+understanding+conference+%28MUC%29+6%2E+In+LDC2003T13%2E)
5. Nancy Chinchor. 2001. Message understanding conference (MUC) 7. In LDC2001T02. [find](https://scholar.google.com/scholar?q=Nancy+Chinchor%2E+2001%2E+Message+understanding+conference+%28MUC%29+7%2E+In+LDC2001T02%2E)
6. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ACE) program-tasks, data, and evaluation. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=George+Doddington%2C+Alexis+Mitchell%2C+Mark+Przybocki%2C+Lance+Ramshaw%2C+Stephanie+Strassel%2C+and+Ralph+Weischedel%2E+2004%2E+The+automatic+content+extraction+%28ACE%29+program%2Dtasks%2C+data%2C+and+evaluation%2E+In+Procee)
7. Lynette Hirschman and Nancy Chinchor. 1997. Coreference task definition (v3.0, 13 jul 97). In Proceedings of the 7th Message Understanding Conference. [find](https://scholar.google.com/scholar?q=Lynette+Hirschman+and+Nancy+Chinchor%2E+1997%2E+Coreference+task+definition+%28v3%2E0%2C+13+jul+97%29%2E+In+Proceedings+of+the+7th+Message+Understanding+Conference%2E)
8. Gordana Ilic Holen. 2013. Critical reflections on evaluation practices in coreference resolution. In Proceedings of the NAACL-HLT Student Research Workshop, pages 1–7, Atlanta, Georgia, June. [find](https://scholar.google.com/scholar?q=Gordana+Ilic+Holen%2E+2013%2E+Critical+reflections+on+evaluation+practices+in+coreference+resolution%2E+In+Proceedings+of+the+NAACL%2DHLT+Student+Research+Workshop%2C+pages+1%E2%80%937%2C+Atlanta%2C+Georgia%2C+June%2E)
9. Daniel Jurafsky and James H. Martin. 2008. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall. Second Edition. [find](https://scholar.google.com/scholar?q=Daniel+Jurafsky+and+James+H%2E+Martin%2E+2008%2E+Speech+and+Language+Processing%3A+An+Introduction+to+Natural+Language+Processing%2C+Computational+Linguistics%2C+and+Speech+Recognition%2E+Prentice+Hall%2E+Second+Edit)
10. Xiaoqiang Luo, Sameer Pradhan, Marta Recasens, and Eduard Hovy. 2014. An extension of BLANC to system mentions. In Proceedings of ACL, Baltimore, Maryland, June. [find](https://scholar.google.com/scholar?q=Xiaoqiang+Luo%2C+Sameer+Pradhan%2C+Marta+Recasens%2C+and+Eduard+Hovy%2E+2014%2E+An+extension+of+BLANC+to+system+mentions%2E+In+Proceedings+of+ACL%2C+Baltimore%2C+Maryland%2C+June%2E)
11. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of HLT-EMNLP, pages 25–32. [find](https://scholar.google.com/scholar?q=Xiaoqiang+Luo%2E+2005%2E+On+coreference+resolution+performance+metrics%2E+In+Proceedings+of+HLT%2DEMNLP%2C+pages+25%E2%80%9332%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/luo-2005-coreference-resolution-performance-metrics/)
12. Sameer Pradhan, Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2007. OntoNotes: A Unified Relational Semantic Representation. International Journal of Semantic Computing, 1(4):405–419. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Eduard+Hovy%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Lance+Ramshaw%2C+and+Ralph+Weischedel%2E+2007%2E+OntoNotes%3A+A+Unified+Relational+Semantic+Representation%2E+International+Journal+of+Semantic+Compu)
13. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of CoNLL: Shared Task, pages 1–27. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Lance+Ramshaw%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Ralph+Weischedel%2C+and+Nianwen+Xue%2E+2011%2E+CoNLL%2D2011+shared+task%3A+Modeling+unrestricted+coreference+in+OntoNotes%2E+In+Proceedings+of+CoNLL%3A) [our card](https://md.mova.elamur.ai/free/en/docs/papers/pradhan-2011-conll-2011-shared-task-modeling/)
14. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of CoNLL: Shared Task, pages 1–40. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+2012%2E+CoNLL%2D2012+shared+task%3A+Modeling+multilingual+unrestricted+coreference+in+OntoNotes%2E+In+Proceedings+of+CoNLL%3A+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/pradhan-2012-conll-2012-shared-task-modeling/)
15. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of EMNLP, pages 968–977. [find](https://scholar.google.com/scholar?q=Altaf+Rahman+and+Vincent+Ng%2E+2009%2E+Supervised+models+for+coreference+resolution%2E+In+Proceedings+of+EMNLP%2C+pages+968%E2%80%93977%2E)
16. Altaf Rahman and Vincent Ng. 2011. Coreference resolution with world knowledge. In Proceedings of ACL, pages 814–824. [find](https://scholar.google.com/scholar?q=Altaf+Rahman+and+Vincent+Ng%2E+2011%2E+Coreference+resolution+with+world+knowledge%2E+In+Proceedings+of+ACL%2C+pages+814%E2%80%93824%2E)
17. Marta Recasens and Eduard Hovy. 2011. BLANC: Implementing the Rand Index for coreference evaluation. Natural Language Engineering, 17(4):485– 510. [find](https://scholar.google.com/scholar?q=Marta+Recasens+and+Eduard+Hovy%2E+2011%2E+BLANC%3A+Implementing+the+Rand+Index+for+coreference+evaluation%2E+Natural+Language+Engineering%2C+17%284%29%3A485%E2%80%93+510%2E)
18. Marta Recasens, Lluı́s Màrquez, Emili Sapena, M. Antònia Martı́, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. Semeval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of SemEval, pages 1–8. [find](https://scholar.google.com/scholar?q=Marta+Recasens%2C+Llu%C4%B1%CC%81s+Ma%CC%80rquez%2C+Emili+Sapena%2C+M%2E+Anto%CC%80nia+Mart%C4%B1%CC%81%2C+Mariona+Taule%CC%81%2C+Ve%CC%81ronique+Hoste%2C+Massimo+Poesio%2C+and+Yannick+Versley%2E+2010%2E+Semeval%2D2010+task+1%3A+Coreference+resolution+in+multiple+)
19. Marta Recasens, Marie-Catherine de Marneffe, and Chris Potts. 2013. The life and death of discourse entities: Identifying singleton mentions. In Proceedings of NAACL-HLT, pages 627–633. [find](https://scholar.google.com/scholar?q=Marta+Recasens%2C+Marie%2DCatherine+de+Marneffe%2C+and+Chris+Potts%2E+2013%2E+The+life+and+death+of+discourse+entities%3A+Identifying+singleton+mentions%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+627%E2%80%93633%2E)
20. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL-IJCNLP, pages 656–664. [find](https://scholar.google.com/scholar?q=Veselin+Stoyanov%2C+Nathan+Gilbert%2C+Claire+Cardie%2C+and+Ellen+Riloff%2E+2009%2E+Conundrums+in+noun+phrase+coreference+resolution%3A+Making+sense+of+the+state%2Dof%2Dthe%2Dart%2E+In+Proceedings+of+ACL%2DIJCNLP%2C+pages+656) [our card](https://md.mova.elamur.ai/free/en/docs/papers/stoyanov-2009-conundrums-noun-phrase-coreference-resolution/)
21. William F. Styler, Steven Bethard an Sean Finan, Martha Palmer, Sameer Pradhan, Piet C de Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, and James Pustejovsky. 2014. Temporal annotation in the clinical domain. Transactions of Computational Linguistics, 2(April):143–154. [find](https://scholar.google.com/scholar?q=William+F%2E+Styler%2C+Steven+Bethard+an+Sean+Finan%2C+Martha+Palmer%2C+Sameer+Pradhan%2C+Piet+C+de+Groen%2C+Brad+Erickson%2C+Timothy+Miller%2C+Chen+Lin%2C+Guergana+Savova%2C+and+James+Pustejovsky%2E+2014%2E+Temporal+annotat)
22. Ozlem Uzuner, Andreea Bodnari, Shuying Shen, Tyler Forbush, John Pestian, and Brett R South. 2012. Evaluating the state of the art in coreference resolution for electronic medical records. Journal of American Medical Informatics Association, 19(5), September. [find](https://scholar.google.com/scholar?q=Ozlem+Uzuner%2C+Andreea+Bodnari%2C+Shuying+Shen%2C+Tyler+Forbush%2C+John+Pestian%2C+and+Brett+R+South%2E+2012%2E+Evaluating+the+state+of+the+art+in+coreference+resolution+for+electronic+medical+records%2E+Journal+of+)
23. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference, pages 45–52. [find](https://scholar.google.com/scholar?q=Marc+Vilain%2C+John+Burger%2C+John+Aberdeen%2C+Dennis+Connolly%2C+and+Lynette+Hirschman%2E+1995%2E+A+model+theoretic+coreference+scoring+scheme%2E+In+Proceedings+of+the+6th+Message+Understanding+Conference%2C+pages+4) [our card](https://md.mova.elamur.ai/free/en/docs/papers/vilain-1995-model-theoretic-coreference-scoring-scheme/)
24. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A large training corpus for enhanced processing. In Joseph Olive, Caitlin Christianson, and John McCary, editors, Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation. Springer. [find](https://scholar.google.com/scholar?q=Ralph+Weischedel%2C+Eduard+Hovy%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Robert+Belvin%2C+Sameer+Pradhan%2C+Lance+Ramshaw%2C+and+Nianwen+Xue%2E+2011%2E+OntoNotes%3A+A+large+training+corpus+for+enhanced+processing%2E+In+Josep)