[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/luo-2005-coreference-resolution-performance-metrics/) [home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / On Coreference Resolution Performance Metrics # On Coreference Resolution Performance Metrics **Authors:** [Xiaoqiang Luo](https://md.mova.elamur.ai/free/en/docs/papers/authors/xiaoqiang-luo/) · **Year:** 2005 · **Venue:** Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005) **Link:** [aclanthology.org/H05-1004](https://aclanthology.org/H05-1004/) (ACL Anthology H05-1004) **License of the paper:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, pre-2016) — [creativecommons.org/licenses/by-nc-sa/3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) [**319** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/luo-2005-coreference-resolution-performance-metrics/) of the 140673 papers whose bibliographies we read[Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing](https://aclanthology.org/volumes/H05-1/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/H05-1004/) - [PDF](https://aclanthology.org/H05-1004.pdf) **Authors (1)** 1. Luo, Xiaoqiang Citations and author metrics from OpenAlex (); authors with the same name are not always told apart. ## Summary The paper criticises existing coreference metrics, notably MUC (which ignores singletons and rewards over-merging) and the ACE value, and proposes CEAF (Constrained Entity-Alignment F-measure). CEAF finds the best one-to-one alignment between key and response entities, computed with the Kuhn-Munkres (Hungarian) algorithm, under a chosen entity-similarity function. Recall and precision are the total similarity of the alignment normalised by the self-similarity of the key and of the response respectively. Two similarity functions give the mention-based variant (number of shared mentions) and the entity-based variant (a Dice-like overlap per entity pair). The paper shows that CEAF behaves more sensibly than MUC on degenerate outputs such as putting all mentions in one cluster. CEAF later became one of the three components of the CoNLL coreference score. ## How Mova uses it - `coref/src/score.rs` — Mova's own Rust scorer implements CEAF-e (φ4, entity-based) and CEAF-m (φ3, mention-based), with the optimal entity alignment found by Kuhn-Munkres, computed separately per connected component of the overlap graph. - CEAF-e is one of the three terms of the CoNLL F1 reported for the coref crate (average of MUC, B³ and CEAF-e). - Unit tests check hand-computed CEAF values and the degenerate cases (one giant cluster, all singletons). ## Effectiveness in Mova It is an evaluation metric, not a modelling idea, so it has no "gain" of its own. It is part of every coref measurement, e.g. the default system's CEAF-e F1 on GUM test is 74.68 with singletons and 52.89 without on gold trees (the development notes of `coref`). --- 👨‍🔬💥 Topics: [Coreference resolution](https://md.mova.elamur.ai/free/en/docs/papers/topics/coreference-resolution/) ## Cite The paper itself, not this card. BibTeX from the library's record of the work. ``` @inproceedings{luo2005coreference, title = {{On Coreference Resolution Performance Metrics}}, author = {Luo, Xiaoqiang}, booktitle = {Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing}, year = {2005}, url = {https://aclanthology.org/H05-1004/} } ``` - [← previous Deterministic Coreference Resolution Based on Entity-Centric, Precision-Ranked Rules](https://md.mova.elamur.ai/free/en/docs/papers/lee-2013-deterministic-coreference-resolution-based-entity/) - [next → Evaluating large language models for the tasks of PoS tagging within the Universal Dependency framework](https://md.mova.elamur.ai/free/en/docs/papers/machado-2024-evaluating-large-language-models-tasks/) - [↑ back to papers](https://md.mova.elamur.ai/free/en/docs/papers/) ## References 13 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the Linguistic Coreference Workshop at The First International Conference on Language Resources and Evaluation (LREC’98), pages 563–566. [find](https://scholar.google.com/scholar?q=Amit+Bagga+and+Breck+Baldwin%2E+1998%2E+Algorithms+for+scoring+coreference+chains%2E+In+Proceedings+of+the+Linguistic+Coreference+Workshop+at+The+First+International+Conference+on+Language+Resources+and+Eva) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bagga-1998-algorithms-scoring-coreference-chains/) 2. Egon Balas, Donald Miller, Joseph Pekny, and Paolo Toth. 1991. A parallel shortest augmenting path algorithm for the assignment problem. Journal of the ACM (JACM), 38(4). [find](https://scholar.google.com/scholar?q=Egon+Balas%2C+Donald+Miller%2C+Joseph+Pekny%2C+and+Paolo+Toth%2E+1991%2E+A+parallel+shortest+augmenting+path+algorithm+for+the+assignment+problem%2E+Journal+of+the+ACM+%28JACM%29%2C+38%284%29%2E) 3. Francois Bourgeois and Jean-Claude Lassalle. 1971. An extension of the munkres algorithm for the assignment problem to rectangular matrices. Communications of the ACM, 14(12). [find](https://scholar.google.com/scholar?q=Francois+Bourgeois+and+Jean%2DClaude+Lassalle%2E+1971%2E+An+extension+of+the+munkres+algorithm+for+the+assignment+problem+to+rectangular+matrices%2E+Communications+of+the+ACM%2C+14%2812%29%2E) 4. R. Fletcher. 1987. Practical Methods of Optimization. John Wiley and Sons. [find](https://scholar.google.com/scholar?q=R%2E+Fletcher%2E+1987%2E+Practical+Methods+of+Optimization%2E+John+Wiley+and+Sons%2E) 5. Anshul Gupta and Lexing Ying. 1999. Algorithms for finding maximum matchings in bipartite graphs. Technical Report RC 21576 (97320), IBM T.J. Watson Research Center, October. [find](https://scholar.google.com/scholar?q=Anshul+Gupta+and+Lexing+Ying%2E+1999%2E+Algorithms+for+finding+maximum+matchings+in+bipartite+graphs%2E+Technical+Report+RC+21576+%2897320%29%2C+IBM+T%2EJ%2E+Watson+Research+Center%2C+October%2E) 6. H.W. Kuhn. 1955. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(83). [find](https://scholar.google.com/scholar?q=H%2EW%2E+Kuhn%2E+1955%2E+The+hungarian+method+for+the+assignment+problem%2E+Naval+Research+Logistics+Quarterly%2C+2%2883%29%2E) 7. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Xiaoqiang+Luo%2C+Abe+Ittycheriah%2C+Hongyan+Jing%2C+Nanda+Kambhatla%2C+and+Salim+Roukos%2E+2004%2E+A+mention%2Dsynchronous+coreference+resolution+algorithm+based+on+the+bell+tree%2E+In+Proc%2E+of+ACL%2E) 8. MUC-6. 1995. Proceedings of the Sixth Message Understanding Conference(MUC-6), San Francisco, CA. Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=MUC%2D6%2E+1995%2E+Proceedings+of+the+Sixth+Message+Understanding+Conference%28MUC%2D6%29%2C+San+Francisco%2C+CA%2E+Morgan+Kaufmann%2E) 9. MUC-7. 1998. Proceedings of the Seventh Message Understanding Conference(MUC-7). [find](https://scholar.google.com/scholar?q=MUC%2D7%2E+1998%2E+Proceedings+of+the+Seventh+Message+Understanding+Conference%28MUC%2D7%29%2E) 10. J. Munkres. 1957. Algorithms for the assignment and transportation problems. Journal of SIAM, 5:32–38. [find](https://scholar.google.com/scholar?q=J%2E+Munkres%2E+1957%2E+Algorithms+for+the+assignment+and+transportation+problems%2E+Journal+of+SIAM%2C+5%3A32%E2%80%9338%2E) 11. NIST. 2003a. The ACE evaluation plan. www.nist.gov/speech/tests/ace/index.htm. [find](https://scholar.google.com/scholar?q=NIST%2E+2003a%2E+The+ACE+evaluation+plan%2E+www%2Enist%2Egov%2Fspeech%2Ftests%2Face%2Findex%2Ehtm%2E) 12. NIST. 2003b. Proceedings of ACE’03 workshop. Booklet, Alexandria, VA, September. [find](https://scholar.google.com/scholar?q=NIST%2E+2003b%2E+Proceedings+of+ACE%E2%80%9903+workshop%2E+Booklet%2C+Alexandria%2C+VA%2C+September%2E) 13. M. Vilain, J. Burger, J. Aberdeen, D. Connolly, , and L. Hirschman. 1995. A model-theoretic coreference scoring scheme. In In Proc. of MUC6, pages 45–52. [find](https://scholar.google.com/scholar?q=M%2E+Vilain%2C+J%2E+Burger%2C+J%2E+Aberdeen%2C+D%2E+Connolly%2C+%2C+and+L%2E+Hirschman%2E+1995%2E+A+model%2Dtheoretic+coreference+scoring+scheme%2E+In+In+Proc%2E+of+MUC6%2C+pages+45%E2%80%9352%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/vilain-1995-model-theoretic-coreference-scoring-scheme/)