[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/lee-2011-stanford-s-multi-pass-sieve/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Stanford's Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task

# Stanford's Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task

**Authors:** [Heeyoung Lee](https://md.mova.elamur.ai/free/en/docs/papers/authors/heeyoung-lee/), [Yves Peirsman](https://md.mova.elamur.ai/free/en/docs/papers/authors/yves-peirsman/), [Angel Chang](https://md.mova.elamur.ai/free/en/docs/papers/authors/angel-chang/), [Nathanael Chambers](https://md.mova.elamur.ai/free/en/docs/papers/authors/nathanael-chambers/), [Mihai Surdeanu](https://md.mova.elamur.ai/free/en/docs/papers/authors/mihai-surdeanu/), [Dan Jurafsky](https://md.mova.elamur.ai/free/en/docs/papers/authors/dan-jurafsky/) · **Year:** 2011 · **Venue:** Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task **Link:** [aclanthology.org/W11-1902](https://aclanthology.org/W11-1902/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, before 2016) — summary only

[**150** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/lee-2011-stanford-s-multi-pass-sieve/) of the 140673 papers whose bibliographies we read[Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task](https://aclanthology.org/volumes/W11-19/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/W11-1902/)
- [PDF](https://aclanthology.org/W11-1902.pdf)

**Authors (6)**

1. Heeyoung Lee
2. Yves Peirsman
3. Angel Chang
4. Nathanael Chambers
5. Mihai Surdeanu
6. Dan Jurafsky

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

The system is a sequence of deterministic sieves, ordered from highest to lowest precision. Each sieve links clusters, not single mentions, and the clusters share attributes such as gender, number and animacy. The authors add a high-recall mention-detection sieve in front. Its exclusion rules remove pleonastic "it", numeric entities, partitives, nationality adjectives and a stop list. Post-processing then drops singletons and appositive mentions. Two new sieves are added: a discourse sieve that matches speakers to pronouns (about two points of gain), and semantic sieves based on WordNet, Wikipedia and Freebase, which do not help with predicted mentions. Mention recall is 92.8% with gold annotation and 87.9% with predicted annotation. On the test set the system scores 57.8 in the closed track and 58.3 in the open track, both the best in the task. With gold mentions it scores 60.7 and 61.4. Gold linguistic annotation adds only about 2 points, while gold mentions add about 15.

## Ideas

- I1. Make the mention stage a recall-first sieve with recorded losses: list each exclusion rule and the gold recall it costs on dev. Only rules that remove more spurious than true mentions should stay.
- I2. Log, for every link, the number of the sieve that made it. Every decision then says why, and the error rate can be counted per sieve on dev. This is the transparency that Mova asks for.
- I3. Report gold-mention and predicted-mention scores side by side. The gap is the budget that mention detection can spend, which tells the team where effort pays.
- I4. Give each new sieve a precision check on held-out data before it enters the order. The paper's semantic sieves hurt predicted mentions, which is the case such a check would catch.

## For Mova

The code is `coref/src/sieve.rs` (ordered sieves and shared cluster attributes), `coref/src/mention.rs` (mention detection, using the parse from `en/src/parse.rs`) and `coref/src/eval.rs`. The chapter is [35. Reference and coreference](https://md.mova.elamur.ai/free/en/docs/languages/en/35-reference-and-coreference/). The measure is the average of MUC, B-cubed and CEAF-e on dev, with gold and predicted mentions. Run it twice and check that the results are identical. Negative control: a reversed sieve order must score lower.

Topics: [Coreference resolution](https://md.mova.elamur.ai/free/en/docs/papers/topics/coreference-resolution/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{lee2011stanford,
  title = {{Stanford's Multi-Pass Sieve Coreference Resolution System at the CoNLL-2011 Shared Task}},
  author = {Lee, Heeyoung and Peirsman, Yves and Chang, Angel and Chambers, Nathanael and Surdeanu, Mihai and Jurafsky, Dan},
  booktitle = {Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task},
  year = {2011},
  url = {https://aclanthology.org/W11-1902/}
}
```

- [← previous Scalar Implicatures: Pragmatic Inferences or Grammar?](https://md.mova.elamur.ai/free/en/docs/papers/lee-2008-scalar-implicatures-pragmatic-inferences-or/)
- [next → End-to-end Neural Coreference Resolution](https://md.mova.elamur.ai/free/en/docs/papers/lee-2017-end-end-neural-coreference-resolution/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. B. Baldwin. 1997. CogNIAC: high precision coreference with limited knowledge and linguistic resources. In Proceedings of a Workshop on Operational Factors in Practical, Robust Anaphora Resolution for Unrestricted Texts. [find](https://scholar.google.com/scholar?q=B%2E+Baldwin%2E+1997%2E+CogNIAC%3A+high+precision+coreference+with+limited+knowledge+and+linguistic+resources%2E+In+Proceedings+of+a+Workshop+on+Operational+Factors+in+Practical%2C+Robust+Anaphora+Resolution+for+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/baldwin-1997-cogniac-high-precision-coreference-limited/)
2. E. Bengston & D. Roth. 2008. Understanding the value of features for coreference resolution. In EMNLP. [find](https://scholar.google.com/scholar?q=E%2E+Bengston+%26+D%2E+Roth%2E+2008%2E+Understanding+the+value+of+features+for+coreference+resolution%2E+In+EMNLP%2E)
3. Jari Björne, Juho Heimonen, Filip Ginter, Antti Airola, Tapio Pahikkala, and Tapio Salakoski. 2009. Extracting Complex Biological Events with Rich Graph-Based Feature Sets. Proceedings of the Workshop on BioNLP: Shared Task. [find](https://scholar.google.com/scholar?q=Jari+Bjo%CC%88rne%2C+Juho+Heimonen%2C+Filip+Ginter%2C+Antti+Airola%2C+Tapio+Pahikkala%2C+and+Tapio+Salakoski%2E+2009%2E+Extracting+Complex+Biological+Events+with+Rich+Graph%2DBased+Feature+Sets%2E+Proceedings+of+the+Worksho)
4. H. Daumé III and D. Marcu. 2005. A large-scale exploration of effective global features for a joint entity detection and tracking model. In EMNLP-HLT. [find](https://scholar.google.com/scholar?q=H%2E+Daume%CC%81+III+and+D%2E+Marcu%2E+2005%2E+A+large%2Dscale+exploration+of+effective+global+features+for+a+joint+entity+detection+and+tracking+model%2E+In+EMNLP%2DHLT%2E)
5. B. A. Fox 1993. Discourse structure and anaphora: written and conversational English. Cambridge University Press. [find](https://scholar.google.com/scholar?q=B%2E+A%2E+Fox+1993%2E+Discourse+structure+and+anaphora%3A+written+and+conversational+English%2E+Cambridge+University+Press%2E)
6. A. Haghighi and D. Klein. 2010. Coreference resolution in a modular, entity-centered model. In Proc. of HLT-NAACL. [find](https://scholar.google.com/scholar?q=A%2E+Haghighi+and+D%2E+Klein%2E+2010%2E+Coreference+resolution+in+a+modular%2C+entity%2Dcentered+model%2E+In+Proc%2E+of+HLT%2DNAACL%2E)
7. E. Hovy, M. Marcus, M. Palmer, L. Ramshaw, and R. Weischedel 2006. OntoNotes: The 90% Solution. In HLT/NAACL. [find](https://scholar.google.com/scholar?q=E%2E+Hovy%2C+M%2E+Marcus%2C+M%2E+Palmer%2C+L%2E+Ramshaw%2C+and+R%2E+Weischedel+2006%2E+OntoNotes%3A+The+90%25+Solution%2E+In+HLT%2FNAACL%2E)
8. Z. Huang, G. Zeng, W. Xu, and A. Celikyilmaz 2009. Accurate semantic class classifier for coreference resolution. In EMNLP. [find](https://scholar.google.com/scholar?q=Z%2E+Huang%2C+G%2E+Zeng%2C+W%2E+Xu%2C+and+A%2E+Celikyilmaz+2009%2E+Accurate+semantic+class+classifier+for+coreference+resolution%2E+In+EMNLP%2E)
9. J.R. Hobbs. 1977. Resolving pronoun references. Lingua. [find](https://scholar.google.com/scholar?q=J%2ER%2E+Hobbs%2E+1977%2E+Resolving+pronoun+references%2E+Lingua%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/hobbs-1978-resolving-pronoun-references/)
10. H. Ji and D. Lin. 2009. Gender and animacy knowledge discovery from web-scale n-grams for unsupervised person mention detection. In PACLIC. [find](https://scholar.google.com/scholar?q=H%2E+Ji+and+D%2E+Lin%2E+2009%2E+Gender+and+animacy+knowledge+discovery+from+web%2Dscale+n%2Dgrams+for+unsupervised+person+mention+detection%2E+In+PACLIC%2E)
11. Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Jun’ichi Tsujii. 2009. Overview of the BioNLP’09 Shared Task on Event Extraction. Proceedings of the NAACL-HLT 2009 Workshop on Natural Language Processing in Biomedicine (BioNLP’09). [find](https://scholar.google.com/scholar?q=Jin%2DDong+Kim%2C+Tomoko+Ohta%2C+Sampo+Pyysalo%2C+Yoshinobu+Kano%2C+and+Jun%E2%80%99ichi+Tsujii%2E+2009%2E+Overview+of+the+BioNLP%E2%80%9909+Shared+Task+on+Event+Extraction%2E+Proceedings+of+the+NAACL%2DHLT+2009+Workshop+on+Natural+La)
12. V. Ng 2007. Semantic Class Induction and Coreference Resolution. In ACL. [find](https://scholar.google.com/scholar?q=V%2E+Ng+2007%2E+Semantic+Class+Induction+and+Coreference+Resolution%2E+In+ACL%2E)
13. V. Ng and C. Cardie. 2002. Improving Machine Learning Approaches to Coreference Resolution. in ACL 2002 [find](https://scholar.google.com/scholar?q=V%2E+Ng+and+C%2E+Cardie%2E+2002%2E+Improving+Machine+Learning+Approaches+to+Coreference+Resolution%2E+in+ACL+2002)
14. S. Ponzetto and M. Strube. 2006. Exploiting semantic role labeling, Wordnet and Wikipedia for coreference resolution. Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=S%2E+Ponzetto+and+M%2E+Strube%2E+2006%2E+Exploiting+semantic+role+labeling%2C+Wordnet+and+Wikipedia+for+coreference+resolution%2E+Proceedings+of+NAACL%2E)
15. Sameer Pradhan, Lance Ramshaw, Ralph Weischedel, Jessica MacBride, and Linnea Micciulla. 2007. Unrestricted Coreference: Indentifying Entities and Events in OntoNotes. In Proceedings of the IEEE International Conference on Semantic Computing (ICSC). [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Lance+Ramshaw%2C+Ralph+Weischedel%2C+Jessica+MacBride%2C+and+Linnea+Micciulla%2E+2007%2E+Unrestricted+Coreference%3A+Indentifying+Entities+and+Events+in+OntoNotes%2E+In+Proceedings+of+the+IEEE+Inter)
16. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 Shared Task: Modeling Unrestricted Coreference in OntoNotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning (CoNLL 2011). [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Lance+Ramshaw%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Ralph+Weischedel%2C+and+Nianwen+Xue%2E+2011%2E+CoNLL%2D2011+Shared+Task%3A+Modeling+Unrestricted+Coreference+in+OntoNotes%2E+In+Proceedings+of+the+Fi) [our card](https://md.mova.elamur.ai/free/en/docs/papers/pradhan-2011-conll-2011-shared-task-modeling/)
17. K. Raghunathan, H. Lee, S. Rangarajan, N. Chambers, M. Surdeanu, D. Jurafsky, and C. Manning 2010. A Multi-Pass Sieve for Coreference Resolution. In EMNLP. [find](https://scholar.google.com/scholar?q=K%2E+Raghunathan%2C+H%2E+Lee%2C+S%2E+Rangarajan%2C+N%2E+Chambers%2C+M%2E+Surdeanu%2C+D%2E+Jurafsky%2C+and+C%2E+Manning+2010%2E+A+Multi%2DPass+Sieve+for+Coreference+Resolution%2E+In+EMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/raghunathan-2010-multi-pass-sieve-coreference-resolution/)
18. J. Tetreault and J. Allen. 2003. An Empirical Evaluation of Pronoun Resolution and Clausal Structure. In Proceedings of the 2003 International Symposium on Reference Resolution. [find](https://scholar.google.com/scholar?q=J%2E+Tetreault+and+J%2E+Allen%2E+2003%2E+An+Empirical+Evaluation+of+Pronoun+Resolution+and+Clausal+Structure%2E+In+Proceedings+of+the+2003+International+Symposium+on+Reference+Resolution%2E)
19. J. Tetreault and J. Allen. 2004. Dialogue Structure and Pronoun Resolution. In DAARC. [find](https://scholar.google.com/scholar?q=J%2E+Tetreault+and+J%2E+Allen%2E+2004%2E+Dialogue+Structure+and+Pronoun+Resolution%2E+In+DAARC%2E)
20. X. Yang and J. Su. 2007. Coreference Resolution Using Semantic Relatedness Information from Automatically Discovered Patterns. In ACL. [find](https://scholar.google.com/scholar?q=X%2E+Yang+and+J%2E+Su%2E+2007%2E+Coreference+Resolution+Using+Semantic+Relatedness+Information+from+Automatically+Discovered+Patterns%2E+In+ACL%2E)