[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/lingbulk-2010-rosenthal-semi-automated-annotation-prepositional-phrase/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Towards Semi-Automated Annotation for Prepositional Phrase Attachment

# Towards Semi-Automated Annotation for Prepositional Phrase Attachment

**Authors:** Sara Rosenthal, William J. Lipovsky, Kathleen McKeown, Kapil Thadani, [Jacob Andreas](https://md.mova.elamur.ai/free/en/docs/papers/authors/jacob-andreas/) · **Year:** 2010 · **Venue:** Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10) **Link:** [aclanthology.org/L10-1572](https://aclanthology.org/L10-1572/) · **Licence:** copyrighted (ELRA, all rights reserved; listed as [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) by the ACL Anthology) — summary only

[**4** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/lingbulk-2010-rosenthal-semi-automated-annotation-prepositional-phrase/) of the 140673 papers whose bibliographies we read[Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)](https://aclanthology.org/volumes/L10-1/)[licence](https://creativecommons.org/licenses/by-nc/4.0/)

- [Publisher's page](https://aclanthology.org/L10-1572/)
- [PDF](https://aclanthology.org/L10-1572.pdf)

**Authors (5)**

1. Sara Rosenthal
2. William Lipovsky
3. Kathleen McKeown
4. Kapil Thadani
5. Jacob Andreas

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

The paper asks whether untrained crowd workers can label PP attachment well enough to make training data. The system turns Penn Treebank parses into multiple-choice questions: the PP and its alternative noun or verb heads. The study used 1000 questions from Wall Street Journal sentences, each answered by three Amazon Mechanical Turk workers, at a cost of $135 and 49 seconds per task. Workers picked the right attachment in 86.7% of the 3000 responses. The majority vote was right for 90.4% of questions, and 92.2% after the authors removed 18 of 96 wrong-majority cases caused by their own question generator. A unanimous answer was correct in 97% of cases, a bare majority in 82%.

## Ideas

- I1. `annot`: keep a silver PP head only when three independent sources agree (a large model, a `tools/treeagree` rule, and the `en` parse), and drop the rest. Measure the precision of the kept heads on a 300-item gold sample from EWT, and the share kept. Use EWT only, since it is CC BY-SA.
- I2. Error map: the workers failed mostly when the PP attached to a verb or an adjective, and rarely to a noun. Report PP head accuracy in `en parse-eval` split by the gold head's part of speech, on EWT test.
- I3. Two-head check: the paper found items with two correct attachments, which the answer key treated as one. Mark PPs with more than one defensible head in the EWT gold set and score any of them as correct. Measure the ceiling before tuning any model.
- I4. Only the agreement rule is taken from this work. Mova does not use a crowd, and the Penn Treebank data is test only.

## For Mova

The code is `en/src/parse.rs` for the PP scores and `annot` for the silver rule. The chapter is [22. Phrases: noun, adjective, adverb and prepositional phrases](https://md.mova.elamur.ai/free/en/docs/languages/en/22-phrases/). The paper's questions come from WSJ text (LDC-licensed), so any measurement on that text is test only. Its 92.2% is a majority of three people on 1000 questions, so it is not a parser score and must not be read as one.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{rosenthal2010semi,
  title = {{Towards Semi-Automated Annotation for Prepositional Phrase Attachment}},
  author = {Rosenthal, Sara and Lipovsky, William and McKeown, Kathleen and Thadani, Kapil and Andreas, Jacob},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  url = {https://aclanthology.org/L10-1572/}
}
```

- [← previous Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2008-foster-parser-evaluation-bnc-evaluating-4/)
- [next → A Non-Monotonic Arc-Eager Transition System for Dependency Parsing](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2013-honnibal-non-monotonic-arc-eager-transition/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Timothy Baldwin, and David Martinez. 2008. Improving parsing and PP attachment performance with sense information. In Proceedings of ACL, pages 317–325, June. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Timothy+Baldwin%2C+and+David+Martinez%2E+2008%2E+Improving+parsing+and+PP+attachment+performance+with+sense+information%2E+In+Proceedings+of+ACL%2C+pages+317%E2%80%93325%2C+June%2E)
2. Michaela Atterer and Hinrich Schütze. 2007. Prepositional phrase attachment without oracles. Computational Linguistics, 33(4):469–476. [find](https://scholar.google.com/scholar?q=Michaela+Atterer+and+Hinrich+Schu%CC%88tze%2E+2007%2E+Prepositional+phrase+attachment+without+oracles%2E+Computational+Linguistics%2C+33%284%29%3A469%E2%80%93476%2E)
3. Eric Brill. 1995. Transformation-based error-driven learning and natural language processing: a case study in part-of-speech tagging. Computational Linguistics, 21(4):543–565. [find](https://scholar.google.com/scholar?q=Eric+Brill%2E+1995%2E+Transformation%2Dbased+error%2Ddriven+learning+and+natural+language+processing%3A+a+case+study+in+part%2Dof%2Dspeech+tagging%2E+Computational+Linguistics%2C+21%284%29%3A543%E2%80%93565%2E)
4. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of ACL, pages 132–139. [find](https://scholar.google.com/scholar?q=Eugene+Charniak%2E+2000%2E+A+maximum%2Dentropy%2Dinspired+parser%2E+In+Proceedings+of+ACL%2C+pages+132%E2%80%93139%2E)
5. Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational Linguistics, 29(4):589–637. [find](https://scholar.google.com/scholar?q=Michael+Collins%2E+2003%2E+Head%2Ddriven+statistical+models+for+natural+language+parsing%2E+Computational+Linguistics%2C+29%284%29%3A589%E2%80%93637%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/cl-2003-collins-head-driven-statistical-models-natural/)
6. Daniel Gruhl, R. Guha, David Liben-Nowell, and Andrew Tomkins. 2004. Information diffusion through blogspace. In WWW ’04: Proceedings of the 13th international conference on World Wide Web, pages 491–501. [find](https://scholar.google.com/scholar?q=Daniel+Gruhl%2C+R%2E+Guha%2C+David+Liben%2DNowell%2C+and+Andrew+Tomkins%2E+2004%2E+Information+diffusion+through+blogspace%2E+In+WWW+%E2%80%9904%3A+Proceedings+of+the+13th+international+conference+on+World+Wide+Web%2C+pages+491%E2%80%93)
7. Liangjie Hong and Brian D. Davison. 2009. A classification-based approach to question answering in discussion boards. In SIGIR ’09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 171– 178. [find](https://scholar.google.com/scholar?q=Liangjie+Hong+and+Brian+D%2E+Davison%2E+2009%2E+A+classification%2Dbased+approach+to+question+answering+in+discussion+boards%2E+In+SIGIR+%E2%80%9909%3A+Proceedings+of+the+32nd+international+ACM+SIGIR+conference+on+Resear)
8. Gilly Leshed and Joseph ’Jofish’ Kaye. 2006. Understanding how bloggers feel: recognizing affect in blog posts. In CHI ’06 extended abstracts on Human factors in computing systems, pages 1019–1024. [find](https://scholar.google.com/scholar?q=Gilly+Leshed+and+Joseph+%E2%80%99Jofish%E2%80%99+Kaye%2E+2006%2E+Understanding+how+bloggers+feel%3A+recognizing+affect+in+blog+posts%2E+In+CHI+%E2%80%9906+extended+abstracts+on+Human+factors+in+computing+systems%2C+pages+1019%E2%80%931024%2E)
9. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: the Penn Treebank. Computational Linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=Mitchell+P%2E+Marcus%2C+Mary+Ann+Marcinkiewicz%2C+and+Beatrice+Santorini%2E+1993%2E+Building+a+large+annotated+corpus+of+english%3A+the+Penn+Treebank%2E+Computational+Linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
10. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The nombank project: An interim report. In Adam Meyers, editor, Proceedings of the HLT-NAACL Workshop on Frontiers in Corpus Annotation, pages 24– 31, Boston, Massachusetts, USA, May. [find](https://scholar.google.com/scholar?q=Adam+Meyers%2C+Ruth+Reeves%2C+Catherine+Macleod%2C+Rachel+Szekely%2C+Veronika+Zielinska%2C+Brian+Young%2C+and+Ralph+Grishman%2E+2004%2E+The+nombank+project%3A+An+interim+report%2E+In+Adam+Meyers%2C+editor%2C+Proceedings+of+t)
11. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. [↗](https://doi.org/10.1162/0891201053630264) [our card](https://md.mova.elamur.ai/free/en/docs/papers/palmer-2005-proposition-bank-annotated-corpus-semantic/)
12. Sameer S. Pradhan and Nianwen Xue. 2009. Ontonotes: the 90% solution. In Proceedings of NAACL, pages 11– 12. [find](https://scholar.google.com/scholar?q=Sameer+S%2E+Pradhan+and+Nianwen+Xue%2E+2009%2E+Ontonotes%3A+the+90%25+solution%2E+In+Proceedings+of+NAACL%2C+pages+11%E2%80%93+12%2E)
13. Adwait Ratnaparkhi, Jeff Reynar, and Salim Roukos. 1994. A maximum entropy model for prepositional phrase attachment. In Proceedings of HLT, pages 250–255. [find](https://scholar.google.com/scholar?q=Adwait+Ratnaparkhi%2C+Jeff+Reynar%2C+and+Salim+Roukos%2E+1994%2E+A+maximum+entropy+model+for+prepositional+phrase+attachment%2E+In+Proceedings+of+HLT%2C+pages+250%E2%80%93255%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-1994-ratnaparkhi-maximum-entropy-model-prepositional-phrase/)
14. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP, pages 254–263. [find](https://scholar.google.com/scholar?q=Rion+Snow%2C+Brendan+O%E2%80%99Connor%2C+Daniel+Jurafsky%2C+and+Andrew+Y%2E+Ng%2E+2008%2E+Cheap+and+fast%E2%80%94but+is+it+good%3F%3A+evaluating+non%2Dexpert+annotations+for+natural+language+tasks%2E+In+Proceedings+of+EMNLP%2C+pages+254%E2%80%932)
15. Jiri Stetina and Makoto Nagao. 1997. Corpus based pp attachment ambiguity resolution with a semantic dictionary. In Proceedings of the Workshop on Very Large Corpora, pages 66–80. [find](https://scholar.google.com/scholar?q=Jiri+Stetina+and+Makoto+Nagao%2E+1997%2E+Corpus+based+pp+attachment+ambiguity+resolution+with+a+semantic+dictionary%2E+In+Proceedings+of+the+Workshop+on+Very+Large+Corpora%2C+pages+66%E2%80%9380%2E)
16. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: chunking. In Proceedings of the CoNLL workshop on Learning Language in Logic, pages 127–132. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang+and+Sabine+Buchholz%2E+2000%2E+Introduction+to+the+CoNLL%2D2000+shared+task%3A+chunking%2E+In+Proceedings+of+the+CoNLL+workshop+on+Learning+Language+in+Logic%2C+pages+127%E2%80%93132%2E)
17. Alexander S. Yeh and Marc B. Vilain. 1998. Some properties of preposition and subordinate conjunction attachments. In Proceedings of COLING, pages 1436–1442. [find](https://scholar.google.com/scholar?q=Alexander+S%2E+Yeh+and+Marc+B%2E+Vilain%2E+1998%2E+Some+properties+of+preposition+and+subordinate+conjunction+attachments%2E+In+Proceedings+of+COLING%2C+pages+1436%E2%80%931442%2E)
18. Jakub Zavrel, Walter Daelemans, and Jorn Veenstra. 1997. Resolving PP attachment ambiguities with memory-based learning. In Proc. of the Workshop on Computational Language Learning (CoNLL’97), ACL, pages 136–144. [find](https://scholar.google.com/scholar?q=Jakub+Zavrel%2C+Walter+Daelemans%2C+and+Jorn+Veenstra%2E+1997%2E+Resolving+PP+attachment+ambiguities+with+memory%2Dbased+learning%2E+In+Proc%2E+of+the+Workshop+on+Computational+Language+Learning+%28CoNLL%E2%80%9997%29%2C+ACL%2C+pa)