[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/marneffe-2010-was-it-good-it-was/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Was It Good? It Was Provocative.'' Learning the Meaning of Scalar Adjectives

# ``Was It Good? It Was Provocative.'' Learning the Meaning of Scalar Adjectives

**Authors:** [Marie-Catherine de Marneffe](https://md.mova.elamur.ai/free/en/docs/papers/authors/marie-catherine-de-marneffe/), [Christopher D. Manning](https://md.mova.elamur.ai/free/en/docs/papers/authors/christopher-d-manning/) and [Christopher Potts](https://md.mova.elamur.ai/free/en/docs/papers/authors/christopher-potts/) · **Year:** 2010 · **Venue:** Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics **Link:** [aclanthology.org/P10-1018](https://aclanthology.org/P10-1018/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) (ACL Anthology, before 2016) — summary only; nothing copied into a release

[**25** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/marneffe-2010-was-it-good-it-was/) of the 140673 papers whose bibliographies we read[Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics](https://aclanthology.org/volumes/P10-1/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/P10-1018/)
- [PDF](https://aclanthology.org/P10-1018.pdf)

**Authors (3)**

1. Marie-Catherine de Marneffe
2. Christopher D. Manning
3. Christopher Potts

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

Indirect answers to yes/no questions often do not say "yes" or "no" outright. "Was it good? It was provocative" leaves the answer open, while "It was great" does not. The authors mine two kinds of evidence. First, scale order between gradable adjectives comes from online IMDB reviews: a word's expected star rating (from the rating distribution of its reviews) puts "good" below "excellent" on a common scale. A decision rule then maps the question and answer predications to yes, no or uncertain, including negation. Second, for numerical answers ("about 350 people", "seven years old") the authors fit a logistic model of the probability that a number counts as the adjective, using hit counts from Web queries with and without negation. The test set is 224 question-answer pairs from CNN interview transcripts and the Switchboard dialogue corpus, each labelled by 30 Mechanical Turk workers. The system matches the dominant worker answer in 71% of pairs (159 of 224). On the 66 pairs where workers fully agreed the rate is 89%. For "other adjective" answers the method reaches 60%, against 56% for a WordNet-based sentiment method. It correctly marks 18 of 31 oppositional pairs, against 5 for a hand-built antonym list.

## Ideas

- I1. `prag/src/infer.rs`: a yes/no decision table over scale order, as in the paper's rule set (different signs mean "no", a larger magnitude means "yes", negation flips yes and no). Write it as a short rule list with a BTreeMap of scale positions. Measure against the paper's 224 pairs when they can be obtained under a licence that allows it, or against hand-made pairs.
- I2. Agreement as a confidence signal: the paper's logistic fit shows accuracy falls as worker entropy rises. Mova can output "uncertain" when its own rule has no scale position for either predication, and check the rate against a held-out set.
- I3. For number-plus-adjective answers, a rule-based version: a table of typical units and thresholds per adjective (age for "little kids", degrees for "warm weather"), not a Web-count model. The paper's eight well-fitting items are a small test.
- I4. The oppositional-term test (31 items, 18 correct) is a useful negative control: any new antonym table must beat a plain antonym list on these items.

## For Mova

The chapters are [15. Adjectives and adverbs](https://md.mova.elamur.ai/free/en/docs/languages/en/15-adjectives-adverbs/) (scales and antonyms) and [36. Discourse relations and coherence](https://md.mova.elamur.ai/free/en/docs/languages/en/36-discourse-relations/) (indirect answers). The code is `prag/src/infer.rs` for the decision rule and `en/src/dict.rs` for the scale table. Measure: accuracy against the dominant answer on a held-out set of Mova's own dialogues, with a shuffled-scale control that must score lower. Note: the paper's IMDB and CNN text is not covered by this card; its licence was not checked.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{marneffe2010was,
  title = {{Was It Good? It Was Provocative.'' Learning the Meaning of Scalar Adjectives}},
  author = {Marneffe, Marie-Catherine de and Manning, Christopher D. and Potts, Christopher},
  booktitle = {Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/P10-1018/}
}
```

- [← previous A Simple and Accurate Syntax-Agnostic Neural Model for Dependency-based Semantic Role Labeling](https://md.mova.elamur.ai/free/en/docs/papers/marcheggiani-2017-simple-accurate-syntax-agnostic-neural/)
- [next → Dependency-based paraphrasing for recognizing textual entailment](https://md.mova.elamur.ai/free/en/docs/papers/marsi-2007-dependency-based-paraphrasing-recognizing-textual/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. James F. Allen and C. Raymond Perrault. 1980. Analyzing intention in utterances. Artificial Intelligence, 15:143–178. [find](https://scholar.google.com/scholar?q=James+F%2E+Allen+and+C%2E+Raymond+Perrault%2E+1980%2E+Analyzing+intention+in+utterances%2E+Artificial+Intelligence%2C+15%3A143%E2%80%93178%2E)
2. Nicholas Asher and Alex Lascarides. 2003. Logics of Conversation. Cambridge University Press, Cambridge. [find](https://scholar.google.com/scholar?q=Nicholas+Asher+and+Alex+Lascarides%2E+2003%2E+Logics+of+Conversation%2E+Cambridge+University+Press%2C+Cambridge%2E)
3. Sasha Blair-Goldensohn, Kerry Hannan, Ryan McDonald, Tyler Neylon, George A. Reis, and Jeff Reynar. 2008. Building a sentiment summarizer for local service reviews. In WWW Workshop on NLP in the Information Explosion Era (NLPIX). [find](https://scholar.google.com/scholar?q=Sasha+Blair%2DGoldensohn%2C+Kerry+Hannan%2C+Ryan+McDonald%2C+Tyler+Neylon%2C+George+A%2E+Reis%2C+and+Jeff+Reynar%2E+2008%2E+Building+a+sentiment+summarizer+for+local+service+reviews%2E+In+WWW+Workshop+on+NLP+in+the+Infor)
4. Judith A. Chevalier and Dina Mayzlin. 2006. The effect of word of mouth on sales: Online book reviews. Journal of Marketing Research, 43(3):345– 354. [find](https://scholar.google.com/scholar?q=Judith+A%2E+Chevalier+and+Dina+Mayzlin%2E+2006%2E+The+effect+of+word+of+mouth+on+sales%3A+Online+book+reviews%2E+Journal+of+Marketing+Research%2C+43%283%29%3A345%E2%80%93+354%2E)
5. Herbert H. Clark. 1979. Responding to indirect speech acts. Cognitive Psychology, 11:430–477. [find](https://scholar.google.com/scholar?q=Herbert+H%2E+Clark%2E+1979%2E+Responding+to+indirect+speech+acts%2E+Cognitive+Psychology%2C+11%3A430%E2%80%93477%2E)
6. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC-2006). [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+de+Marneffe%2C+Bill+MacCartney%2C+and+Christopher+D%2E+Manning%2E+2006%2E+Generating+typed+dependency+parses+from+phrase+structure+parses%2E+In+Proceedings+of+the+5th+International+Conference+on+L) [our card](https://md.mova.elamur.ai/free/en/docs/papers/map-2006-marneffe-generating-typed-dependency-parses-phrase/)
7. Marie-Catherine de Marneffe, Scott Grimm, and Christopher Potts. 2009. Not a simple ‘yes’ or ‘no’: Uncertainty in indirect answers. In Proceedings of the 10th Annual SIGDIAL Meeting on Discourse and Dialogue. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+de+Marneffe%2C+Scott+Grimm%2C+and+Christopher+Potts%2E+2009%2E+Not+a+simple+%E2%80%98yes%E2%80%99+or+%E2%80%98no%E2%80%99%3A+Uncertainty+in+indirect+answers%2E+In+Proceedings+of+the+10th+Annual+SIGDIAL+Meeting+on+Discourse+and+D)
8. Gilles Fauconnier. 1975. Pragmatic scales and logical structure. Linguistic Inquiry, 6(3):353–375. [find](https://scholar.google.com/scholar?q=Gilles+Fauconnier%2E+1975%2E+Pragmatic+scales+and+logical+structure%2E+Linguistic+Inquiry%2C+6%283%29%3A353%E2%80%93375%2E)
9. Nancy Green and Sandra Carberry. 1994. A hybrid reasoning model for indirect answers. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics, pages 58–65. [find](https://scholar.google.com/scholar?q=Nancy+Green+and+Sandra+Carberry%2E+1994%2E+A+hybrid+reasoning+model+for+indirect+answers%2E+In+Proceedings+of+the+32nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+58%E2%80%9365%2E)
10. Nancy Green and Sandra Carberry. 1999. Interpreting and generating indirect answers. Computational Linguistics, 25(3):389–435. [find](https://scholar.google.com/scholar?q=Nancy+Green+and+Sandra+Carberry%2E+1999%2E+Interpreting+and+generating+indirect+answers%2E+Computational+Linguistics%2C+25%283%29%3A389%E2%80%93435%2E)
11. Beth Ann Hockey, Deborah Rossen-Knill, Beverly Spejewski, Matthew Stone, and Stephen Isard. 1997. Can you predict answers to Y/N questions? Yes, No and Stuff. In Proceedings of Eurospeech 1997, pages 2267–2270. [find](https://scholar.google.com/scholar?q=Beth+Ann+Hockey%2C+Deborah+Rossen%2DKnill%2C+Beverly+Spejewski%2C+Matthew+Stone%2C+and+Stephen+Isard%2E+1997%2E+Can+you+predict+answers+to+Y%2FN+questions%3F+Yes%2C+No+and+Stuff%2E+In+Proceedings+of+Eurospeech+1997%2C+pages+)
12. Laurence R Horn. 1972. On the Semantic Properties of Logical Operators in English. Ph.D. thesis, UCLA, Los Angeles. [find](https://scholar.google.com/scholar?q=Laurence+R+Horn%2E+1972%2E+On+the+Semantic+Properties+of+Logical+Operators+in+English%2E+Ph%2ED%2E+thesis%2C+UCLA%2C+Los+Angeles%2E)
13. Nan Hu, Paul A. Pavlou, and Jennifer Zhang. 2006. Can online reviews reveal a product’s true quality?: Empirical findings and analytical modeling of online word-of-mouth communication. In Proceedings of Electronic Commerce (EC), pages 324–330. [find](https://scholar.google.com/scholar?q=Nan+Hu%2C+Paul+A%2E+Pavlou%2C+and+Jennifer+Zhang%2E+2006%2E+Can+online+reviews+reveal+a+product%E2%80%99s+true+quality%3F%3A+Empirical+findings+and+analytical+modeling+of+online+word%2Dof%2Dmouth+communication%2E+In+Proceedings+)
14. Daniel Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual, draft 13. Technical Report 97-02, University of Colorado, Boulder Institute of Cognitive Science. [find](https://scholar.google.com/scholar?q=Daniel+Jurafsky%2C+Elizabeth+Shriberg%2C+and+Debra+Biasca%2E+1997%2E+Switchboard+SWBD%2DDAMSL+shallow%2Ddiscourse%2Dfunction+annotation+coders+manual%2C+draft+13%2E+Technical+Report+97%2D02%2C+University+of+Colorado%2C+Bould)
15. Hans Kamp and Barbara H. Partee. 1995. Prototype theory and compositionality. Cognition, 57(2):129– 191. [find](https://scholar.google.com/scholar?q=Hans+Kamp+and+Barbara+H%2E+Partee%2E+1995%2E+Prototype+theory+and+compositionality%2E+Cognition%2C+57%282%29%3A129%E2%80%93+191%2E)
16. Christopher Kennedy and Louise McNally. 2005. Scale structure and the semantic typology of gradable predicates. Language, 81(2):345–381. [find](https://scholar.google.com/scholar?q=Christopher+Kennedy+and+Louise+McNally%2E+2005%2E+Scale+structure+and+the+semantic+typology+of+gradable+predicates%2E+Language%2C+81%282%29%3A345%E2%80%93381%2E)
17. Christopher Kennedy. 2007. Vagueness and grammar: The semantics of relative and absolute gradable adjectives. Linguistics and Philosophy, 30(1):1–45. [find](https://scholar.google.com/scholar?q=Christopher+Kennedy%2E+2007%2E+Vagueness+and+grammar%3A+The+semantics+of+relative+and+absolute+gradable+adjectives%2E+Linguistics+and+Philosophy%2C+30%281%29%3A1%E2%80%9345%2E)
18. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Meeting of the Association of Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dan+Klein+and+Christopher+D%2E+Manning%2E+2003%2E+Accurate+unlexicalized+parsing%2E+In+Proceedings+of+the+41st+Meeting+of+the+Association+of+Computational+Linguistics%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2003-klein-accurate-unlexicalized-parsing/)
19. Stephen C. Levinson. 2000. Presumptive Meanings: The Theory of Generalized Conversational Implicature. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Stephen+C%2E+Levinson%2E+2000%2E+Presumptive+Meanings%3A+The+Theory+of+Generalized+Conversational+Implicature%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
20. Saif Mohammad, Bonnie Dorr, and Graeme Hirst. 2008. Computing word-pair antonymy. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-2008). [find](https://scholar.google.com/scholar?q=Saif+Mohammad%2C+Bonnie+Dorr%2C+and+Graeme+Hirst%2E+2008%2E+Computing+word%2Dpair+antonymy%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+Computational+Natural+Languag)
21. Robert Munro, Steven Bethard, Victor Kuperman, Vicky Tzuyin Lai, Robin Melnick, Christopher Potts, Tyler Schnoebelen, and Harry Tily. 2010. Crowdsourcing and language studies: The new generation of linguistic data. In NAACL 2010 Workshop on Creating Speech and Language Data With Amazon’s Mechanical Turk. [find](https://scholar.google.com/scholar?q=Robert+Munro%2C+Steven+Bethard%2C+Victor+Kuperman%2C+Vicky+Tzuyin+Lai%2C+Robin+Melnick%2C+Christopher+Potts%2C+Tyler+Schnoebelen%2C+and+Harry+Tily%2E+2010%2E+Crowdsourcing+and+language+studies%3A+The+new+generation+of+li)
22. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1):1–135. [find](https://scholar.google.com/scholar?q=Bo+Pang+and+Lillian+Lee%2E+2008%2E+Opinion+mining+and+sentiment+analysis%2E+Foundations+and+Trends+in+Information+Retrieval%2C+2%281%29%3A1%E2%80%93135%2E)
23. C. Raymond Perrault and James F. Allen. 1980. A plan-based analysis of indirect speech acts. American Journal of Computational Linguistics, 6(3-4):167–182. [find](https://scholar.google.com/scholar?q=C%2E+Raymond+Perrault+and+James+F%2E+Allen%2E+1980%2E+A+plan%2Dbased+analysis+of+indirect+speech+acts%2E+American+Journal+of+Computational+Linguistics%2C+6%283%2D4%29%3A167%E2%80%93182%2E)
24. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of KDD-2008. [find](https://scholar.google.com/scholar?q=Victor+S%2E+Sheng%2C+Foster+Provost%2C+and+Panagiotis+G%2E+Ipeirotis%2E+2008%2E+Get+another+label%3F+improving+data+quality+and+data+mining+using+multiple%2C+noisy+labelers%2E+In+Proceedings+of+KDD%2D2008%2E)
25. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-2008). [find](https://scholar.google.com/scholar?q=Rion+Snow%2C+Brendan+O%E2%80%99Connor%2C+Daniel+Jurafsky%2C+and+Andrew+Y%2E+Ng%2E+2008%2E+Cheap+and+fast+%E2%80%93+but+is+it+good%3F+evaluating+non%2Dexpert+annotations+for+natural+language+tasks%2E+In+Proceedings+of+the+Conference+on)
26. Henk Zeevat. 1994. Questions and exhaustivity in update semantics. In Harry Bunt, Reinhard Muskens, and Gerrit Rentier, editors, Proceedings of the International Workshop on Computational Semantics, pages 211–221. [find](https://scholar.google.com/scholar?q=Henk+Zeevat%2E+1994%2E+Questions+and+exhaustivity+in+update+semantics%2E+In+Harry+Bunt%2C+Reinhard+Muskens%2C+and+Gerrit+Rentier%2C+editors%2C+Proceedings+of+the+International+Workshop+on+Computational+Semantics%2C+p)