[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/michael-2018-crowdsourcing-question-answer-meaning-representations/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Crowdsourcing Question-Answer Meaning Representations

# Crowdsourcing Question-Answer Meaning Representations

**Authors:** [Julian Michael](https://md.mova.elamur.ai/free/en/docs/papers/authors/julian-michael/), [Gabriel Stanovsky](https://md.mova.elamur.ai/free/en/docs/papers/authors/gabriel-stanovsky/), [Luheng He](https://md.mova.elamur.ai/free/en/docs/papers/authors/luheng-he/), [Ido Dagan](https://md.mova.elamur.ai/free/en/docs/papers/authors/ido-dagan/), [Luke Zettlemoyer](https://md.mova.elamur.ai/free/en/docs/papers/authors/luke-zettlemoyer/) · **Year:** 2018 · **Venue:** NAACL 2018 (short papers) **Link:** [aclanthology.org/N18-2089](https://aclanthology.org/N18-2089/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (paper); the QAMR data's licence to be checked before test use

[**68** citations](https://openalex.org/works?filter=cites:W2771039833) openalex-snapshot · 2026-10-14[**44** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/michael-2018-crowdsourcing-question-answer-meaning-representations/) of the 140673 papers whose bibliographies we read[Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)](https://aclanthology.org/volumes/N18-2/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/N18-2089/)
- [PDF](https://aclanthology.org/N18-2089.pdf)
- [DOI 10.18653/v1/n18-2089](https://doi.org/10.18653/v1/n18-2089)
- [OpenAlex](https://openalex.org/W2771039833)

**Authors (5)**

1. [Julian Michael](https://openalex.org/A5072681909) [Semantic Scholar](https://www.semanticscholar.org/author/38614754) · h 20 · 8897 citations · 56 works
2. [Gabriel Stanovsky](https://openalex.org/A5082136238) [ORCID](https://orcid.org/0000-0002-2420-8979) · [Semantic Scholar](https://www.semanticscholar.org/author/2157025) · h 21 · 2331 citations · 140 works
3. [Luheng He](https://openalex.org/A5025887224) [Semantic Scholar](https://www.semanticscholar.org/author/2265599) · h 25 · 5772 citations · 46 works
4. [Ido Dagan](https://openalex.org/A5033006575) [ORCID](https://orcid.org/0000-0003-3415-3374) · [Semantic Scholar](https://www.semanticscholar.org/author/7465342) · h 59 · 16076 citations · 427 works
5. [Luke Zettlemoyer](https://openalex.org/A5067919401) [ORCID](https://orcid.org/0009-0008-8296-0764) · [Semantic Scholar](https://www.semanticscholar.org/author/1982950) · h 91 · 78722 citations · 492 works

Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart.

## What it is about

A QAMR represents a sentence's predicate-argument structure as a set of free-form wh-question / answer-span pairs about all its content words, not only verbs («How old is Pierre Vinken? – 61 years old», «What will he join the board as? – nonexecutive director»). Crowd workers get up to four target words and write as many pairs as they can, each containing a target word, with bonuses for more; two validators answer or reject each question. The data has over 5,000 sentences (Wikinews, Wikipedia, a PTB sample) and about 100,000 questions. On shared PTB sentences, single annotators cover over 60% of the relations in PropBank, NomBank and QA-SRL, and five cover over 90%. The free questions bring in words not in the sentence (38.7% of questions), such as types (*What country…*) and relations (*Who did Baruch work for?*, *last name*), and capture implicit arguments and inferred relations.

## Ideas

- I1. A gold-free round-trip check for the frame reader and student: for every frame Mova reads from a lesson line, generate one wh-question per filled role (agent → «Who V patient?», place → «Where did agent V?», cause → «Why did …?») and ask the student on the same text; the answer must be the role's filler. Failures point at reader/question mismatches (word order, `statement_order`, roles named differently in questions). Report the round-trip rate per role on the course lessons; run twice for determinism.
- I2. Cover the non-verbal relations QAMR shows to be frequent: age («61 years old»), apposition titles («Jeremiah's scribe» → work for), names («What is the judge's last name?»), nationality and place-in-place («What state is the town in?»). Check which of these Mova's `Fact` frames already produce (`is a`, `has`, `lives in`) by counting course questions of each shape in `world knowledge diagnose` step 1–2 failures.
- I3. Use typed wh-phrases as answer constraints: «What town…», «Which animal…» require an option whose head is a hyponym of the type (WordNet, as for `nominals.tsv`). An option of the wrong type is unsupported however well its words overlap. Measure on «right option outscored» cases.

## For Mova

[26. Questions](https://md.mova.elamur.ai/free/en/docs/languages/en/26-questions/) (wh-questions), `30-pragmatics-overview.md` §5, `36-discourse-relations.md` §9 (questions under discussion). Code: `prag/src/frames.rs` (question generation from frames, `Ask::Role`), `world/src/knowledge.rs` (round trip, typed options), `world/src/qframe.rs`. Measures: round-trip rate per role, diagnosis step shares, course tests and exams.

Topics: [Semantic role labeling](https://md.mova.elamur.ai/free/en/docs/papers/topics/semantic-role-labeling/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/n18-2089](https://doi.org/10.18653/v1/n18-2089)

```
@inproceedings{michael2018crowdsourcing,
  title = {{Crowdsourcing Question-Answer Meaning Representations}},
  author = {Michael, Julian and Stanovsky, Gabriel and He, Luheng and Dagan, Ido and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2089},
  url = {https://doi.org/10.18653/v1/n18-2089}
}
```

- [← previous Cross-Lingual and Cross-Domain Transfer Learning for POS Tagging in Historical Germanic Low-Resource Languages](https://md.mova.elamur.ai/free/en/docs/papers/miani-2026-cross-lingual-cross-domain-transfer/)
- [next → Modeling the Evolution of English Noun Compounds with Feature-Rich Diachronic Compositionality Prediction](https://md.mova.elamur.ai/free/en/docs/papers/miletic-2025-modeling-evolution-english-noun-compounds/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The nombank project: An interim report. In HLT-NAACL 2004 workshop: Frontiers in corpus annotation. [find](https://scholar.google.com/scholar?q=Adam+Meyers%2C+Ruth+Reeves%2C+Catherine+Macleod%2C+Rachel+Szekely%2C+Veronika+Zielinska%2C+Brian+Young%2C+and+Ralph+Grishman%2E+2004%2E+The+nombank+project%3A+An+interim+report%2E+In+HLT%2DNAACL+2004+workshop%3A+Frontiers+in)
2. Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (ucca). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Omri+Abend+and+Ari+Rappoport%2E+2013%2E+Universal+conceptual+cognitive+annotation+%28ucca%29%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/abend-2013-universal-conceptual-cognitive-annotation-ucca/)
3. Preslav Nakov. 2008. Noun Compound Interpretation Using Paraphrasing Verbs: Feasibility Study. Springer Berlin Heidelberg, Berlin, Heidelberg. [find](https://scholar.google.com/scholar?q=Preslav+Nakov%2E+2008%2E+Noun+Compound+Interpretation+Using+Paraphrasing+Verbs%3A+Feasibility+Study%2E+Springer+Berlin+Heidelberg%2C+Berlin%2C+Heidelberg%2E)
4. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV). [↗](https://doi.org/10.1007/s11263-016-0966-6)
5. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Tri+Nguyen%2C+Mir+Rosenberg%2C+Xia+Song%2C+Jianfeng+Gao%2C+Saurabh+Tiwary%2C+Rangan+Majumder%2C+and+Li+Deng%2E+2016%2E+MS+MARCO%3A+A+human+generated+machine+reading+comprehension+dataset%2E+In+Advances+in+Neural+Informat)
6. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of the 17th International Conference on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Collin+F+Baker%2C+Charles+J+Fillmore%2C+and+John+B+Lowe%2E+1998%2E+The+berkeley+framenet+project%2E+In+Proceedings+of+the+17th+International+Conference+on+Computational+Linguistics%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lex-1998-baker-berkeley-framenet/)
7. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics. [↗](https://doi.org/10.1162/0891201053630264) [our card](https://md.mova.elamur.ai/free/en/docs/papers/palmer-2005-proposition-bank-annotated-corpus-semantic/)
8. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the Linguistic Annotation Workshop. [find](https://scholar.google.com/scholar?q=Laura+Banarescu%2C+Claire+Bonial%2C+Shu+Cai%2C+Madalina+Georgescu%2C+Kira+Griffitt%2C+Ulf+Hermjakob%2C+Kevin+Knight%2C+Philipp+Koehn%2C+Martha+Palmer%2C+and+Nathan+Schneider%2E+2013%2E+Abstract+meaning+representation+for+s) [our card](https://md.mova.elamur.ai/free/en/docs/papers/banarescu-2013-abstract-meaning-representation-sembanking/)
9. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. [↗](https://arxiv.org/abs/1606.05250)
10. Valerio Basile, Johan Bos, Kilian Evang, and Noortje Venhuizen. 2012. Developing a large semantically annotated corpus. In Proceedings of the 2012 International Conference on Language Resources and Evaluation. [find](https://scholar.google.com/scholar?q=Valerio+Basile%2C+Johan+Bos%2C+Kilian+Evang%2C+and+Noortje+Venhuizen%2E+2012%2E+Developing+a+large+semantically+annotated+corpus%2E+In+Proceedings+of+the+2012+International+Conference+on+Language+Resources+and+Ev)
11. Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. Transactions of the Association for Computational Linguistics. [↗](https://doi.org/10.1162/tacl_a_00152)
12. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, pages 193–203. [find](https://scholar.google.com/scholar?q=Matthew+Richardson%2C+Christopher+JC+Burges%2C+and+Erin+Renshaw%2E+2013%2E+MCTest%3A+A+challenge+dataset+for+the+open%2Ddomain+machine+comprehension+of+text%2E+In+EMNLP%2C+pages+193%E2%80%93203%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/missing-2013-richardson-mctest-challenge-dataset-open-domain/)
13. Manjuan Duan, Ethan Hill, and Michael White. 2016. Generating disambiguating paraphrases for structurally ambiguous sentences. In Proceedings of the 10th Linguistic Annotation Workshop. [find](https://scholar.google.com/scholar?q=Manjuan+Duan%2C+Ethan+Hill%2C+and+Michael+White%2E+2016%2E+Generating+disambiguating+paraphrases+for+structurally+ambiguous+sentences%2E+In+Proceedings+of+the+10th+Linguistic+Annotation+Workshop%2E)
14. Filipe de Sá Mesquita, Jordan Schmidek, and Denilson Barbosa. 2013. Effectiveness and efficiency of open relation extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 447–457. [find](https://scholar.google.com/scholar?q=Filipe+de+Sa%CC%81+Mesquita%2C+Jordan+Schmidek%2C+and+Denilson+Barbosa%2E+2013%2E+Effectiveness+and+efficiency+of+open+relation+extraction%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natural+Lan)
15. Dan Flickinger, Stephan Oepen, and Emily M. Bender. 2017. Sustainable Development and Refinement of Complex Linguistic Annotations at Scale. [↗](https://doi.org/10.1007/978-94-024-0881-2_14)
16. Matthew Gerber and Joyce Y Chai. 2010. Beyond nombank: A study of implicit arguments for nominal predicates. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1583–1592. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matthew+Gerber+and+Joyce+Y+Chai%2E+2010%2E+Beyond+nombank%3A+A+study+of+implicit+arguments+for+nominal+predicates%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C)
17. Rudolf Schneider, Tom Oberhauser, Tobias Klatt, Felix A. Gers, and Alexander Loser. 2017. Analysing errors of open information extraction systems. CoRR, abs/1707.07499. [↗](https://arxiv.org/abs/1707.07499)
18. Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Luheng+He%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2015%2E+Question%2Danswer+driven+semantic+role+labeling%3A+Using+natural+language+to+annotate+natural+language%2E+In+Proceedings+of+the+2015+Conference+on+Empirica) [our card](https://md.mova.elamur.ai/free/en/docs/papers/he-2015-question-answer-driven-semantic-role/)
19. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603. [↗](https://arxiv.org/abs/1611.01603)
20. Luheng He, Julian Michael, Mike Lewis, and Luke Zettlemoyer. 2016. Human-in-the-loop parsing. [find](https://scholar.google.com/scholar?q=Luheng+He%2C+Julian+Michael%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2016%2E+Human%2Din%2Dthe%2Dloop+parsing%2E)
21. Gabriel Stanovsky and Ido Dagan. 2016a. Annotating and predicting non-restrictive noun phrase modifications. In Proceedings of the 54rd Annual Meeting of the Association for Computational Linguistics (ACL 2016). [find](https://scholar.google.com/scholar?q=Gabriel+Stanovsky+and+Ido+Dagan%2E+2016a%2E+Annotating+and+predicting+non%2Drestrictive+noun+phrase+modifications%2E+In+Proceedings+of+the+54rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+)
22. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+J%2E+Bethard%2C+and+David+Mc%2DClosky%2E+2014%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+Association+for+Computation)
23. Gabriel Stanovsky and Ido Dagan. 2016b. Creating a large benchmark for open information extraction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gabriel+Stanovsky+and+Ido+Dagan%2E+2016b%2E+Creating+a+large+benchmark+for+open+information+extraction%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+A)
24. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics. based on their agreement. In this way, the quality of workers in either stage could be quickly assessed so spammers or low-quality workers could be disqualified before causing much damage. [↗](https://doi.org/10.21236/ada273556) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
25. Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. Question constraints In both stages, we define a valid question to Chenguang Wang, Alan Akbik, laura chiticariu, Yunyao Li, Fei Xia, and Anbang Xu. 2017. Crowd-in-the-loop: A hybrid approach for annotating semantic roles. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1914–1923. Association for Computational Linguistics. (1) contain at least one word from the sentence, (2) be about the sentence’s meaning, (3) be answered obviously and explicitly in the sentence, Keenon Werling, Arun Tejasvi Chaganty, Percy S Liang, and Christopher D Manning. 2015. On-the-job learning with bayesian decision theory. In Advances in Neural Information Processing Systems. (4) not be a yes/no question, and (5) not be redundant, Aaron Steven White, Drew Reisinger, Sakaguchi Tim Vieira, Sheng Zhang Rachel Rudinger Kyle Rawlins, and Benjamin Van Durme. 2016. Universal decompositional semantics on universal dependencies. [find](https://scholar.google.com/scholar?q=Gabriel+Stanovsky%2C+Julian+Michael%2C+Luke+Zettlemoyer%2C+and+Ido+Dagan%2E+2018%2E+Supervised+open+information+extraction%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association) [our card](https://md.mova.elamur.ai/free/en/docs/papers/white-2016-universal-decompositional-semantics-universal-dependencies/)