[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/fitzgerald-2018-large-scale-qa-srl-parsing/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Large-Scale QA-SRL Parsing

# Large-Scale QA-SRL Parsing

**Authors:** Nicholas FitzGerald, [Julian Michael](https://md.mova.elamur.ai/free/en/docs/papers/authors/julian-michael/), [Luheng He](https://md.mova.elamur.ai/free/en/docs/papers/authors/luheng-he/) and [Luke Zettlemoyer](https://md.mova.elamur.ai/free/en/docs/papers/authors/luke-zettlemoyer/) · **Year:** 2018 · **Venue:** Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) **Link:** [aclanthology.org/P18-1191](https://aclanthology.org/P18-1191/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) (ACL Anthology)

[**84** citations](https://openalex.org/works?filter=cites:W2798326384) openalex · 2026-10-19[**58** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/fitzgerald-2018-large-scale-qa-srl-parsing/) of the 140673 papers whose bibliographies we read[Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://aclanthology.org/volumes/P18-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/P18-1191/)
- [PDF](https://aclanthology.org/P18-1191.pdf)
- [DOI 10.18653/v1/p18-1191](https://doi.org/10.18653/v1/p18-1191)
- [OpenAlex](https://openalex.org/W2798326384)

**Authors (4)**

1. [Nicholas FitzGerald](https://openalex.org/A5073439637) [Semantic Scholar](https://www.semanticscholar.org/author/143883142) · h 15 · 2880 citations · 25 works
2. [Julian Michael](https://openalex.org/A5072681909) [Semantic Scholar](https://www.semanticscholar.org/author/38614754) · h 20 · 9171 citations · 56 works
3. [Luheng He](https://openalex.org/A5025887224) [Semantic Scholar](https://www.semanticscholar.org/author/2265599) · h 26 · 6321 citations · 46 works
4. [Luke Zettlemoyer](https://openalex.org/A5067919401) [ORCID](https://orcid.org/0009-0008-8296-0764) · [Semantic Scholar](https://www.semanticscholar.org/author/1982950) · h 93 · 82307 citations · 519 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

The paper releases QA-SRL Bank 2.0, which has 265,140 valid question-answer pairs for 64,018 sentences and 133,479 verbs across Wikipedia, Wikinews and science textbooks. A streamlined crowdsourcing interface, with an autocomplete automaton over the seven question slots and a separate validation stage with two validators, was used; the whole set took nine days and cost $43,647, or 16.5 cents per valid question. The parser is a pipeline: span detection (a BIO tagger, or a span-based model that scores every span) followed by question generation (a local slot classifier, or a sequential LSTM decoder). The final parser reaches 82.6% question accuracy and 77.6% span accuracy in a human evaluation. Re-running the parser to over-generate questions and validating them with the crowd adds 20% valid questions to the training and development sets.

## Ideas

- I1. The seven-slot question template (WH, AUX, SBJ, TGT, OBJ, PREP, MISC) as a deterministic grammar for school questions in `world/src/qa.rs`. The paper's autocomplete automaton is a grammaticality filter, and it can be written as a finite-state check in Rust.
- I2. Span detection from UD subtrees: candidate answers are the subtrees of the verb's dependents, scored by a count-based rule, and kept above a threshold. Measure span F1 on dev with the paper's IOU of at least 0.5.
- I3. Data expansion as a deterministic loop: generate candidate question-answer pairs from the parser, keep those not already covered by an answer, and validate them by a second, independent source. Keep the held-out test untouched.
- I4. Negative control: shuffle the question slots of correct questions. Question accuracy must drop, otherwise the slots are not doing any work.

## For Mova

The chapters are [26. Questions](https://md.mova.elamur.ai/free/en/docs/languages/en/26-questions/) and [13. Verbs](https://md.mova.elamur.ai/free/en/docs/languages/en/13-verbs/). The code is `world/src/qa.rs` and `en/src/parse.rs` for the subtrees. The measures are span F1 at IOU 0.5 on dev, question accuracy against a human-checked sample, and the negative control. Run each twice.

Topics: [Semantic role labeling](https://md.mova.elamur.ai/free/en/docs/papers/topics/semantic-role-labeling/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/p18-1191](https://doi.org/10.18653/v1/p18-1191)

```
@inproceedings{fitzgerald2018large,
  title = {{Large-Scale QA-SRL Parsing}},
  author = {FitzGerald, Nicholas and Michael, Julian and He, Luheng and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1191},
  url = {https://doi.org/10.18653/v1/p18-1191}
}
```

- [← previous Penn-Helsinki Parsed Corpus of Early Modern English: First Parsing Results and Analysis](https://md.mova.elamur.ai/free/en/docs/papers/findings-2022-kulick-penn-helsinki-parsed-corpus-early/)
- [next → A Text Normalisation System for Non-Standard English Words](https://md.mova.elamur.ai/free/en/docs/papers/flint-2017-text-normalisation-system-non-standard/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (UCCA). In ACL 2013. [find](https://scholar.google.com/scholar?q=Omri+Abend+and+Ari+Rappoport%2E+2013%2E+Universal+conceptual+cognitive+annotation+%28UCCA%29%2E+In+ACL+2013%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/abend-2013-universal-conceptual-cognitive-annotation-ucca/)
2. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The Berkeley Framenet project. In ICCL 1998. [find](https://scholar.google.com/scholar?q=Collin+F+Baker%2C+Charles+J+Fillmore%2C+and+John+B+Lowe%2E+1998%2E+The+Berkeley+Framenet+project%2E+In+ICCL+1998%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lex-1998-baker-berkeley-framenet/)
3. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In 7th Linguistic Annotation Workshop and Interoperability with Discourse. [find](https://scholar.google.com/scholar?q=Laura+Banarescu%2C+Claire+Bonial%2C+Shu+Cai%2C+Madalina+Georgescu%2C+Kira+Griffitt%2C+Ulf+Hermjakob%2C+Kevin+Knight%2C+Philipp+Koehn%2C+Martha+Palmer%2C+and+Nathan+Schneider%2E+2013%2E+Abstract+meaning+representation+for+s) [our card](https://md.mova.elamur.ai/free/en/docs/papers/banarescu-2013-abstract-meaning-representation-sembanking/)
4. Valerio Basile, Johan Bos, Kilian Evang, and Noortje Venhuizen. 2012. Developing a large semantically annotated corpus. In LREC 2012. [find](https://scholar.google.com/scholar?q=Valerio+Basile%2C+Johan+Bos%2C+Kilian+Evang%2C+and+Noortje+Venhuizen%2E+2012%2E+Developing+a+large+semantically+annotated+corpus%2E+In+LREC+2012%2E)
5. Claire Bonial, Olga Babko-Malaya, Jinho D Choi, Jena Hwang, and Martha Palmer. 2010. Propbank annotation guidelines. [find](https://scholar.google.com/scholar?q=Claire+Bonial%2C+Olga+Babko%2DMalaya%2C+Jinho+D+Choi%2C+Jena+Hwang%2C+and+Martha+Palmer%2E+2010%2E+Propbank+annotation+guidelines%2E)
6. Ronan Collobert and Jason Weston. 2007. Fast semantic extraction using a novel neural network architecture. In ACL 2007. [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+2007%2E+Fast+semantic+extraction+using+a+novel+neural+network+architecture%2E+In+ACL+2007%2E)
7. Manjuan Duan, Ethan Hill, and Michael White. 2016. Generating disambiguating paraphrases for structurally ambiguous sentences. In 10th Linguistic Annotation Workshop. [find](https://scholar.google.com/scholar?q=Manjuan+Duan%2C+Ethan+Hill%2C+and+Michael+White%2E+2016%2E+Generating+disambiguating+paraphrases+for+structurally+ambiguous+sentences%2E+In+10th+Linguistic+Annotation+Workshop%2E)
8. Josef Ruppenhofer, Michael Ellsworth, Miriam RL Petruck, Christopher R Johnson, and Jan Scheffczyk. 2016. FrameNet II: Extended theory and practice. Institut für Deutsche Sprache, Bibliothek. [find](https://scholar.google.com/scholar?q=Josef+Ruppenhofer%2C+Michael+Ellsworth%2C+Miriam+RL+Petruck%2C+Christopher+R+Johnson%2C+and+Jan+Scheffczyk%2E+2016%2E+FrameNet+II%3A+Extended+theory+and+practice%2E+Institut+fu%CC%88r+Deutsche+Sprache%2C+Bibliothek%2E)
9. Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In EMNLP 2015. [find](https://scholar.google.com/scholar?q=Nicholas+FitzGerald%2C+Oscar+Ta%CC%88ckstro%CC%88m%2C+Kuzman+Ganchev%2C+and+Dipanjan+Das%2E+2015%2E+Semantic+role+labeling+with+neural+network+factors%2E+In+EMNLP+2015%2E)
10. Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In NIPS 2015. [find](https://scholar.google.com/scholar?q=Rupesh+K+Srivastava%2C+Klaus+Greff%2C+and+Ju%CC%88rgen+Schmidhuber%2E+2015%2E+Training+very+deep+networks%2E+In+NIPS+2015%2E)
11. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In NIPS 2016. [find](https://scholar.google.com/scholar?q=Yarin+Gal+and+Zoubin+Ghahramani%2E+2016%2E+A+theoretically+grounded+application+of+dropout+in+recurrent+neural+networks%2E+In+NIPS+2016%2E)
12. Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In ACL 2018. [find](https://scholar.google.com/scholar?q=Luheng+He%2C+Kenton+Lee%2C+Omer+Levy%2C+and+Luke+Zettlemoyer%2E+2018%2E+Jointly+predicting+predicates+and+arguments+in+neural+semantic+role+labeling%2E+In+ACL+2018%2E)
13. Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and whats next. In ACL 2017. [find](https://scholar.google.com/scholar?q=Luheng+He%2C+Kenton+Lee%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2017%2E+Deep+semantic+role+labeling%3A+What+works+and+whats+next%2E+In+ACL+2017%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/he-2017-deep-semantic-role-labeling-what/)
14. Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In EMNLP 2015. [find](https://scholar.google.com/scholar?q=Luheng+He%2C+Mike+Lewis%2C+and+Luke+S%2E+Zettlemoyer%2E+2015%2E+Question%2Danswer+driven+semantic+role+labeling%3A+Using+natural+language+to+annotate+natural+language%2E+In+EMNLP+2015%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/he-2015-question-answer-driven-semantic-role/)
15. Luheng He, Julian Michael, Mike Lewis, and Luke Zettlemoyer. 2016. Human-in-the-loop parsing. In EMNLP 2016. [find](https://scholar.google.com/scholar?q=Luheng+He%2C+Julian+Michael%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2016%2E+Human%2Din%2Dthe%2Dloop+parsing%2E+In+EMNLP+2016%2E)
16. Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In CVPR 2017. [find](https://scholar.google.com/scholar?q=Aniruddha+Kembhavi%2C+Minjoon+Seo%2C+Dustin+Schwenk%2C+Jonghyun+Choi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2017%2E+Are+you+smarter+than+a+sixth+grader%3F+textbook+question+answering+for+multimodal+machine+comp)
17. Gabriel Stanovsky and Ido Dagan. 2016. Creating a large benchmark for open information extraction. In EMNLP 2016. [find](https://scholar.google.com/scholar?q=Gabriel+Stanovsky+and+Ido+Dagan%2E+2016%2E+Creating+a+large+benchmark+for+open+information+extraction%2E+In+EMNLP+2016%2E)
18. Swabha Swayamdipta, Sam Thomson, Chris Dyer, and Noah A Smith. 2017. Frame-semantic parsing with softmax-margin segmental rnns and a syntactic scaffold. arXiv preprint arXiv:1706.09528. [↗](https://arxiv.org/abs/1706.09528)
19. Adam R Teichert, Adam Poliak, Benjamin Van Durme, and Matthew R Gormley. 2017. Semantic proto-role labeling. In AAAI 2017, pages 4459–4466. [find](https://scholar.google.com/scholar?q=Adam+R+Teichert%2C+Adam+Poliak%2C+Benjamin+Van+Durme%2C+and+Matthew+R+Gormley%2E+2017%2E+Semantic+proto%2Drole+labeling%2E+In+AAAI+2017%2C+pages+4459%E2%80%934466%2E)
20. Chenguang Wang, Alan Akbik, Laura Chiticariu, Yunyao Li, Fei Xia, and Anbang Xu. 2018. Crowd-in-the-loop: A hybrid approach for annotating semantic roles. In EMNLP 2017. [find](https://scholar.google.com/scholar?q=Chenguang+Wang%2C+Alan+Akbik%2C+Laura+Chiticariu%2C+Yunyao+Li%2C+Fei+Xia%2C+and+Anbang+Xu%2E+2018%2E+Crowd%2Din%2Dthe%2Dloop%3A+A+hybrid+approach+for+annotating+semantic+roles%2E+In+EMNLP+2017%2E)
21. Aaron Steven White, Kyle Rawlins, and Benjamin Van Durme. 2017. The semantic proto-role linking model. In ACL 2017. [find](https://scholar.google.com/scholar?q=Aaron+Steven+White%2C+Kyle+Rawlins%2C+and+Benjamin+Van+Durme%2E+2017%2E+The+semantic+proto%2Drole+linking+model%2E+In+ACL+2017%2E)
22. Bishan Yang and Tom Mitchell. 2017. A joint sequential and relational model for frame-semantic parsing. In EMNLP 2017, pages 1247–1256. [find](https://scholar.google.com/scholar?q=Bishan+Yang+and+Tom+Mitchell%2E+2017%2E+A+joint+sequential+and+relational+model+for+frame%2Dsemantic+parsing%2E+In+EMNLP+2017%2C+pages+1247%E2%80%931256%2E)
23. Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In ACL 2015. [find](https://scholar.google.com/scholar?q=Jie+Zhou+and+Wei+Xu%2E+2015%2E+End%2Dto%2Dend+learning+of+semantic+role+labeling+using+recurrent+neural+networks%2E+In+ACL+2015%2E)
24. Kenton Lee, Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2017. End-to-end neural coreference resolution. In EMNLP 2017. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Luheng+He%2C+Mike+Lewis%2C+and+Luke+S%2E+Zettlemoyer%2E+2017%2E+End%2Dto%2Dend+neural+coreference+resolution%2E+In+EMNLP+2017%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lee-2017-end-end-neural-coreference-resolution/)
25. Kenton Lee, Shimi Salant, Tom Kwiatkowski, Ankur Parikh, Dipanjan Das, and Jonathan Berant. 2016. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436. [↗](https://arxiv.org/abs/1611.01436)
26. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In ACL 2014, pages 55–60. [find](https://scholar.google.com/scholar?q=Christopher+Manning%2C+Mihai+Surdeanu%2C+John+Bauer%2C+Jenny+Finkel%2C+Steven+Bethard%2C+and+David+McClosky%2E+2014%2E+The+stanford+corenlp+natural+language+processing+toolkit%2E+In+ACL+2014%2C+pages+55%E2%80%9360%2E)
27. Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2018. Crowdsourcing question-answer meaning representations. In NAACL 2018. [find](https://scholar.google.com/scholar?q=Julian+Michael%2C+Gabriel+Stanovsky%2C+Luheng+He%2C+Ido+Dagan%2C+and+Luke+Zettlemoyer%2E+2018%2E+Crowdsourcing+question%2Danswer+meaning+representations%2E+In+NAACL+2018%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/michael-2018-crowdsourcing-question-answer-meaning-representations/)
28. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics. [find](https://scholar.google.com/scholar?q=Martha+Palmer%2C+Daniel+Gildea%2C+and+Paul+Kingsbury%2E+2005%2E+The+proposition+bank%3A+An+annotated+corpus+of+semantic+roles%2E+Computational+Linguistics%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/palmer-2005-proposition-bank-annotated-corpus-semantic/)
29. Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. TACL. [find](https://scholar.google.com/scholar?q=Drew+Reisinger%2C+Rachel+Rudinger%2C+Francis+Ferraro%2C+Craig+Harman%2C+Kyle+Rawlins%2C+and+Benjamin+Van+Durme%2E+2015%2E+Semantic+proto%2Droles%2E+TACL%2E)