[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/weston-2015-towards-ai-complete-question-answering/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

# Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

**Authors:** [Jason Weston](https://md.mova.elamur.ai/free/en/docs/papers/authors/jason-weston/), Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, Tomas Mikolov · **Year:** 2015 · **Venue:** arXiv:1502.05698 (later ICLR 2016) **Link:** [arxiv.org/abs/1502.05698](https://arxiv.org/abs/1502.05698) (DOI [10.48550/arXiv.1502.05698](https://doi.org/10.48550/arXiv.1502.05698)) **License of the paper:** arXiv non-exclusive distribution license — [arxiv.org/licenses/nonexclusive-distrib/1.0](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) (the bAbI data is released under [CC BY 3.0](https://creativecommons.org/licenses/by/3.0/))

[**363** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/weston-2015-towards-ai-complete-question-answering/) of the 140673 papers whose bibliographies we readarXiv:1502.05698[licence](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

- [Publisher's page](https://arxiv.org/abs/1502.05698)
- [PDF](https://arxiv.org/pdf/1502.05698)
- [arXiv 1502.05698](https://arxiv.org/abs/1502.05698)

**Authors (7)**

1. Jason Weston
2. Antoine Bordes
3. Sumit Chopra
4. Alexander M. Rush
5. Bart van Merriënboer
6. Armand Joulin
7. Tomas Mikolov

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## Summary

The paper proposes bAbI, a suite of 20 synthetic question-answering tasks meant to work like unit tests for text understanding and reasoning. Stories are produced by a simple world simulator, similar to a text adventure, in which characters move between locations and pick up, drop or hand over objects. Each task targets one skill: retrieving one, two or three supporting facts, argument relations, yes/no questions, counting, lists, negation, coreference, time reasoning, basic deduction and induction, positional and size reasoning, path finding and agents' motivations. Answers are single words or short lists, so scoring is unambiguous, and the data is noise-free so a human can answer everything. As an illustration, the authors extend Memory Networks with adaptive numbers of memory hops, multi-word answers and better sentence representations; the improved models solve many but not all tasks, and a structured-SVM pipeline with external NLP tools fails several. The value of the suite is that passing or failing a given task points to a specific missing ability.

## How Mova uses it

- `world/src/babi.rs` — the `world babi` command solves all 20 tasks with a reader built on the `en` parse tree, verb classes, and a first-level `relation` block, without training on the questions.
- [Relations of space, size and time](https://md.mova.elamur.ai/free/en/docs/global/seeds/shortcuts/relations/) — the relation seed (inverses such as north/south, plane vectors, transitivity of bigger/smaller, *fits inside* = smaller than, order of parts of the day) cites bAbI tasks 4, 14, 17, 18 and 19; facts come from the story, inferences come from this shared seed.
- `world babi-learn` — agents' motivations (hungry → kitchen, etc.) are learned automatically from the training split and written back as a seed.
- `world babi-contrast` — contrast variants in the spirit of Gardner et al. 2020 (new names, places, objects, synonym verbs) test that the reader does not depend on bAbI's fixed vocabulary.

## Effectiveness in Mova

From the "Ideas from papers through the bench" table in Mova's development notes: test **99.8%, 20/20 tasks ≥ 95%**; training split 99.7% (not tuned to). Motivations were learned from the training split (375/375). Majority voting in induction was worse than using the freshest example (97.6% vs 99.5%). On contrast variants accuracy first dropped to 74.6%, exposing parser defects (*attic* tagged ADJ, *Taras* lemmatised as "tara", *Following* as a verb, *is* as a main verb) and two lexicon gaps (*hurry*, *abandon*); after fixing them contrast accuracy reached **99.7%, 20/20**, with no drop on the original (99.8%).

---

👨‍🔬💥

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 1502.05698](https://arxiv.org/abs/1502.05698)

```
@misc{weston2015ai,
  title = {{Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks}},
  author = {Weston, Jason and Bordes, Antoine and Chopra, Sumit and Rush, Alexander M. and Merriënboer, Bart van and Joulin, Armand and Mikolov, Tomas},
  year = {2015},
  eprint = {1502.05698},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1502.05698}
}
```

- [← previous BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://md.mova.elamur.ai/free/en/docs/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/)
- [next → Essentials of English Grammar for the Use of Schools](https://md.mova.elamur.ai/free/en/docs/papers/whitney-1877-essentials-english-grammar-use/)
- [↑ back to papers](https://md.mova.elamur.ai/free/en/docs/papers/)

## References

34 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bache, K. and Lichman, M. UCI machine learning repository, [http://archive.ics.uci.edu/ml](http://archive.ics.uci.edu/ml). Berant, Jonathan, Chou, Andrew, Frostig, Roy, and Liang, Percy. Semantic parsing on freebase from question-answer pairs. In EMNLP, pp. 1533–1544, 2013. [↗](http://archive.ics.uci.edu/ml)
2. Berant, Jonathan, Srikumar, Vivek, Chen, Pei-Chun, Huang, Brad, Manning, Christopher D, Vander Linden, Abby, Harding, Brittany, and Clark, Peter. Modeling biological processes for reading comprehension. In Proc. EMNLP, 2014. [find](https://scholar.google.com/scholar?q=Berant%2C+Jonathan%2C+Srikumar%2C+Vivek%2C+Chen%2C+Pei%2DChun%2C+Huang%2C+Brad%2C+Manning%2C+Christopher+D%2C+Vander+Linden%2C+Abby%2C+Harding%2C+Brittany%2C+and+Clark%2C+Peter%2E+Modeling+biological+processes+for+reading+comprehensio)
3. Bordes, Antoine, Usunier, Nicolas, Collobert, Ronan, and Weston, Jason. Towards understanding situated natural language. In AISTATS, 2010. [find](https://scholar.google.com/scholar?q=Bordes%2C+Antoine%2C+Usunier%2C+Nicolas%2C+Collobert%2C+Ronan%2C+and+Weston%2C+Jason%2E+Towards+understanding+situated+natural+language%2E+In+AISTATS%2C+2010%2E)
4. Bordes, Antoine, Usunier, Nicolas, Chopra, Sumit, and Weston, Jason. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075, 2015. [find](https://scholar.google.com/scholar?q=Bordes%2C+Antoine%2C+Usunier%2C+Nicolas%2C+Chopra%2C+Sumit%2C+and+Weston%2C+Jason%2E+Large%2Dscale+simple+question+answering+with+memory+networks%2E+arXiv+preprint+arXiv%3A1506%2E02075%2C+2015%2E)
5. Chen, David L and Mooney, Raymond J. Learning to interpret natural language navigation instructions from observations. San Francisco, CA, pp. 859–865, 2011. [find](https://scholar.google.com/scholar?q=Chen%2C+David+L+and+Mooney%2C+Raymond+J%2E+Learning+to+interpret+natural+language+navigation+instructions+from+observations%2E+San+Francisco%2C+CA%2C+pp%2E+859%E2%80%93865%2C+2011%2E)
6. Collobert, Ronan, Weston, Jason, Bottou, Léon, Karlen, Michael, Kavukcuoglu, Koray, and Kuksa, Pavel. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537, 2011. [find](https://scholar.google.com/scholar?q=Collobert%2C+Ronan%2C+Weston%2C+Jason%2C+Bottou%2C+Le%CC%81on%2C+Karlen%2C+Michael%2C+Kavukcuoglu%2C+Koray%2C+and+Kuksa%2C+Pavel%2E+Natural+language+processing+%28almost%29+from+scratch%2E+The+Journal+of+Machine+Learning+Research%2C+12%3A2)
7. Curtis, Jon, Matthews, Gavin, and Baxter, David. On the effective use of cyc in a question answering system. In IJCAI Workshop on Knowledge and Reasoning for Answering Questions, pp. 61–70, 2005. [find](https://scholar.google.com/scholar?q=Curtis%2C+Jon%2C+Matthews%2C+Gavin%2C+and+Baxter%2C+David%2E+On+the+effective+use+of+cyc+in+a+question+answering+system%2E+In+IJCAI+Workshop+on+Knowledge+and+Reasoning+for+Answering+Questions%2C+pp%2E+61%E2%80%9370%2C+2005%2E)
8. Fader, Anthony, Zettlemoyer, Luke, and Etzioni, Oren. Paraphrase-driven learning for open question answering. In ACL, pp. 1608–1618, 2013. [find](https://scholar.google.com/scholar?q=Fader%2C+Anthony%2C+Zettlemoyer%2C+Luke%2C+and+Etzioni%2C+Oren%2E+Paraphrase%2Ddriven+learning+for+open+question+answering%2E+In+ACL%2C+pp%2E+1608%E2%80%931618%2C+2013%2E)
9. Fader, Anthony, Zettlemoyer, Luke, and Etzioni, Oren. Open question answering over curated and extracted knowledge bases. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1156–1165. ACM, 2014. [find](https://scholar.google.com/scholar?q=Fader%2C+Anthony%2C+Zettlemoyer%2C+Luke%2C+and+Etzioni%2C+Oren%2E+Open+question+answering+over+curated+and+extracted+knowledge+bases%2E+In+Proceedings+of+the+20th+ACM+SIGKDD+international+conference+on+Knowledge+di)
10. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. arXiv:1410.5401, 2014. [find](https://scholar.google.com/scholar?q=Graves%2C+Alex%2C+Wayne%2C+Greg%2C+and+Danihelka%2C+Ivo%2E+arXiv%3A1410%2E5401%2C+2014%2E)
11. Halevy, Alon, Norvig, Peter, and Pereira, Fernando. The unreasonable effectiveness of data. Intelligent Systems, IEEE, 24(2):8–12, 2009. [find](https://scholar.google.com/scholar?q=Halevy%2C+Alon%2C+Norvig%2C+Peter%2C+and+Pereira%2C+Fernando%2E+The+unreasonable+effectiveness+of+data%2E+Intelligent+Systems%2C+IEEE%2C+24%282%29%3A8%E2%80%9312%2C+2009%2E)
12. Hermann, Karl Moritz, Kočiský, Tomáš, Grefenstette, Edward, Espeholt, Lasse, Kay, Will, Suleyman, Mustafa, and Blunsom, Phil. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS), 2015. URL http://arxiv.org/abs/1506.03340. [↗](http://arxiv.org/abs/1506.03340)
13. Hill, Felix, Bordes, Antoine, Chopra, Sumit, and Weston, Jason. The goldilocks principle: Reading children’s books with explicit memory representation s. arXiv preprint arXiv:1511.02301, 2015. [find](https://scholar.google.com/scholar?q=Hill%2C+Felix%2C+Bordes%2C+Antoine%2C+Chopra%2C+Sumit%2C+and+Weston%2C+Jason%2E+The+goldilocks+principle%3A+Reading+children%E2%80%99s+books+with+explicit+memory+representation+s%2E+arXiv+preprint+arXiv%3A1511%2E02301%2C+2015%2E)
14. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+Sepp+and+Schmidhuber%2C+Ju%CC%88rgen%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A+1735%E2%80%931780%2C+1997%2E)
15. Kumar, Ankit, Irsoy, Ozan, Su, Jonathan, Bradbury, James, English, Robert, Pierce, B̃rian, Ondruska, Peter, Gulrajani, Ishaan, and Socher, Richard. Ask me anything: Dynamic memory networks for natural language processing. http://arxiv.org/abs/1506.07285, 2015. 2013. URL Neural turing machines. arXiv… [↗](http://arxiv.org/abs/1506.07285)
16. Liang, Percy. Lambda dependency-based compositional semantics. arXiv preprint arXiv:1309.4408, 2013. [find](https://scholar.google.com/scholar?q=Liang%2C+Percy%2E+Lambda+dependency%2Dbased+compositional+semantics%2E+arXiv+preprint+arXiv%3A1309%2E4408%2C+2013%2E)
17. Liang, Percy, Jordan, Michael I, and Klein, Dan. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389–446, 2013. [find](https://scholar.google.com/scholar?q=Liang%2C+Percy%2C+Jordan%2C+Michael+I%2C+and+Klein%2C+Dan%2E+Learning+dependency%2Dbased+compositional+semantics%2E+Computational+Linguistics%2C+39%282%29%3A389%E2%80%93446%2C+2013%2E)
18. Minsky, Marvin and Papert, Seymour. Perceptron: an introduction to computational geometry. The MIT Press, Cambridge, expanded edition, 19:88, 1969. [find](https://scholar.google.com/scholar?q=Minsky%2C+Marvin+and+Papert%2C+Seymour%2E+Perceptron%3A+an+introduction+to+computational+geometry%2E+The+MIT+Press%2C+Cambridge%2C+expanded+edition%2C+19%3A88%2C+1969%2E)
19. Montfort, Nick. Twisty Little Passages: an approach to interactive fiction. Mit Press, 2005. [find](https://scholar.google.com/scholar?q=Montfort%2C+Nick%2E+Twisty+Little+Passages%3A+an+approach+to+interactive+fiction%2E+Mit+Press%2C+2005%2E)
20. Müller, K-R, Smola, Alex J, Rätsch, Gunnar, Schölkopf, Bernhard, Kohlmorgen, Jens, and Vapnik, Vladimir. Predicting time series with support vector machines. In Artificial Neural NetworksI-CANN’97, pp. 999–1004. Springer, 1997. [find](https://scholar.google.com/scholar?q=Mu%CC%88ller%2C+K%2DR%2C+Smola%2C+Alex+J%2C+Ra%CC%88tsch%2C+Gunnar%2C+Scho%CC%88lkopf%2C+Bernhard%2C+Kohlmorgen%2C+Jens%2C+and+Vapnik%2C+Vladimir%2E+Predicting+time+series+with+support+vector+machines%2E+In+Artificial+Neural+NetworksI%2DCANN%E2%80%9997%2C)
21. Ng, Andrew Y, Jordan, Michael I, Weiss, Yair, et al. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems, 2:849–856, 2002. [find](https://scholar.google.com/scholar?q=Ng%2C+Andrew+Y%2C+Jordan%2C+Michael+I%2C+Weiss%2C+Yair%2C+et+al%2E+On+spectral+clustering%3A+Analysis+and+an+algorithm%2E+Advances+in+neural+information+processing+systems%2C+2%3A849%E2%80%93856%2C+2002%2E)
22. Peng, Baolin, Lu, Zhengdong, Li, Hang, and Wong, Kam-Fai. Towards neural network-based reasoning. arXiv preprint arXiv:1508.05508, 2015. — [arXiv:1508.05508](https://arxiv.org/abs/1508.05508) [find](https://scholar.google.com/scholar?q=Peng%2C+Baolin%2C+Lu%2C+Zhengdong%2C+Li%2C+Hang%2C+and+Wong%2C+Kam%2DFai%2E+Towards+neural+network%2Dbased+reasoning%2E+arXiv+preprint+arXiv%3A1508%2E05508%2C+2015%2E+%E2%80%94+%5BarXiv%3A1508%2E05508%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1508%2E05508%29)
23. Raghunathan, Karthik, Lee, Heeyoung, Rangarajan, Sudarshan, Chambers, Nathanael, Surdeanu, Mihai, Jurafsky, Dan, and Manning, Christopher. A multi-pass sieve for coreference resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 492–501. Associatio… [find](https://scholar.google.com/scholar?q=Raghunathan%2C+Karthik%2C+Lee%2C+Heeyoung%2C+Rangarajan%2C+Sudarshan%2C+Chambers%2C+Nathanael%2C+Surdeanu%2C+Mihai%2C+Jurafsky%2C+Dan%2C+and+Manning%2C+Christopher%2E+A+multi%2Dpass+sieve+for+coreference+resolution%2E+In+Proceedings) [our card](https://md.mova.elamur.ai/free/en/docs/papers/raghunathan-2010-multi-pass-sieve-coreference-resolution/)
24. Richardson, Matthew, Burges, Christopher JC, and Renshaw, Erin. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, pp. 193–203, 2013. [find](https://scholar.google.com/scholar?q=Richardson%2C+Matthew%2C+Burges%2C+Christopher+JC%2C+and+Renshaw%2C+Erin%2E+Mctest%3A+A+challenge+dataset+for+the+open%2Ddomain+machine+comprehension+of+text%2E+In+EMNLP%2C+pp%2E+193%E2%80%93203%2C+2013%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/missing-2013-richardson-mctest-challenge-dataset-open-domain/)
25. Rumelhart, David E, Hinton, Geoffrey E, and Williams, Ronald J. Learning internal representations by error propagation. Technical report, DTIC Document, 1985. [find](https://scholar.google.com/scholar?q=Rumelhart%2C+David+E%2C+Hinton%2C+Geoffrey+E%2C+and+Williams%2C+Ronald+J%2E+Learning+internal+representations+by+error+propagation%2E+Technical+report%2C+DTIC+Document%2C+1985%2E)
26. Soon, Wee Meng, Ng, Hwee Tou, and Lim, Daniel Chung Yong. A machine learning approach to coreference resolution of noun phrases. Computational linguistics, 27(4):521–544, 2001. [find](https://scholar.google.com/scholar?q=Soon%2C+Wee+Meng%2C+Ng%2C+Hwee+Tou%2C+and+Lim%2C+Daniel+Chung+Yong%2E+A+machine+learning+approach+to+coreference+resolution+of+noun+phrases%2E+Computational+linguistics%2C+27%284%29%3A521%E2%80%93544%2C+2001%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/soon-2001-machine-learning-approach-coreference-resolution/)
27. Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and Fergus, Rob. End-to-end memory networks. Proceedings of NIPS, 2015. [find](https://scholar.google.com/scholar?q=Sukhbaatar%2C+Sainbayar%2C+Szlam%2C+Arthur%2C+Weston%2C+Jason%2C+and+Fergus%2C+Rob%2E+End%2Dto%2Dend+memory+networks%2E+Proceedings+of+NIPS%2C+2015%2E)
28. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014. [find](https://scholar.google.com/scholar?q=Sutskever%2C+Ilya%2C+Vinyals%2C+Oriol%2C+and+Le%2C+Quoc+VV%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+3104%E2%80%933112%2C+2014%2E)
29. UzZaman, Naushad, Llorens, Hector, Allen, James, Derczynski, Leon, Verhagen, Marc, and Pustejovsky, James. Tempeval-3: Evaluating events, time expressions, and temporal relations. arXiv preprint arXiv:1206.5333, 2012. [find](https://scholar.google.com/scholar?q=UzZaman%2C+Naushad%2C+Llorens%2C+Hector%2C+Allen%2C+James%2C+Derczynski%2C+Leon%2C+Verhagen%2C+Marc%2C+and+Pustejovsky%2C+James%2E+Tempeval%2D3%3A+Evaluating+events%2C+time+expressions%2C+and+temporal+relations%2E+arXiv+preprint+arXiv)
30. Weston, Jason, Chopra, Sumit, and Bordes, Antoine. Memory networks. CoRR, abs/1410.3916, 2014. [find](https://scholar.google.com/scholar?q=Weston%2C+Jason%2C+Chopra%2C+Sumit%2C+and+Bordes%2C+Antoine%2E+Memory+networks%2E+CoRR%2C+abs%2F1410%2E3916%2C+2014%2E)
31. Winograd, Terry. Understanding natural language. Cognitive psychology, 3(1):1–191, 1972. [find](https://scholar.google.com/scholar?q=Winograd%2C+Terry%2E+Understanding+natural+language%2E+Cognitive+psychology%2C+3%281%29%3A1%E2%80%93191%2C+1972%2E)
32. Yao, Xuchen, Berant, Jonathan, and Van Durme, Benjamin. Freebase qa: Information extraction or semantic parsing? ACL 2014, pp. 82, 2014. [find](https://scholar.google.com/scholar?q=Yao%2C+Xuchen%2C+Berant%2C+Jonathan%2C+and+Van+Durme%2C+Benjamin%2E+Freebase+qa%3A+Information+extraction+or+semantic+parsing%3F+ACL+2014%2C+pp%2E+82%2C+2014%2E)
33. Yu, Mo, Gormley, Matthew R, and Dredze, Mark. Factor-based compositional embedding models. NIPS 2014 workshop on Learning Semantics, 2014. [find](https://scholar.google.com/scholar?q=Yu%2C+Mo%2C+Gormley%2C+Matthew+R%2C+and+Dredze%2C+Mark%2E+Factor%2Dbased+compositional+embedding+models%2E+NIPS+2014+workshop+on+Learning+Semantics%2C+2014%2E)
34. Zhu, Xiaojin, Ghahramani, Zoubin, Lafferty, John, et al. Semi-supervised learning using gaussian fields and harmonic functions. In ICML, volume 3, pp. 912–919, 2003. [find](https://scholar.google.com/scholar?q=Zhu%2C+Xiaojin%2C+Ghahramani%2C+Zoubin%2C+Lafferty%2C+John%2C+et+al%2E+Semi%2Dsupervised+learning+using+gaussian+fields+and+harmonic+functions%2E+In+ICML%2C+volume+3%2C+pp%2E+912%E2%80%93919%2C+2003%2E)