[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/bengio-2009-curriculum-learning/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / Curriculum Learning

# Curriculum Learning

**Authors:** Yoshua Bengio, Jérôme Louradour, Ronan Collobert, [Jason Weston](https://md.mova.elamur.ai/free/en/docs/papers/authors/jason-weston/) · **Year:** 2009 · **Venue:** ICML 2009 **Link:** [ronan.collobert.com/pub/2009 curriculum icml.pdf](https://ronan.collobert.com/pub/2009_curriculum_icml.pdf) (author copy; DOI [10.1145/1553374.1553380](https://doi.org/10.1145/1553374.1553380)) **License of the paper:** author copy, read-only (no open license) — unknown

[**659** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/bengio-2009-curriculum-learning/) of the 140673 papers whose bibliographies we readICML 2009

- [Publisher's page](https://ronan.collobert.com/pub/2009_curriculum_icml.pdf)
- [PDF](https://ronan.collobert.com/pub/2009_curriculum_icml.pdf)

**Authors (4)**

1. Yoshua Bengio
2. Jérôme Louradour
3. Ronan Collobert
4. Jason Weston

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## Summary

The paper brings a pedagogical intuition into machine learning: present training examples in a meaningful order, from easy to hard, instead of at random. The authors formalise this as a sequence of training distributions that start concentrated on simple examples, gradually reweight towards harder ones and end at the target distribution. They interpret a curriculum as a kind of continuation method: the learner first optimises a smoothed, easier version of a non-convex objective and then moves towards the real one, which can steer it into better local minima. Experiments cover a toy convex setting, shape recognition and a neural language model whose vocabulary is grown step by step. Even simple two-stage curricula speed up convergence and improve generalisation, with the clearest gains on test data, so the curriculum also acts somewhat like a regulariser. The work coined the term and gave the theoretical framing for ordering training data in deep learning.

## How Mova uses it

- math/src/bin/mathsolve.rs, flag `--curriculum` of the `steps` model (word-problem solver as a transition system trained by a structured perceptron, math/src/steps.rs): before normal training the model gets 3 epochs on problems needing one operation, then 3 epochs on problems with at most two, then all problems in the usual order.
- The staging is by number of reduce operations in the gold action sequence, combined with the "worked examples from simple to complex" idea from Sweller.
- It is an optional experiment flag, not part of the default training.

## Effectiveness in Mova

Measured on SVAMP: 57.3% → 55.7% with the staged curriculum, i.e. no gain (recorded as "zero"). A naive variant that simply sorts all training problems by difficulty dropped to 50.0%, because it breaks perceptron averaging. The flag is therefore off by default.

---

👨‍🔬💥

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{bengio2009curriculum,
  title = {{Curriculum Learning}},
  author = {Bengio, Yoshua and Louradour, Jérôme and Collobert, Ronan and Weston, Jason},
  booktitle = {ICML 2009},
  year = {2009},
  url = {https://ronan.collobert.com/pub/2009_curriculum_icml.pdf}
}
```

- [← previous CogNIAC: high precision coreference with limited knowledge and linguistic resources](https://md.mova.elamur.ai/free/en/docs/papers/baldwin-1997-cogniac-high-precision-coreference-limited/)
- [next → Pre-annotation Matters: A Comparative Study on POS and Dependency Annotation for an Alsatian Dialect](https://md.mova.elamur.ai/free/en/docs/papers/bernhard-2025-pre-annotation-matters-comparative-study/)
- [↑ back to papers](https://md.mova.elamur.ai/free/en/docs/papers/)

## References

33 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Allgower, E. L., & Georg, K. (1980). Numerical continuation methods. An introduction. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Allgower%2C+E%2E+L%2E%2C+%26+Georg%2C+K%2E+%281980%29%2E+Numerical+continuation+methods%2E+An+introduction%2E+Springer%2DVerlag%2E)
2. Bengio, Y. (2009). Learning deep architectures for AI. Foundations & Trends in Mach. Learn., to appear. [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E+%282009%29%2E+Learning+deep+architectures+for+AI%2E+Foundations+%26+Trends+in+Mach%2E+Learn%2E%2C+to+appear%2E)
3. Bengio, Y., Ducharme, R., & Vincent, P. (2001). A neural probabilistic language model. Adv. Neural Inf. Proc. Sys. 13 (pp. 932–938). [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E%2C+Ducharme%2C+R%2E%2C+%26+Vincent%2C+P%2E+%282001%29%2E+A+neural+probabilistic+language+model%2E+Adv%2E+Neural+Inf%2E+Proc%2E+Sys%2E+13+%28pp%2E+932%E2%80%93938%29%2E)
4. Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. Adv. Neural Inf. Proc. Sys. 19 (pp. 153–160). [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E%2C+Lamblin%2C+P%2E%2C+Popovici%2C+D%2E%2C+%26+Larochelle%2C+H%2E+%282007%29%2E+Greedy+layer%2Dwise+training+of+deep+networks%2E+Adv%2E+Neural+Inf%2E+Proc%2E+Sys%2E+19+%28pp%2E+153%E2%80%93160%29%2E)
5. Cohn, D., Ghahramani, Z., & Jordan, M. (1995). Active learning with statistical models. Adv. Neural Inf. Proc. Sys. 7 (pp. 705–712). [find](https://scholar.google.com/scholar?q=Cohn%2C+D%2E%2C+Ghahramani%2C+Z%2E%2C+%26+Jordan%2C+M%2E+%281995%29%2E+Active+learning+with+statistical+models%2E+Adv%2E+Neural+Inf%2E+Proc%2E+Sys%2E+7+%28pp%2E+705%E2%80%93712%29%2E)
6. Coleman, T., & Wu, Z. (1994). Parallel continuation-based global optimization for molecular conformation and protein folding (Technical Report). Cornell University, Dept. of Computer Science. [find](https://scholar.google.com/scholar?q=Coleman%2C+T%2E%2C+%26+Wu%2C+Z%2E+%281994%29%2E+Parallel+continuation%2Dbased+global+optimization+for+molecular+conformation+and+protein+folding+%28Technical+Report%29%2E+Cornell+University%2C+Dept%2E+of+Computer+Science%2E)
7. Collobert, R., & Weston, J. (2008). A unified architecture for natural language processing: Deep neural networks with multitask learning. [find](https://scholar.google.com/scholar?q=Collobert%2C+R%2E%2C+%26+Weston%2C+J%2E+%282008%29%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E)
8. Int. Conf. Mach. Learn. 2008 (pp. 160–167). [find](https://scholar.google.com/scholar?q=Int%2E+Conf%2E+Mach%2E+Learn%2E+2008+%28pp%2E+160%E2%80%93167%29%2E)
9. Derényi, I., Geszti, T., & Györgyi, G. (1994). Generalization in the programed teaching of a perceptron. Physical Review E, 50, 3192–3200. [find](https://scholar.google.com/scholar?q=Dere%CC%81nyi%2C+I%2E%2C+Geszti%2C+T%2E%2C+%26+Gyo%CC%88rgyi%2C+G%2E+%281994%29%2E+Generalization+in+the+programed+teaching+of+a+perceptron%2E+Physical+Review+E%2C+50%2C+3192%E2%80%933200%2E)
10. Elman, J. L. (1993). Learning and development in neural networks: The importance of starting small. Cognition, 48, 781–799. [find](https://scholar.google.com/scholar?q=Elman%2C+J%2E+L%2E+%281993%29%2E+Learning+and+development+in+neural+networks%3A+The+importance+of+starting+small%2E+Cognition%2C+48%2C+781%E2%80%93799%2E)
11. Erhan, D., Manzagol, P.-A., Bengio, Y., Bengio, S., & Vincent, P. (2009). The difficulty of training deep architectures and the effect of unsupervised pre-training. AI & Stat.’2009. [find](https://scholar.google.com/scholar?q=Erhan%2C+D%2E%2C+Manzagol%2C+P%2E%2DA%2E%2C+Bengio%2C+Y%2E%2C+Bengio%2C+S%2E%2C+%26+Vincent%2C+P%2E+%282009%29%2E+The+difficulty+of+training+deep+architectures+and+the+effect+of+unsupervised+pre%2Dtraining%2E+AI+%26+Stat%2E%E2%80%992009%2E)
12. Freund, Y., & Haussler, D. (1994). Unsupervised learning of distributions on binary vectors using two layer networks (Technical Report UCSC-CRL-94-25). University of California, Santa Cruz. [find](https://scholar.google.com/scholar?q=Freund%2C+Y%2E%2C+%26+Haussler%2C+D%2E+%281994%29%2E+Unsupervised+learning+of+distributions+on+binary+vectors+using+two+layer+networks+%28Technical+Report+UCSC%2DCRL%2D94%2D25%29%2E+University+of+California%2C+Santa+Cruz%2E)
13. Håstad, J., & Goldmann, M. (1991). On the power of small-depth threshold circuits. Computational Complexity, 1, 113–129. [find](https://scholar.google.com/scholar?q=Ha%CC%8Astad%2C+J%2E%2C+%26+Goldmann%2C+M%2E+%281991%29%2E+On+the+power+of+small%2Ddepth+threshold+circuits%2E+Computational+Complexity%2C+1%2C+113%E2%80%93129%2E)
14. Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527–1554. [find](https://scholar.google.com/scholar?q=Hinton%2C+G%2E+E%2E%2C+Osindero%2C+S%2E%2C+%26+Teh%2C+Y%2E%2DW%2E+%282006%29%2E+A+fast+learning+algorithm+for+deep+belief+nets%2E+Neural+Computation%2C+18%2C+1527%E2%80%931554%2E)
15. Hinton, G. E., & Salakhutdinov, R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504–507. [find](https://scholar.google.com/scholar?q=Hinton%2C+G%2E+E%2E%2C+%26+Salakhutdinov%2C+R%2E+%282006%29%2E+Reducing+the+dimensionality+of+data+with+neural+networks%2E+Science%2C+313%2C+504%E2%80%93507%2E)
16. Krueger, K. A., & Dayan, P. (2009). Flexible shaping: how learning in small steps helps. Cognition, 110, 380–394. [find](https://scholar.google.com/scholar?q=Krueger%2C+K%2E+A%2E%2C+%26+Dayan%2C+P%2E+%282009%29%2E+Flexible+shaping%3A+how+learning+in+small+steps+helps%2E+Cognition%2C+110%2C+380%E2%80%93394%2E)
17. Larochelle, H., Erhan, D., Courville, A., Bergstra, J., & Bengio, Y. (2007). An empirical evaluation of deep architectures on problems with many factors of variation. Int. Conf. Mach. Learn. (pp. 473–480). [find](https://scholar.google.com/scholar?q=Larochelle%2C+H%2E%2C+Erhan%2C+D%2E%2C+Courville%2C+A%2E%2C+Bergstra%2C+J%2E%2C+%26+Bengio%2C+Y%2E+%282007%29%2E+An+empirical+evaluation+of+deep+architectures+on+problems+with+many+factors+of+variation%2E+Int%2E+Conf%2E+Mach%2E+Learn%2E+%28pp%2E+473%E2%80%93)
18. Peterson, G. B. (2004). A day of great illumination: B. F. Skinner’s discovery of shaping. Journal of the Experimental Analysis of Behavior, 82, 317–328. [find](https://scholar.google.com/scholar?q=Peterson%2C+G%2E+B%2E+%282004%29%2E+A+day+of+great+illumination%3A+B%2E+F%2E+Skinner%E2%80%99s+discovery+of+shaping%2E+Journal+of+the+Experimental+Analysis+of+Behavior%2C+82%2C+317%E2%80%93328%2E)
19. Ranzato, M., Boureau, Y., & LeCun, Y. (2008). Sparse feature learning for deep belief networks. Adv. Neural Inf. Proc. Sys. 20 (pp. 1185–1192). [find](https://scholar.google.com/scholar?q=Ranzato%2C+M%2E%2C+Boureau%2C+Y%2E%2C+%26+LeCun%2C+Y%2E+%282008%29%2E+Sparse+feature+learning+for+deep+belief+networks%2E+Adv%2E+Neural+Inf%2E+Proc%2E+Sys%2E+20+%28pp%2E+1185%E2%80%931192%29%2E)
20. Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007). Efficient learning of sparse representations with an energy-based model. Adv. Neural Inf. Proc. Sys. 19 (pp. 1137–1144). [find](https://scholar.google.com/scholar?q=Ranzato%2C+M%2E%2C+Poultney%2C+C%2E%2C+Chopra%2C+S%2E%2C+%26+LeCun%2C+Y%2E+%282007%29%2E+Efficient+learning+of+sparse+representations+with+an+energy%2Dbased+model%2E+Adv%2E+Neural+Inf%2E+Proc%2E+Sys%2E+19+%28pp%2E+1137%E2%80%931144%29%2E)
21. Rohde, D., & Plaut, D. (1999). Language acquisition in the absence of explicit negative evidence: How important is starting small? Cognition, 72, 67–109. [find](https://scholar.google.com/scholar?q=Rohde%2C+D%2E%2C+%26+Plaut%2C+D%2E+%281999%29%2E+Language+acquisition+in+the+absence+of+explicit+negative+evidence%3A+How+important+is+starting+small%3F+Cognition%2C+72%2C+67%E2%80%93109%2E)
22. Salakhutdinov, R., & Hinton, G. (2007). Learning a nonlinear embedding by preserving class neighbourhood structure. AI & Stat.’2007. [find](https://scholar.google.com/scholar?q=Salakhutdinov%2C+R%2E%2C+%26+Hinton%2C+G%2E+%282007%29%2E+Learning+a+nonlinear+embedding+by+preserving+class+neighbourhood+structure%2E+AI+%26+Stat%2E%E2%80%992007%2E)
23. Salakhutdinov, R., & Hinton, G. (2008). Using Deep Belief Nets to learn covariance kernels for Gaussian processes. Adv. Neural Inf. Proc. Sys. 20 (pp. 1249– 1256). [find](https://scholar.google.com/scholar?q=Salakhutdinov%2C+R%2E%2C+%26+Hinton%2C+G%2E+%282008%29%2E+Using+Deep+Belief+Nets+to+learn+covariance+kernels+for+Gaussian+processes%2E+Adv%2E+Neural+Inf%2E+Proc%2E+Sys%2E+20+%28pp%2E+1249%E2%80%93+1256%29%2E)
24. Salakhutdinov, R., Mnih, A., & Hinton, G. (2007). Restricted Boltzmann machines for collaborative filtering. [find](https://scholar.google.com/scholar?q=Salakhutdinov%2C+R%2E%2C+Mnih%2C+A%2E%2C+%26+Hinton%2C+G%2E+%282007%29%2E+Restricted+Boltzmann+machines+for+collaborative+filtering%2E)
25. Int. Conf. Mach. Learn. 2007 (pp. 791–798). [find](https://scholar.google.com/scholar?q=Int%2E+Conf%2E+Mach%2E+Learn%2E+2007+%28pp%2E+791%E2%80%93798%29%2E)
26. Sanger, T. D. (1994). Neural network learning control of robot manipulators using gradually increasing task difficulty. IEEE Trans. on Robotics and Automation, 10. [find](https://scholar.google.com/scholar?q=Sanger%2C+T%2E+D%2E+%281994%29%2E+Neural+network+learning+control+of+robot+manipulators+using+gradually+increasing+task+difficulty%2E+IEEE+Trans%2E+on+Robotics+and+Automation%2C+10%2E)
27. Schwenk, H., & Gauvain, J.-L. (2002). Connectionist language modeling for large vocabulary continuous speech recognition. International Conference on Acoustics, Speech and Signal Processing (pp. 765– 768). Orlando, Florida. [find](https://scholar.google.com/scholar?q=Schwenk%2C+H%2E%2C+%26+Gauvain%2C+J%2E%2DL%2E+%282002%29%2E+Connectionist+language+modeling+for+large+vocabulary+continuous+speech+recognition%2E+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28pp%2E+765%E2%80%93+)
28. Skinner, B. F. (1958). Reinforcement today. American Psychologist, 13, 94–99. [find](https://scholar.google.com/scholar?q=Skinner%2C+B%2E+F%2E+%281958%29%2E+Reinforcement+today%2E+American+Psychologist%2C+13%2C+94%E2%80%9399%2E)
29. Thrun, S. (1996). Explanation-based neural network learning: A lifelong learning approach. Boston, MA: Kluwer Academic Publishers. [find](https://scholar.google.com/scholar?q=Thrun%2C+S%2E+%281996%29%2E+Explanation%2Dbased+neural+network+learning%3A+A+lifelong+learning+approach%2E+Boston%2C+MA%3A+Kluwer+Academic+Publishers%2E)
30. Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008). Extracting and composing robust features with denoising autoencoders. Int. Conf. Mach. Learn. (pp. 1096–1103). [find](https://scholar.google.com/scholar?q=Vincent%2C+P%2E%2C+Larochelle%2C+H%2E%2C+Bengio%2C+Y%2E%2C+%26+Manzagol%2C+P%2E%2DA%2E+%282008%29%2E+Extracting+and+composing+robust+features+with+denoising+autoencoders%2E+Int%2E+Conf%2E+Mach%2E+Learn%2E+%28pp%2E+1096%E2%80%931103%29%2E)
31. Weston, J., Ratle, F., & Collobert, R. (2008). Deep learning via semi-supervised embedding. [find](https://scholar.google.com/scholar?q=Weston%2C+J%2E%2C+Ratle%2C+F%2E%2C+%26+Collobert%2C+R%2E+%282008%29%2E+Deep+learning+via+semi%2Dsupervised+embedding%2E)
32. Int. Conf. Mach. Learn. 2008 (pp. 1168–1175). [find](https://scholar.google.com/scholar?q=Int%2E+Conf%2E+Mach%2E+Learn%2E+2008+%28pp%2E+1168%E2%80%931175%29%2E)
33. Wu, Z. (1997). Global continuation for distance geometry problems. SIAM Journal of Optimization, 7, 814–836. [find](https://scholar.google.com/scholar?q=Wu%2C+Z%2E+%281997%29%2E+Global+continuation+for+distance+geometry+problems%2E+SIAM+Journal+of+Optimization%2C+7%2C+814%E2%80%93836%2E)