[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/eldan-2023-tinystories-small-language-models-still/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

# TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

**Authors:** Ronen Eldan and Yuanzhi Li · **Year:** 2023 · **Venue:** arXiv:2305.07759 (preprint, Microsoft Research) **Link:** [arxiv.org/abs/2305.07759](https://arxiv.org/abs/2305.07759) · **Licence:** arXiv non-exclusive distribution licence — summary only

[**44** citations](https://openalex.org/works?filter=cites:W4376652984) openalex-snapshot · 2026-10-14[**158** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/eldan-2023-tinystories-small-language-models-still/) of the 140673 papers whose bibliographies we read[arXiv:2305.07759](https://openalex.org/S4306400194)[licence](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

- [Publisher's page](https://arxiv.org/abs/2305.07759)
- [PDF](https://arxiv.org/pdf/2305.07759)
- [arXiv 2305.07759](https://arxiv.org/abs/2305.07759)
- [OpenAlex](https://openalex.org/W4376652984)

**Authors (2)**

1. [Ronen Eldan](https://openalex.org/A5011758954) [ORCID](https://orcid.org/0000-0002-0678-741X) · h 23 · 3688 citations · 93 works
2. [Yuanzhi Li](https://openalex.org/A5101879892) [ORCID](https://orcid.org/0009-0004-4418-9308) · h 32 · 7209 citations · 148 works

Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart.

## What it is about

The paper introduces TinyStories, a synthetic corpus of short stories written by GPT-3.5 and GPT-4 with only words that a 3–4-year-old would know (about 1,500 basic words). Each story is generated from three random words (noun, verb, adjective) and a random set of features such as dialogue or a twist, to force diversity. Small GPT-Neo models trained on it, below 10M parameters or with a single transformer block, write fluent multi-paragraph stories with almost perfect grammar. An evaluation framework asks GPT-4 to grade completions on grammar, creativity, consistency and plot. Grammar appears first in small models; consistency and creativity need larger hidden size; one-layer models fail at following instructions. A 28M-parameter model with eight layers already beats the 1.5B GPT2-XL on some of the paper's factual, reasoning and consistency prompts. Training takes under 30 hours on one V100 GPU. Memorisation is ruled out by n-gram overlap with the training set.

## Ideas

- I1. A closed-vocabulary parse test: generate sentences only from the child-vocabulary word list, and measure LAS and UAS on that subset with `en parse-eval`. A closed vocabulary separates syntax from lexical coverage.
- I2. Score Mova's school answers on separate axes, like the paper's rubric, so that grammar and consistency gains are reported apart. Keep the counters in `world/src/knowledge.rs`.
- I3. An entity-tracking test: story openings where the continuation must keep names, places and objects. Check in `world/src/story.rs` that a mention after a gap refers to the same entity. The paper's one-layer failures show where the hard cases are.
- I4. Overlap check for any generated text: report the n-gram overlap with the lesson corpus next to each output, as the paper does before claiming diversity.

## For Mova

Chapters: [19. The lexicon: vocabulary, multiword expressions and register](https://md.mova.elamur.ai/free/en/docs/languages/en/19-lexicon-mwe-register/) (the basic vocabulary) and [35. Reference and coreference](https://md.mova.elamur.ai/free/en/docs/languages/en/35-reference-and-coreference/) (entity tracking). Code: `en/src/parse.rs`, `world/src/story.rs`, `world/src/knowledge.rs`. Measure: closed-vocabulary LAS/UAS and entity-tracking accuracy, run twice.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 2305.07759](https://arxiv.org/abs/2305.07759)

```
@misc{eldan2023tinystories,
  title = {{TinyStories: How Small Can Language Models Be and Still Speak Coherent English?}},
  author = {Eldan, Ronen and Li, Yuanzhi},
  year = {2023},
  eprint = {2305.07759},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.07759}
}
```

- [← previous Can predicate-argument relationships be extracted from UD trees?](https://md.mova.elamur.ai/free/en/docs/papers/ek-2021-can-predicate-argument-relationships-be/)
- [next → Shape of Synth to Come: Why We Should Use Synthetic Data for English Surface Realization](https://md.mova.elamur.ai/free/en/docs/papers/elder-2020-shape-synth-come-why-we/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

33 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Common crawl. Accessed: 2019. [find](https://scholar.google.com/scholar?q=Common+crawl%2E+Accessed%3A+2019%2E)
2. Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020. [↗](https://arxiv.org/abs/2012.09816)
3. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Leo+Gao%2C+Phil+Wang%2C+Connor+Leahy%2C+and+Stella+Biderman%2E+GPT%2DNeo%3A+Large+Scale+Autoregressive+Language+Modeling+with+Mesh%2DTensorflow%2C+March+2021%2E)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+)
5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. [↗](https://arxiv.org/abs/2303.12712)
6. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341, 2019. [↗](https://arxiv.org/abs/1906.04341)
7. John H Flavell, Eleanor R Flavell, Frances L Green, and Louis J Moses. Young children’s understanding of fact beliefs versus value beliefs. Child development, 61(4):915–928, 1990. [↗](https://doi.org/10.2307/1130865)
8. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018. [↗](https://arxiv.org/abs/1803.03635)
9. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [↗](https://arxiv.org/abs/2101.00027)
10. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. [↗](https://arxiv.org/abs/1503.02531)
11. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. [↗](https://arxiv.org/abs/2203.15556)
12. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. [↗](https://arxiv.org/abs/1904.09751)
13. Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869–6898, 2017. [↗](https://doi.org/10.48550/arxiv.1609.07061)
14. Samy Jelassi, Michael Sander, and Yuanzhi Li. Vision transformers provably learn spatial structure. Advances in Neural Information Processing Systems, 35:37822–37836, 2022. [↗](https://doi.org/10.52202/068431-2741)
15. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. [↗](https://arxiv.org/abs/1705.03551)
16. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. [↗](https://arxiv.org/abs/2001.08361)
17. Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+The+winograd+schema+challenge%2E+In+Thirteenth+international+conference+on+the+principles+of+knowledge+representation+and+reasoning%2C+2012%2E)
18. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. arXiv preprint arXiv:1506.01066, 2015. [↗](https://arxiv.org/abs/1506.01066)
19. Yuchen Li, Yuanzhi Li, and Andrej Risteski. How do transformers learn topic structure: Towards a mechanistic understanding. arXiv preprint arXiv:2303.04245, 2023. [↗](https://arxiv.org/abs/2303.04245)
20. Wick Miller and Susan Ervin. The development of grammar in child language. Monographs of the Society for Research in Child Development, pages 9–34, 1964. [↗](https://doi.org/10.2307/1165752)
21. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E)
22. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. [↗](https://arxiv.org/abs/1606.06031)
23. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E)
24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. [↗](https://doi.org/10.48550/arxiv.1910.10683)
25. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. [↗](https://arxiv.org/abs/1910.01108)
26. Michael Santacroce, Zixin Wen, Yelong Shen, and Yuanzhi Li. What matters in the structured pruning of generative language models? arXiv preprint arXiv:2302.03773, 2023. [↗](https://arxiv.org/abs/2302.03773)
27. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. [↗](https://arxiv.org/abs/2206.04615)
28. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984, 2020. [↗](https://arxiv.org/abs/2004.02984)
29. Wilson L Taylor. “cloze procedure”: A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953. [find](https://scholar.google.com/scholar?q=Wilson+L+Taylor%2E+%E2%80%9Ccloze+procedure%E2%80%9D%3A+A+new+tool+for+measuring+readability%2E+Journalism+quarterly%2C+30%284%29%3A415%E2%80%93433%2C+1953%2E)
30. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418, 2019. [↗](https://arxiv.org/abs/1905.09418)
31. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019. [↗](https://arxiv.org/abs/1911.00359)
32. Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1–191, 1972. [↗](https://doi.org/10.1016/0010-0285(72)90002-3)
33. Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017. [↗](https://arxiv.org/abs/1709.00103)