[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/ud-2022-yaari-aligned-multimodal-movie-treebank-audio/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank

# The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank

**Authors:** Adam Yaari, Jan DeWitt, Henry Hu et al. · **Year:** 2022 · **Venue:** Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022) **Link:** [aclanthology.org/2022.emnlp-main.648](https://aclanthology.org/2022.emnlp-main.648/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**4** citations](https://openalex.org/works?filter=cites:W4385573922) openalex · 2026-10-18[**3** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/ud-2022-yaari-aligned-multimodal-movie-treebank-audio/) of the 140673 papers whose bibliographies we read[Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing](https://aclanthology.org/volumes/2022.emnlp-main/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2022.emnlp-main.648/)
- [PDF](https://aclanthology.org/2022.emnlp-main.648.pdf)
- [DOI 10.18653/v1/2022.emnlp-main.648](https://doi.org/10.18653/v1/2022.emnlp-main.648)
- [OpenAlex](https://openalex.org/W4385573922)

**Authors (10)**

1. [Adam Yaari](https://openalex.org/A5005071392) [ORCID](https://orcid.org/0000-0002-1703-9097) · h 3 · 107 citations · 16 works
2. [Jan DeWitt](https://openalex.org/A5051911880) h 1 · 5 citations · 3 works
3. Henry Hu
4. [Bennett Stankovits](https://openalex.org/A5073598704) h 2 · 11 citations · 5 works
5. [Sue Felshin](https://openalex.org/A5052537898) h 11 · 467 citations · 38 works
6. [Yevgeni Berzak](https://openalex.org/A5023679744) [ORCID](https://orcid.org/0000-0003-4474-1727) · h 15 · 608 citations · 72 works
7. [Helena Aparicio](https://openalex.org/A5067302367) [ORCID](https://orcid.org/0000-0002-8619-1500) · h 3 · 149 citations · 19 works
8. [Boris Katz](https://openalex.org/A5002527110) [ORCID](https://orcid.org/0000-0002-3871-5824) · h 30 · 3456 citations · 198 works
9. [Ignacio Cases](https://openalex.org/A5068054004) h 12 · 758 citations · 33 works
10. [Andrei Barbu](https://openalex.org/A5018507577) [ORCID](https://orcid.org/0000-0001-7626-9266) · h 16 · 1082 citations · 87 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The Aligned Multimodal Movie Treebank (AMMT) takes dialogue from Hollywood films, with word-level alignment to the audio and video, POS tags and UD dependency parses. It has 31,264 sentences and 218,090 words. The authors also describe a tool for fast audio alignment during annotation. The parses were first made by Stanza and then corrected by annotators. Accuracy is lower on very short sentences (under 3 words) and on long ones (over 20 words) than on average, and the authors note a bias in favour of Stanza. The treebank is the first in UD that combines speech, video and parse.

## Ideas

- I1. `en parse-eval` with a length split (under 3, 3 to 20, over 20 words) in `en/src/parse.rs`, reported next to the mean, on AMMT and EWT (test only for AMMT).
- I2. `en` on movie dialogue: LAS of the perceptron parser on AMMT against EWT, to measure the dialogue gap for Mova's spoken data (test only).
- I3. A word-alignment check in the `annot` review loop, for spoken source material, measured by the share of tokens whose timing disagrees with a forced alignment (no new model).

## For Mova

The paper gives a spoken, scripted English test set with a length profile, which is useful for the `en` parser. The audio side is out of scope for Mova. Relevant chapters are `languages/en/38-dialogue-and-discourse-markers` and `37-register-and-variation`. Measure first the LAS of `en` on AMMT split by sentence length.

Topics: [Dependency parsing](https://md.mova.elamur.ai/free/en/docs/papers/topics/dependency-parsing/) · [Treebanks](https://md.mova.elamur.ai/free/en/docs/papers/topics/treebanks/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2022.emnlp-main.648](https://doi.org/10.18653/v1/2022.emnlp-main.648)

```
@inproceedings{yaari2022aligned,
  title = {{The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank}},
  author = {Yaari, Adam and DeWitt, Jan and Hu, Henry and Stankovits, Bennett and Felshin, Sue and Berzak, Yevgeni and Aparicio, Helena and Katz, Boris and Cases, Ignacio and Barbu, Andrei},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.648},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.648}
}
```

- [← previous Universal Dependencies and Semantics for English and Hebrew Child-directed Speech](https://md.mova.elamur.ai/free/en/docs/papers/ud-2022-szubert-universal-dependencies-semantics-english-hebrew/)
- [next → Difficulties in Handling Mathematical Expressions in Universal Dependencies](https://md.mova.elamur.ai/free/en/docs/papers/ud-2023-levine-difficulties-handling-mathematical-expressions-universal/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lars Ahrenberg. 2007. Lines: An English-Swedish Parallel Treebank. In Proceedings of the 16th Nordic Conference of Computational Linguistics (NODAL-IDA 2007). [find](https://scholar.google.com/scholar?q=Lars+Ahrenberg%2E+2007%2E+Lines%3A+An+English%2DSwedish+Parallel+Treebank%2E+In+Proceedings+of+the+16th+Nordic+Conference+of+Computational+Linguistics+%28NODAL%2DIDA+2007%29%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-2007-ahrenberg-lines-english-swedish-parallel-treebank/)
2. Yevgeni Berzak, Yan Huang, Andrei Barbu, Anna Korhonen, and Boris Katz. 2016. Anchoring and agreement in syntactic annotations. Conference on Empirical Methods in Natural Language Processing (EMNLP) 2016. [find](https://scholar.google.com/scholar?q=Yevgeni+Berzak%2C+Yan+Huang%2C+Andrei+Barbu%2C+Anna+Korhonen%2C+and+Boris+Katz%2E+2016%2E+Anchoring+and+agreement+in+syntactic+annotations%2E+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29+2)
3. Paul Boersma. 2001. Praat, a system for doing phonetics by computer. Glot. Int., 5(9):341–345. [find](https://scholar.google.com/scholar?q=Paul+Boersma%2E+2001%2E+Praat%2C+a+system+for+doing+phonetics+by+computer%2E+Glot%2E+Int%2E%2C+5%289%29%3A341%E2%80%93345%2E)
4. Andrew Caines, Michael McCarthy, and Paula Buttery. 2017. Parsing transcripts of speech. Conference on Empirical Methods in Natural Language Processing (EMNLP) 2017. [find](https://scholar.google.com/scholar?q=Andrew+Caines%2C+Michael+McCarthy%2C+and+Paula+Buttery%2E+2017%2E+Parsing+transcripts+of+speech%2E+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29+2017%2E)
5. John J Godfrey, Edward C Holliman, and Jane Mc-Daniel. 1992. SWITCHBOARD: Telephone Speech Corpus for Research and Development. In IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1. [find](https://scholar.google.com/scholar?q=John+J+Godfrey%2C+Edward+C+Holliman%2C+and+Jane+Mc%2DDaniel%2E+1992%2E+SWITCHBOARD%3A+Telephone+Speech+Corpus+for+Research+and+Development%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech%2C+and+Signal+Proces)
6. Google. 2020. Speech-to-text: Automatic speech recognition — Google Cloud. [find](https://scholar.google.com/scholar?q=Google%2E+2020%2E+Speech%2Dto%2Dtext%3A+Automatic+speech+recognition+%E2%80%94+Google+Cloud%2E)
7. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision. [find](https://scholar.google.com/scholar?q=Tsung%2DYi+Lin%2C+Michael+Maire%2C+Serge+Belongie%2C+James+Hays%2C+Pietro+Perona%2C+Deva+Ramanan%2C+Piotr+Doll%C3%A1r%2C+and+C+Lawrence+Zitnick%2E+2014%2E+Microsoft+COCO%3A+Common+Objects+in+Context%2E+In+European+Conference+on+C)
8. Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. [find](https://scholar.google.com/scholar?q=Mitchell+Marcus%2C+Beatrice+Santorini%2C+and+Mary+Ann+Marcinkiewicz%2E+1993%2E+Building+a+Large+Annotated+Corpus+of+English%3A+The+Penn+Treebank%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
9. Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and Music Signal Analysis in Python. In Proceedings of the 14th Python in Science Conference, volume 8. [find](https://scholar.google.com/scholar?q=Brian+McFee%2C+Colin+Raffel%2C+Dawen+Liang%2C+Daniel+PW+Ellis%2C+Matt+McVicar%2C+Eric+Battenberg%2C+and+Oriol+Nieto%2E+2015%2E+librosa%3A+Audio+and+Music+Signal+Analysis+in+Python%2E+In+Proceedings+of+the+14th+Python+in+)
10. Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal Dependencies v1: A Multilingual Treebank Collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Marie%2DCatherine+De+Marneffe%2C+Filip+Ginter%2C+Yoav+Goldberg%2C+Jan+Hajic%2C+Christopher+D+Manning%2C+Ryan+McDonald%2C+Slav+Petrov%2C+Sampo+Pyysalo%2C+Natalia+Silveira%2C+et+al%2E+2016%2E+Universal+Dependenci) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2016-nivre-universal-dependencies-v1-multilingual-treebank/)
11. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection. In Proceedings of the 12th Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Marie%2DCatherine+de+Marneffe%2C+Filip+Ginter%2C+Jan+Hajic%CC%8C%2C+Christopher+D%2E+Manning%2C+Sampo+Pyysalo%2C+Sebastian+Schuster%2C+Francis+Tyers%2C+and+Daniel+Zeman%2E+2020%2E+Universal+Dependencies+v2%3A+An+Eve) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2020-nivre-universal-dependencies-v2-evergrowing-multilingual/)
12. Joakim Nivre, Jens Nilsson, and Johan Hall. 2006. Talbanken05: A Swedish Treebank with Phrase Structure and Dependency Annotation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06). [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Jens+Nilsson%2C+and+Johan+Hall%2E+2006%2E+Talbanken05%3A+A+Swedish+Treebank+with+Phrase+Structure+and+Dependency+Annotation%2E+In+Proceedings+of+the+Fifth+International+Conference+on+Language+Reso)
13. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. [↗](https://doi.org/10.18653/v1/2020.acl-demos.14) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2020-qi-stanza-python-natural-language-processing/)
14. Natalia Silveira, Timothy Dozat, Marie-Catherine De Marneffe, Samuel R Bowman, Miriam Connor, John Bauer, and Christopher D Manning. 2014. A Gold Standard Dependency Corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14). [find](https://scholar.google.com/scholar?q=Natalia+Silveira%2C+Timothy+Dozat%2C+Marie%2DCatherine+De+Marneffe%2C+Samuel+R+Bowman%2C+Miriam+Connor%2C+John+Bauer%2C+and+Christopher+D+Manning%2E+2014%2E+A+Gold+Standard+Dependency+Corpus+for+English%2E+In+Proceedings)
15. Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. *Figure 5: Distribution of sentence lengths in AMMT. Most sentences are quite short. The mean sentence length is 6.97 words long. Compare to standard corpora derived from written sources like the English Web Treebank (15.33 words/sentence) long and the Penn Treebank (23.73 words/sentence in the test set).* *Table 3: The distribution of POS tags (left), and the most common dependencies (right). There is a long tail of dependencies.* 25050 14003 12825 12325 11274 9286 7830 6653 5693 4958 4306 3996 3912 3682 3335 3322 3134 Count nsubj advmod obj det case aux cop obl mark amod xcomp nmod:poss discourse cc compound conj vocative Dependencies 9829 12464 13688 18965 3746 12984 6275 25457 1835 7202 36370 8679 30301 2140 10 28139 6 Count ADJ ADP ADV AUX CCONJ DET INTJ NOUN NUM PART PRON PROPN PUNCT SCONJ SYM VERB X POS [find](https://scholar.google.com/scholar?q=Milan+Straka%2E+2018%2E+UDPipe+2%2E0+prototype+at+CoNLL+2018+UD+shared+task%2E+In+Proceedings+of+the+CoNLL+2018+Shared+Task%3A+Multilingual+Parsing+from+Raw+Text+to+Universal+Dependencies%2E+%2AFigure+5%3A+Distributi) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2018-straka-udpipe-2-0-prototype-conll/)