[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/stephenson-2022-bert-he-predict-contrastive-focus/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model

# BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model

**Authors:** Brooke Stephenson, Laurent Besacier, Laurent Girin, Thomas Hueber · **Year:** 2022 · **Venue:** Interspeech 2022 (arXiv:2207.01718) **Link:** [arxiv.org/abs/2207.01718](https://arxiv.org/abs/2207.01718) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**5** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/stephenson-2022-bert-he-predict-contrastive-focus/) of the 140673 papers whose bibliographies we readarXiv:2207.01718[licence](http://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://arxiv.org/abs/2207.01718)
- [PDF](https://arxiv.org/pdf/2207.01718)
- [arXiv 2207.01718](https://arxiv.org/abs/2207.01718)

**Authors (4)**

1. Brooke Stephenson
2. Laurent Besacier
3. Laurent Girin
4. Thomas Hueber

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

Personal pronouns are usually unstressed, so a stressed pronoun («HE loves her») signals contrastive focus that needs discourse knowledge to predict. From audiobooks of public-domain novels (five for training, ~41.6K utterances; one for testing, each read by three speakers) the authors derive three-level acoustic prominence labels and hand-check 493 contrastively stressed pronouns (393 stressed by at least two of three readers; rater kappa 0.85–0.90). A fine-tuned BERT predicting prominence per word is best overall but barely beats a per-word majority baseline, and on contrastive pronouns its recall is only about 0.24; giving it the previous one or two sentences helps little. A FastSpeech 2 model conditioned on the labels can then render the prominence, as a listening test confirms.

## Ideas

- I1. Mova reads text, but the tales mark emphasis in writing: italics rendered as `_he_`, small capitals or CAPITALS in Gutenberg texts. A pronoun marked this way is contrastively focused, and a contrastive pronoun takes the antecedent the unmarked one would not (the parallel-role preference reverses: «John hit Bill and then *he* hit him» → *he* = Bill). Add an «emphasised pronoun» flag in the frame reader's pronoun resolution (`prag/src/frames.rs`, `Ctx`) that skips the top salience candidate in favour of the next agreeing one, with the reason printed. First count emphasised pronouns in the tale corpus; if there are few, record the rule in the chapter only.
- I2. Keep the emphasis marks through tokenisation into a token flag rather than stripping them, so that focus-sensitive readings (contrast, *only*) can use them; check where the text pipeline currently drops underscores.

## For Mova

[29. Ellipsis, word order and information structure](https://md.mova.elamur.ai/free/en/docs/languages/en/29-ellipsis-word-order-information-structure/) (focus and contrast), `35-reference-and-coreference.md` §2 (salience), `05-writing-system.md`/`07-punctuation-capitals-numbers.md` (emphasis in writing). Code: `prag/src/frames.rs` (pronoun resolution), the `en` tokenizer. Measure: number of emphasised pronouns in the tales and frame bench coref on sentences that contain them.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [arXiv 2207.01718](https://arxiv.org/abs/2207.01718)

```
@misc{stephenson2022bert,
  title = {{BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model}},
  author = {Stephenson, Brooke and Besacier, Laurent and Girin, Laurent and Hueber, Thomas},
  year = {2022},
  eprint = {2207.01718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.01718}
}
```

- [← previous A preference semantics for imperatives](https://md.mova.elamur.ai/free/en/docs/papers/starr-2020-preference-semantics-imperatives/)
- [next → Intensives and Down-toners: a study in English adverbs (Heidelberg: Winter, 1901) + Studies in English, Written and Spoken (Zutphen, 1894)](https://md.mova.elamur.ai/free/en/docs/papers/stoffel-1894-intensives-down-toners-study/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

28 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Rooth, “A theory of focus interpretation,” Natural Language Semantics, vol. 1, no. 1, pp. 75–116, 1992. [find](https://scholar.google.com/scholar?q=M%2E+Rooth%2C+%E2%80%9CA+theory+of+focus+interpretation%2C%E2%80%9D+Natural+Language+Semantics%2C+vol%2E+1%2C+no%2E+1%2C+pp%2E+75%E2%80%93116%2C+1992%2E)
2. T. Kenter, M. Sharma, and R. Clark, “Improving the prosody of RNN-based english text-to-speech synthesis by incorporating a BERT model,” in Proc. of Interspeech, Shanghai, China (virtual conference), 2020. [find](https://scholar.google.com/scholar?q=T%2E+Kenter%2C+M%2E+Sharma%2C+and+R%2E+Clark%2C+%E2%80%9CImproving+the+prosody+of+RNN%2Dbased+english+text%2Dto%2Dspeech+synthesis+by+incorporating+a+BERT+model%2C%E2%80%9D+in+Proc%2E+of+Interspeech%2C+Shanghai%2C+China+%28virtual+conference%29%2C+)
3. Y. Zou, S. Liu, X. Yin, H. Lin, C. Wang, H. Zhang, and Z. Ma, “Fine-grained prosody modeling in neural speech synthesis using ToBI representation,” in Proc. of Interspeech, Brno, Czech Republik, 2021. [find](https://scholar.google.com/scholar?q=Y%2E+Zou%2C+S%2E+Liu%2C+X%2E+Yin%2C+H%2E+Lin%2C+C%2E+Wang%2C+H%2E+Zhang%2C+and+Z%2E+Ma%2C+%E2%80%9CFine%2Dgrained+prosody+modeling+in+neural+speech+synthesis+using+ToBI+representation%2C%E2%80%9D+in+Proc%2E+of+Interspeech%2C+Brno%2C+Czech+Republik%2C+2021%2E)
4. A. Talman, A. Suni, H. Celikkanat, S. Kakouros, J. Tiedemann, and M. Vainio, “Predicting prosodic prominence from text with pre-trained contextualized word representations,” in Proc. of Nordic Conference on Computational Linguistics (NoDaLiDa), Turku, Finland, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Talman%2C+A%2E+Suni%2C+H%2E+Celikkanat%2C+S%2E+Kakouros%2C+J%2E+Tiedemann%2C+and+M%2E+Vainio%2C+%E2%80%9CPredicting+prosodic+prominence+from+text+with+pre%2Dtrained+contextualized+word+representations%2C%E2%80%9D+in+Proc%2E+of+Nordic+Confere)
5. Y. Xiao, L. He, H. Ming, and F. K. Soong, “Improving prosody with linguistic and BERT derived features in multi-speaker based Mandarin Chinese neural TTS,” in Proc. of IEEE ICASSP, Barcelona, Spain (virtual conference), 2020. [find](https://scholar.google.com/scholar?q=Y%2E+Xiao%2C+L%2E+He%2C+H%2E+Ming%2C+and+F%2E+K%2E+Soong%2C+%E2%80%9CImproving+prosody+with+linguistic+and+BERT+derived+features+in+multi%2Dspeaker+based+Mandarin+Chinese+neural+TTS%2C%E2%80%9D+in+Proc%2E+of+IEEE+ICASSP%2C+Barcelona%2C+Spain+%28v)
6. Z. Hodari, A. Moinet, S. Karlapati, J. Lorenzo-Trueba, T. Merritt, A. Joly, A. Abbas, P. Karanasou, and T. Drugman, “CAMP: a two-stage approach to modelling prosody in context,” in Proc. of IEEE ICASSP, Toronto, Canada, 2021. [find](https://scholar.google.com/scholar?q=Z%2E+Hodari%2C+A%2E+Moinet%2C+S%2E+Karlapati%2C+J%2E+Lorenzo%2DTrueba%2C+T%2E+Merritt%2C+A%2E+Joly%2C+A%2E+Abbas%2C+P%2E+Karanasou%2C+and+T%2E+Drugman%2C+%E2%80%9CCAMP%3A+a+two%2Dstage+approach+to+modelling+prosody+in+context%2C%E2%80%9D+in+Proc%2E+of+IEEE+ICASS)
7. D. Stanton, Y. Wang, and R. Skerry-Ryan, “Predicting expressive speaking style from text in end-to-end speech synthesis,” in Proc. of IEEE Spoken Language Tech. Workshop (SLT), Athens, Greece, 2018. [find](https://scholar.google.com/scholar?q=D%2E+Stanton%2C+Y%2E+Wang%2C+and+R%2E+Skerry%2DRyan%2C+%E2%80%9CPredicting+expressive+speaking+style+from+text+in+end%2Dto%2Dend+speech+synthesis%2C%E2%80%9D+in+Proc%2E+of+IEEE+Spoken+Language+Tech%2E+Workshop+%28SLT%29%2C+Athens%2C+Greece%2C+2018%2E)
8. T. Hayashi, S. Watanabe, T. Toda, K. Takeda, S. Toshniwal, and K. Livescu, “Pre-trained text embeddings for enhanced text-to-speech synthesis,” in Proc. of Interspeech, Graz, Austria, 2019. [find](https://scholar.google.com/scholar?q=T%2E+Hayashi%2C+S%2E+Watanabe%2C+T%2E+Toda%2C+K%2E+Takeda%2C+S%2E+Toshniwal%2C+and+K%2E+Livescu%2C+%E2%80%9CPre%2Dtrained+text+embeddings+for+enhanced+text%2Dto%2Dspeech+synthesis%2C%E2%80%9D+in+Proc%2E+of+Interspeech%2C+Graz%2C+Austria%2C+2019%2E)
9. A. Nenkova, J. Brenier, A. Kothari, S. Calhoun, L. Whitton, D. Beaver, and D. Jurafsky, “To memorize or to predict: Prominence labeling in conversational speech,” in Proc. of the Conf. of the North Am. Chapter of the Assoc. for Computational Linguistics, Rochester, New York, 2007. [find](https://scholar.google.com/scholar?q=A%2E+Nenkova%2C+J%2E+Brenier%2C+A%2E+Kothari%2C+S%2E+Calhoun%2C+L%2E+Whitton%2C+D%2E+Beaver%2C+and+D%2E+Jurafsky%2C+%E2%80%9CTo+memorize+or+to+predict%3A+Prominence+labeling+in+conversational+speech%2C%E2%80%9D+in+Proc%2E+of+the+Conf%2E+of+the+North+Am)
10. J. Hirschberg, “Pitch accent in context: Predicting intonational prominence from text,” Artificial Intelligence, vol. 63, no. 1–2, pp. 305–340, 1993. [find](https://scholar.google.com/scholar?q=J%2E+Hirschberg%2C+%E2%80%9CPitch+accent+in+context%3A+Predicting+intonational+prominence+from+text%2C%E2%80%9D+Artificial+Intelligence%2C+vol%2E+63%2C+no%2E+1%E2%80%932%2C+pp%2E+305%E2%80%93340%2C+1993%2E)
11. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. of ACL, Florence, Italy, 2019. [find](https://scholar.google.com/scholar?q=J%2E+Devlin%2C+M%2E%2DW%2E+Chang%2C+K%2E+Lee%2C+and+K%2E+Toutanova%2C+%E2%80%9CBERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2C%E2%80%9D+in+Proc%2E+of+ACL%2C+Florence%2C+Italy%2C+2019%2E)
12. A. Suni, J. Simko, D. Aalto, and M. Vainio, “Hierarchical representation and estimation of prosody using continuous wavelet transform,” Computer Speech and Language, vol. 45, pp. 123– 136, 2017. [find](https://scholar.google.com/scholar?q=A%2E+Suni%2C+J%2E+Simko%2C+D%2E+Aalto%2C+and+M%2E+Vainio%2C+%E2%80%9CHierarchical+representation+and+estimation+of+prosody+using+continuous+wavelet+transform%2C%E2%80%9D+Computer+Speech+and+Language%2C+vol%2E+45%2C+pp%2E+123%E2%80%93+136%2C+2017%2E)
13. T. Raitio, R. Rasipuram, and D. Castellani, “Controllable neural text-to-speech synthesis using intuitive prosodic features,” in Proc. of Interspeech, Shanghai, China (virtual conference), 2020. [find](https://scholar.google.com/scholar?q=T%2E+Raitio%2C+R%2E+Rasipuram%2C+and+D%2E+Castellani%2C+%E2%80%9CControllable+neural+text%2Dto%2Dspeech+synthesis+using+intuitive+prosodic+features%2C%E2%80%9D+in+Proc%2E+of+Interspeech%2C+Shanghai%2C+China+%28virtual+conference%29%2C+2020%2E)
14. Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” in Int. Conf. on Machine Learning. PMLR, 2018. [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+D%2E+Stanton%2C+Y%2E+Zhang%2C+R%2E%2DS%2E+Ryan%2C+E%2E+Battenberg%2C+J%2E+Shor%2C+Y%2E+Xiao%2C+Y%2E+Jia%2C+F%2E+Ren%2C+and+R%2E+A%2E+Saurous%2C+%E2%80%9CStyle+tokens%3A+Unsupervised+style+modeling%2C+control+and+transfer+in+end%2Dto%2Dend+speech+syn)
15. L. Badino, J. S. Andersson, J. Yamagishi, and R. A. Clark, “Identification of contrast and its emphatic realization in HMM based speech synthesis,” in Proc. of Interspeech, Brighton, UK, 2009. [find](https://scholar.google.com/scholar?q=L%2E+Badino%2C+J%2E+S%2E+Andersson%2C+J%2E+Yamagishi%2C+and+R%2E+A%2E+Clark%2C+%E2%80%9CIdentification+of+contrast+and+its+emphatic+realization+in+HMM+based+speech+synthesis%2C%E2%80%9D+in+Proc%2E+of+Interspeech%2C+Brighton%2C+UK%2C+2009%2E)
16. A. Suni, S. Kakouros, M. Vainio, and J. Šimko, “Prosodic prominence and boundaries in sequence-to-sequence speech synthesis,” in Proc. of ISCA Int. Conf. on Speech Prosody, Tokyo, Japan, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Suni%2C+S%2E+Kakouros%2C+M%2E+Vainio%2C+and+J%2E+S%CC%8Cimko%2C+%E2%80%9CProsodic+prominence+and+boundaries+in+sequence%2Dto%2Dsequence+speech+synthesis%2C%E2%80%9D+in+Proc%2E+of+ISCA+Int%2E+Conf%2E+on+Speech+Prosody%2C+Tokyo%2C+Japan%2C+2020%2E)
17. S. Latif, I. Kim, I. Calapodescu, and L. Besacier, “Controlling prosody in end-to-end TTS: A case study on contrastive focus generation,” in Proc. of the Conf. on Computational Natural Language Learning (CoNLL), Punta Cana, Dominican Republic, 2021. [find](https://scholar.google.com/scholar?q=S%2E+Latif%2C+I%2E+Kim%2C+I%2E+Calapodescu%2C+and+L%2E+Besacier%2C+%E2%80%9CControlling+prosody+in+end%2Dto%2Dend+TTS%3A+A+case+study+on+contrastive+focus+generation%2C%E2%80%9D+in+Proc%2E+of+the+Conf%2E+on+Computational+Natural+Language+Learni)
18. J. Reeve, “Chapterize,” chapterize, 2016. [https://github.com/JonathanReeve/](https://github.com/JonathanReeve/) [↗](https://github.com/JonathanReeve/)
19. M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd, “spaCy: Industrial-strength Natural Language Processing in Python,” [https://spacy.io/](https://spacy.io/), 2020. [↗](https://spacy.io/)
20. M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal Forced Aligner: Trainable text-speech alignment using Kaldi,” in Proc. of Interspeech, Stockholm, Sweden, 2017. [find](https://scholar.google.com/scholar?q=M%2E+McAuliffe%2C+M%2E+Socolof%2C+S%2E+Mihuc%2C+M%2E+Wagner%2C+and+M%2E+Sonderegger%2C+%E2%80%9CMontreal+Forced+Aligner%3A+Trainable+text%2Dspeech+alignment+using+Kaldi%2C%E2%80%9D+in+Proc%2E+of+Interspeech%2C+Stockholm%2C+Sweden%2C+2017%2E)
21. J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960. [find](https://scholar.google.com/scholar?q=J%2E+Cohen%2C+%E2%80%9CA+coefficient+of+agreement+for+nominal+scales%2C%E2%80%9D+Educational+and+Psychological+Measurement%2C+vol%2E+20%2C+no%2E+1%2C+pp%2E+37%E2%80%9346%2C+1960%2E)
22. Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947. [find](https://scholar.google.com/scholar?q=Q%2E+McNemar%2C+%E2%80%9CNote+on+the+sampling+error+of+the+difference+between+correlated+proportions+or+percentages%2C%E2%80%9D+Psychometrika%2C+vol%2E+12%2C+no%2E+2%2C+pp%2E+153%E2%80%93157%2C+1947%2E)
23. Y. Ren, C. Hu, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fast-Speech 2: Fast and high-quality end-to-end text-to-speech,” arXiv preprint arXiv:2006.04558, 2020. [↗](https://arxiv.org/abs/2006.04558)
24. C. Wang, W.-N. Hsu, Y. Adi, A. Polyak, A. Lee, P.-J. Chen, J. Gu, and J. Pino, “FAIRSEQ S 2 : A scalable and integrable speech synthesis toolkit,” arXiv preprint arXiv:2109.06912, 2021. [↗](https://arxiv.org/abs/2109.06912)
25. R. Yamamoto, E. Song, and J.-M. Kim, “Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” in Proc. of ICASSP, Barcelona, Spain (virtual conference), 2020. [find](https://scholar.google.com/scholar?q=R%2E+Yamamoto%2C+E%2E+Song%2C+and+J%2E%2DM%2E+Kim%2C+%E2%80%9CParallel+WaveGAN%3A+A+fast+waveform+generation+model+based+on+generative+adversarial+networks+with+multi%2Dresolution+spectrogram%2C%E2%80%9D+in+Proc%2E+of+ICASSP%2C+Barcelona%2C+Spa)
26. B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” in Proc. of Interspeech, Shanghai, China (virtual conference), 2020. [find](https://scholar.google.com/scholar?q=B%2E+Desplanques%2C+J%2E+Thienpondt%2C+and+K%2E+Demuynck%2C+%E2%80%9CECAPA%2DTDNN%3A+Emphasized+channel+attention%2C+propagation+and+aggregation+in+TDNN+based+speaker+verification%2C%E2%80%9D+in+Proc%2E+of+Interspeech%2C+Shanghai%2C+China+%28vi)
27. M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624. [↗](https://arxiv.org/abs/2106.04624)
28. N. Jillings, D. Moffat, B. De Man, and J. D. Reiss, “Web Audio Evaluation Tool: A browser-based listening test environment,” in Proc. of the Sound and Music Computing Conf., Maynooth, Ireland, 2015. [find](https://scholar.google.com/scholar?q=N%2E+Jillings%2C+D%2E+Moffat%2C+B%2E+De+Man%2C+and+J%2E+D%2E+Reiss%2C+%E2%80%9CWeb+Audio+Evaluation+Tool%3A+A+browser%2Dbased+listening+test+environment%2C%E2%80%9D+in+Proc%2E+of+the+Sound+and+Music+Computing+Conf%2E%2C+Maynooth%2C+Ireland%2C+2015%2E)