[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/garcia-2025-assessing-agreement-competence-large-language/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / Assessing the Agreement Competence of Large Language Models

# Assessing the Agreement Competence of Large Language Models

**Authors:** Alba Táboas García, [Leo Wanner](https://md.mova.elamur.ai/free/en/docs/papers/authors/leo-wanner/) · **Year:** 2025 · **Venue:** Proceedings of the Eighth International Conference on Dependency Linguistics (Depling, SyntaxFest 2025) **Link:** [aclanthology.org/2025.depling-1.4](https://aclanthology.org/2025.depling-1.4/) (ACL Anthology 2025.depling-1.4) **License of the paper:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) — [creativecommons.org/licenses/by/4.0](https://creativecommons.org/licenses/by/4.0/)

[**0** citations](https://openalex.org/works?filter=cites:W7112808468) openalex-snapshot · 2026-10-14[Proceedings of the Eighth International Conference on Dependency Linguistics (Depling, SyntaxFest 2025)](https://aclanthology.org/volumes/2025.depling-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2025.depling-1.4/)
- [PDF](https://aclanthology.org/2025.depling-1.4.pdf)
- [OpenAlex](https://openalex.org/W7112808468)

**Authors (2)**

1. Alba Táboas García
2. Leo Wanner

Citations and author metrics from openalex-snapshot (2026-10-14); authors with the same name are not always told apart.

## Summary

Agreement is a common probe for whether a language model has learned hierarchical sentence structure rather than surface word order. Most such probes have targeted English, where agreement is essentially subject-verb number. This paper extends the evaluation to four morphologically richer languages: Italian, Portuguese, Spanish and Russian. For each language, fluent linguists hand-built ten test suites covering agreement inside noun phrases and within the clause, in gender, number and person, plus case for Russian. Each item pairs a grammatical sentence with one or more deliberately corrupted variants, and the harder items put distance and an "attractor" with different features between controller and target. A model passes an item if it prefers the grammatical form. Twenty-five monolingual and multilingual models were tested on more than 5,000 handmade items. Models do reasonably well overall but fail on the complex constructions with attractors. Monolingual models beat multilingual ones, and model size or training data volume matter little.

## How Mova uses it

- [Attraction through an of-phrase: "A pattern of arrests indicate"](https://md.mova.elamur.ai/free/en/docs/en/seeds/errors/attraction-of-phrase/): cited as evidence that attractor constructions are the hardest for LLMs. This motivates the warning rules `en.errors.attraction-of` and `en.errors.attraction-of-auxcop`, which flag a singular subject with a plural of-phrase followed by a plural (VBP) verb ("A pattern of arrests indicate"), with an exception list of quantity nouns (lot, number, majority...).
- [The relative-clause verb agrees with the antecedent](https://md.mova.elamur.ai/free/en/docs/en/seeds/errors/relative-clause-agreement/): cited for relative clauses with attractors being the hardest case. It backs `en.errors.relcl-plural-antecedent` and `en.errors.relcl-singular-antecedent`, which compare the number of the antecedent noun with the tag of the relative-clause verb. A mismatch is a text error or a wrong attachment of the relative clause.
- In Mova the paper's evaluation idea is reused in reverse: instead of probing a model, these rules check annotations (from Mova's own tagger or an LLM annotator) for exactly the construction where agreement breaks down.

## Effectiveness in Mova

Not measured separately. The paper is one supporting source, alongside Fowler and BLiMP, for four rules in the English error-detection rule set. Gold-data checks reported in the seeds: attraction rule on EWT 2.18, 7 checks / 1 hit for verbs (exactly "pattern ... indicate") and 13/5 for copula or auxiliary; GUM 16/3 and 27/8. Relative-clause rules on EWT 2.18: plural antecedent + VBZ 338/1, singular antecedent + VBP 601/17.

---

👨‍🔬💥

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{garcia2025assessing,
  title = {{Assessing the Agreement Competence of Large Language Models}},
  author = {García, Alba Táboas and Wanner, Leo},
  booktitle = {Proceedings of the Eighth International Conference on Dependency Linguistics (Depling, SyntaxFest 2025)},
  year = {2025},
  url = {https://aclanthology.org/2025.depling-1.4/}
}
```

- [← previous A Dictionary of Modern English Usage](https://md.mova.elamur.ai/free/en/docs/papers/fowler-1926-dictionary-modern-english-usage/)
- [next → Evaluating Models' Local Decision Boundaries via Contrast Sets](https://md.mova.elamur.ai/free/en/docs/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
- [↑ back to papers](https://md.mova.elamur.ai/free/en/docs/papers/)

## References

36 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amine Abdaoui, Camille Pradel, and Grégoire Sigel. 2020. Load what you need: Smaller versions of mutililingual BERT. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 119–123, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Amine+Abdaoui%2C+Camille+Pradel%2C+and+Gr%C3%A9goire+Sigel%2E+2020%2E+Load+what+you+need%3A+Smaller+versions+of+mutililingual+BERT%2E+In+Proceedings+of+SustaiNLP%3A+Workshop+on+Simple+and+Efficient+Natural+Language+Proc)
2. José Canete, Gabriel Chaperon, Rodrigo Fuentes, and Jorge Pérez. 2020. Spanish pre-trained BERT model and evaluation data. PML4DC at ICLR, 2020. [find](https://scholar.google.com/scholar?q=Jos%C3%A9+Canete%2C+Gabriel+Chaperon%2C+Rodrigo+Fuentes%2C+and+Jorge+P%C3%A9rez%2E+2020%2E+Spanish+pre%2Dtrained+BERT+model+and+evaluation+data%2E+PML4DC+at+ICLR%2C+2020%2E)
3. José Cañete, Sebastian Donoso, Felipe Bravo-Marquez, Andrés Carvallo, and Vladimir Araujo. 2022. AL-BETO and DistilBETO: Lightweight Spanish language models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4291–4298, Marseille, France. European Language Resources… [find](https://scholar.google.com/scholar?q=Jos%C3%A9+Ca%C3%B1ete%2C+Sebastian+Donoso%2C+Felipe+Bravo%2DMarquez%2C+Andr%C3%A9s+Carvallo%2C+and+Vladimir+Araujo%2E+2022%2E+AL%2DBETO+and+DistilBETO%3A+Lightweight+Spanish+language+models%2E+In+Proceedings+of+the+Thirteenth+Language+)
4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
5. Greville G. Corbett. 2006. Agreement. Cambridge University Press, Cambridge, UK. [find](https://scholar.google.com/scholar?q=Greville+G%2E+Corbett%2E+2006%2E+Agreement%2E+Cambridge+University+Press%2C+Cambridge%2C+UK%2E)
6. Nicholas Kluge Corrêa, Aniket Sen, Sophia Falk, and Shiza Fatimah. 2024. Tucano: Advancing neural text generation for portuguese. Preprint, arXiv:2411.07854. — [arXiv:2411.07854](https://arxiv.org/abs/2411.07854) [find](https://scholar.google.com/scholar?q=Nicholas+Kluge+Corr%C3%AAa%2C+Aniket+Sen%2C+Sophia+Falk%2C+and+Shiza+Fatimah%2E+2024%2E+Tucano%3A+Advancing+neural+text+generation+for+portuguese%2E+Preprint%2C+arXiv%3A2411%2E07854%2E+%E2%80%94+%5BarXiv%3A2411%2E07854%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
7. Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Marc Pàmies, Joan Llop-Palao, Joaquín Silveira-Ocampo, Casimiro Pio Carrino, Carme Armentano Oller, Carlos Rodríguez Penagos, Aitor Gonzalez-Agirre, and Marta Villegas Montserrat. 2022. Maria: Spanish language models. Procesamiento del Lenguaje Natural… [find](https://scholar.google.com/scholar?q=Asier+Guti%C3%A9rrez%2DFandi%C3%B1o%2C+Jordi+Armengol%2DEstap%C3%A9%2C+Marc+P%C3%A0mies%2C+Joan+Llop%2DPalao%2C+Joaqu%C3%ADn+Silveira%2DOcampo%2C+Casimiro+Pio+Carrino%2C+Carme+Armentano+Oller%2C+Carlos+Rodr%C3%ADguez+Penagos%2C+Aitor+Gonzalez%2DAgirre%2C+and)
8. Iria de Dios-Flores and Marcos Garcia. 2022. A computational psycholinguistic evaluation of the syntactic abilities of galician bert models at the interface of dependency resolution and training time. Procesamiento del lenguaje natural, 69:15–26. [find](https://scholar.google.com/scholar?q=Iria+de+Dios%2DFlores+and+Marcos+Garcia%2E+2022%2E+A+computational+psycholinguistic+evaluation+of+the+syntactic+abilities+of+galician+bert+models+at+the+interface+of+dependency+resolution+and+training+time%2E)
9. John Hale. 2001. A probabilistic earley parser as a psycholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies. [find](https://scholar.google.com/scholar?q=John+Hale%2E+2001%2E+A+probabilistic+earley+parser+as+a+psycholinguistic+model%2E+In+Proceedings+of+the+Second+Meeting+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+on+Langu)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
11. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Jon+Gauthier%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2020%2E+A+systematic+assessment+of+syntactic+generalization+in+neural+language+models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+th)
12. Richard Futrell, Ethan Wilcox, Takashi Morita, and Roger Levy. 2018. RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency. arXiv preprint arXiv:1809.01329. [find](https://scholar.google.com/scholar?q=Richard+Futrell%2C+Ethan+Wilcox%2C+Takashi+Morita%2C+and+Roger+Levy%2E+2018%2E+RNNs+as+psycholinguistic+subjects%3A+Syntactic+state+and+grammatical+dependency%2E+arXiv+preprint+arXiv%3A1809%2E01329%2E)
13. Carina Kauf and Anna Ivanova. 2023. A better way to do masked language model scoring. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 925–935, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Carina+Kauf+and+Anna+Ivanova%2E+2023%2E+A+better+way+to+do+masked+language+model+scoring%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short+Papers)
14. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Richard+Futrell%2C+Ethan+Wilcox%2C+Takashi+Morita%2C+Peng+Qian%2C+Miguel+Ballesteros%2C+and+Roger+Levy%2E+2019%2E+Neural+language+models+as+psycholinguistic+subjects%3A+Representations+of+syntactic+state%2E+In+Proceedi)
15. Daria Kryvosheieva and Roger Levy. 2025. Controlled evaluation of syntactic knowledge in multilingual language models. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 402–413, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Daria+Kryvosheieva+and+Roger+Levy%2E+2025%2E+Controlled+evaluation+of+syntactic+knowledge+in+multilingual+language+models%2E+In+Proceedings+of+the+First+Workshop+on+Language+Models+for+Low%2DResource+Language)
16. Yuri Kuratov and Mikhail Arkhipov. 2019. Adaptation of deep bidirectional multilingual transformers for russian language. Preprint, arXiv:1905.07213. [find](https://scholar.google.com/scholar?q=Yuri+Kuratov+and+Mikhail+Arkhipov%2E+2019%2E+Adaptation+of+deep+bidirectional+multilingual+transformers+for+russian+language%2E+Preprint%2C+arXiv%3A1905%2E07213%2E)
17. Marcos Garcia and Alfredo Crespo-Otero. 2022. A targeted assessment of the syntactic abilities of transformer models for galician-portuguese. In Computational Processing of the Portuguese Language, pages 46–56, Cham. Springer International Publishing. [find](https://scholar.google.com/scholar?q=Marcos+Garcia+and+Alfredo+Crespo%2DOtero%2E+2022%2E+A+targeted+assessment+of+the+syntactic+abilities+of+transformer+models+for+galician%2Dportuguese%2E+In+Computational+Processing+of+the+Portuguese+Language%2C+pa)
18. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… [find](https://scholar.google.com/scholar?q=Xi+Victoria+Lin%2C+Todor+Mihaylov%2C+Mikel+Artetxe%2C+Tianlu+Wang%2C+Shuohui+Chen%2C+Daniel+Simig%2C+Myle+Ott%2C+Naman+Goyal%2C+Shruti+Bhosale%2C+Jingfei+Du%2C+Ramakanth+Pasunuru%2C+Sam+Shleifer%2C+Punit+Singh+Koura%2C+Vishrav)
19. Yoav Goldberg. 2019. Assessing BERT’s syntactic abilities. arXiv preprint arXiv:1901.05287. [find](https://scholar.google.com/scholar?q=Yoav+Goldberg%2E+2019%2E+Assessing+BERT%E2%80%99s+syntactic+abilities%2E+arXiv+preprint+arXiv%3A1901%2E05287%2E)
20. Aitor Gonzalez-Agirre, Marc Pàmies, Joan Llop, Irene Baucells, Severino Da Dalt, Daniel Tamayo, José Javier Saiz, Ferran Espuña, Jaume Prats, Javier Aula-Blasco, Mario Mina, Iñigo Pikabea, Adrián Rubio, Alexander Shvets, Anna Sallés, Iñaki La-cunza, Jorge Palomar, Júlia Falcão, Lucía Tormo, Luis Vas… — [arXiv:2502.08489](https://arxiv.org/abs/2502.08489) [find](https://scholar.google.com/scholar?q=Aitor+Gonzalez%2DAgirre%2C+Marc+P%C3%A0mies%2C+Joan+Llop%2C+Irene+Baucells%2C+Severino+Da+Dalt%2C+Daniel+Tamayo%2C+Jos%C3%A9+Javier+Saiz%2C+Ferran+Espu%C3%B1a%2C+Jaume+Prats%2C+Javier+Aula%2DBlasco%2C+Mario+Mina%2C+I%C3%B1igo+Pikabea%2C+Adri%C3%A1n+Rubi)
21. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. [find](https://scholar.google.com/scholar?q=Tal+Linzen%2C+Emmanuel+Dupoux%2C+and+Yoav+Goldberg%2E+2016%2E+Assessing+the+ability+of+LSTMs+to+learn+syntax%2Dsensitive+dependencies%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+4%3A521%E2%80%93535%2E)
22. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… [find](https://scholar.google.com/scholar?q=Kristina+Gulordava%2C+Piotr+Bojanowski%2C+Edouard+Grave%2C+Tal+Linzen%2C+and+Marco+Baroni%2E+2018%2E+Colorless+green+recurrent+networks+dream+hierarchically%2E+In+Proceedings+of+the+2018+Conference+of+the+North+Ame)
23. Lorenzo De Mattei, Michele Cafagna, Felice Dell’Orletta, Malvina Nissim, and Marco Guerini. 2020. Geppetto carves italian into a language model. In Proceedings of the Seventh Italian Conference on Computational Linguistics, CLiC-it 2020, Bologna, Italy, March 1-3, 2021, volume 2769. CEUR-WS.org. Ita… [find](https://scholar.google.com/scholar?q=Lorenzo+De+Mattei%2C+Michele+Cafagna%2C+Felice+Dell%E2%80%99Orletta%2C+Malvina+Nissim%2C+and+Marco+Guerini%2E+2020%2E+Geppetto+carves+italian+into+a+language+model%2E+In+Proceedings+of+the+Seventh+Italian+Conference+on+Com)
24. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020. Bertimbau: Pretrained bert models for brazilian portuguese. In Intelligent Systems: 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20–23, 2020, Proceedings, Part I, page 403–417, Berlin, Heidelberg. Springer-Verlag. [find](https://scholar.google.com/scholar?q=F%C3%A1bio+Souza%2C+Rodrigo+Nogueira%2C+and+Roberto+Lotufo%2E+2020%2E+Bertimbau%3A+Pretrained+bert+models+for+brazilian+portuguese%2E+In+Intelligent+Systems%3A+9th+Brazilian+Conference%2C+BRACIS+2020%2C+Rio+Grande%2C+Brazil%2C+)
25. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler–gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, Brussels, Belgium. Association for Computati… [find](https://scholar.google.com/scholar?q=Ethan+Wilcox%2C+Roger+Levy%2C+Takashi+Morita%2C+and+Richard+Futrell%2E+2018%2E+What+do+RNN+language+models+learn+about+filler%E2%80%93gap+dependencies%3F+In+Proceedings+of+the+2018+EMNLP+Workshop+BlackboxNLP%3A+Analyzing+a)
26. Igor Mel’čuk. 1993. Agreement, governance and congruence. Linguisticae investigationes, 17(2):307– 373. [find](https://scholar.google.com/scholar?q=Igor+Mel%E2%80%99c%CC%8Cuk%2E+1993%2E+Agreement%2C+governance+and+congruence%2E+Linguisticae+investigationes%2C+17%282%29%3A307%E2%80%93+373%2E)
27. Igor Mel’čuk. 2009. Dependency in natural language. In Dependency in Linguistic Description, pages 1– 110. John Benjamins Publishing Company. [find](https://scholar.google.com/scholar?q=Igor+Mel%E2%80%99c%CC%8Cuk%2E+2009%2E+Dependency+in+natural+language%2E+In+Dependency+in+Linguistic+Description%2C+pages+1%E2%80%93+110%2E+John+Benjamins+Publishing+Company%2E)
28. Ethan Wilcox, Peng Qian, Richard Futrell, Miguel Ballesteros, and Roger Levy. 2019. Structural supervision improves learning of non-local grammatical dependencies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec… [find](https://scholar.google.com/scholar?q=Ethan+Wilcox%2C+Peng+Qian%2C+Richard+Futrell%2C+Miguel+Ballesteros%2C+and+Roger+Levy%2E+2019%2E+Structural+supervision+improves+learning+of+non%2Dlocal+grammatical+dependencies%2E+In+Proceedings+of+the+2019+Conferenc)
29. Kanishka Misra. 2022. minicons: Enabling flexible behavioral and representational analyses of transformer language models. arXiv preprint arXiv:2203.13112. [find](https://scholar.google.com/scholar?q=Kanishka+Misra%2E+2022%2E+minicons%3A+Enabling+flexible+behavioral+and+representational+analyses+of+transformer+language+models%2E+arXiv+preprint+arXiv%3A2203%2E13112%2E)
30. Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. 2020. Cross-linguistic syntactic evaluation of word prediction models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5523–5539, Online. Association for Comput… [find](https://scholar.google.com/scholar?q=Aaron+Mueller%2C+Garrett+Nicolai%2C+Panayiota+Petrou%2DZeniou%2C+Natalia+Talmina%2C+and+Tal+Linzen%2E+2020%2E+Cross%2Dlinguistic+syntactic+evaluation+of+word+prediction+models%2E+In+Proceedings+of+the+58th+Annual+Meeti)
31. BigScience Workshop. 2023. Bloom: A 176b-parameter open-access multilingual language model. Preprint, arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=BigScience+Workshop%2E+2023%2E+Bloom%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+model%2E+Preprint%2C+arXiv%3A2211%2E05100%2E)
32. Karolina Zaczynska, Nils Feldhus, Robert Schwarzen-berg, Aleksandra Gabryszak, and Sebastian Möller. 2020. Evaluating german transformer language models with syntactic agreement tests. CoRR, abs/2007.03765. [find](https://scholar.google.com/scholar?q=Karolina+Zaczynska%2C+Nils+Feldhus%2C+Robert+Schwarzen%2Dberg%2C+Aleksandra+Gabryszak%2C+and+Sebastian+M%C3%B6ller%2E+2020%2E+Evaluating+german+transformer+language+models+with+syntactic+agreement+tests%2E+CoRR%2C+abs%2F2007%2E)
33. Vivi Nastase, Giuseppe Samo, Chunyang Jiang, and Paola Merlo. 2024. Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement. In Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), pages 631–643, Pisa, Italy. CEUR Workshop… [find](https://scholar.google.com/scholar?q=Vivi+Nastase%2C+Giuseppe+Samo%2C+Chunyang+Jiang%2C+and+Paola+Merlo%2E+2024%2E+Exploring+syntactic+information+in+sentence+embeddings+through+multilingual+subject%2Dverb+agreement%2E+In+Proceedings+of+the+10th+Itali)
34. Dmitry Zmitrovich, Aleksandr Abramov, Andrey Kalmykov, Vitaly Kadulin, Maria Tikhonova, Ekaterina Taktasheva, Danil Astafurov, Mark Baushenko, Artem Snegirev, Tatiana Shavrina, Sergei S. Markov, Vladislav Mikhailov, and Alena Fenogenova. 2024. A family of pretrained transformer language models for R… [find](https://scholar.google.com/scholar?q=Dmitry+Zmitrovich%2C+Aleksandr+Abramov%2C+Andrey+Kalmykov%2C+Vitaly+Kadulin%2C+Maria+Tikhonova%2C+Ekaterina+Taktasheva%2C+Danil+Astafurov%2C+Mark+Baushenko%2C+Artem+Snegirev%2C+Tatiana+Shavrina%2C+Sergei+S%2E+Markov%2C+Vladi)
35. Laura Pérez-Mayos, Alba Táboas García, Simon Mille, and Leo Wanner. 2021. Assessing the syntactic capabilities of transformer-based multilingual language models. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3799–3812, Online. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Laura+P%C3%A9rez%2DMayos%2C+Alba+T%C3%A1boas+Garc%C3%ADa%2C+Simon+Mille%2C+and+Leo+Wanner%2E+2021%2E+Assessing+the+syntactic+capabilities+of+transformer%2Dbased+multilingual+language+models%2E+In+Findings+of+the+Association+for+Com)
36. Tiago Pimentel and Clara Meister. 2024. How to compute the probability of a word. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358–18375, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tiago+Pimentel+and+Clara+Meister%2E+2024%2E+How+to+compute+the+probability+of+a+word%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+18358%E2%80%9318375%2C+Miami%2C+F)