[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/ziems-2022-value-understanding-dialect-disparity-nlu/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / VALUE: Understanding Dialect Disparity in NLU

# VALUE: Understanding Dialect Disparity in NLU

**Authors:** [Caleb Ziems](https://md.mova.elamur.ai/free/en/docs/papers/authors/caleb-ziems/), Jiaao Chen, Camille Harris, Jessica Anderson, [Diyi Yang](https://md.mova.elamur.ai/free/en/docs/papers/authors/diyi-yang/) · **Year:** 2022 · **Venue:** Proceedings of the 60th Annual Meeting of the ACL (Volume 1: Long Papers), pp. 3701–3720 **Link:** [aclanthology.org/2022.acl-long.258](https://aclanthology.org/2022.acl-long.258/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**19** citations](https://openalex.org/works?filter=cites:W4226263264) openalex · 2026-10-18[**47** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/ziems-2022-value-understanding-dialect-disparity-nlu/) of the 140673 papers whose bibliographies we read[Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://aclanthology.org/volumes/2022.acl-long/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2022.acl-long.258/)
- [PDF](https://aclanthology.org/2022.acl-long.258.pdf)
- [DOI 10.18653/v1/2022.acl-long.258](https://doi.org/10.18653/v1/2022.acl-long.258)
- [OpenAlex](https://openalex.org/W4226263264)

**Authors (5)**

1. [Caleb Ziems](https://openalex.org/A5002621773) [ORCID](https://orcid.org/0000-0002-2792-6298) · h 13 · 1276 citations · 55 works
2. [Jiaao Chen](https://openalex.org/A5069870254) h 2 · 585 citations · 4 works
3. [Camille Harris](https://openalex.org/A5065590661) [ORCID](https://orcid.org/0009-0000-9227-1470) · h 5 · 143 citations · 14 works
4. [Jessica Anderson](https://openalex.org/A5101587247) [ORCID](https://orcid.org/0000-0002-1346-6059) · h 13 · 562 citations · 89 works
5. [Diyi Yang](https://openalex.org/A5089413311) [ORCID](https://orcid.org/0000-0003-1220-3983) · h 49 · 15219 citations · 489 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The paper builds VALUE, a version of the GLUE benchmark transformed into synthetic African American Vernacular English (AAVE). Eleven morphosyntactic and lexical rules (copula deletion, negative concord, negative inversion, completive "done", remote "been", existential "dey/it", "gonna/finna", "got", inflection, null genitives, relativiser deletion) are applied by POS tags and dependency relations, and a learned SAE-to-AAVE word list is added. Native AAVE speakers judged 2,556 transformed pairs with three raters each; majority-vote accuracy is 91% to 98% per rule and unanimous accuracy 58% to 85%, with weak agreement (Krippendorff's alpha 0.26). RoBERTa trained on SAE and tested on the AAVE set is about 1.5 points lower on SST-2 and 0.9 lower on QNLI than a model trained on SAE and AAVE together. Perturbation analysis on MNLI finds the highest error rate for negative inversion (9.09% of the 11 tested instances flipped).

## Ideas

- I1. Per-rule flip rate: for each transformation, count how often a correct answer of the school student flips to a wrong one, and report the rate with its sample size, as the paper does in its Table 6. A rule with n = 11 needs a wide interval, and the report should say so.
- I2. Keep the rules as data with their conditions (dependency label, POS, lemma list) in one table. The paper's rule for negative concord requires an indefinite object; Mova can express the same condition as a check on the tree, so a reviewer can read why a rule did or did not fire.
- I3. Reject a rule that fails its check. The paper keeps a rule only when its validation accuracy passes; Mova can use the same gate, with a threshold written in the repo before the measurement.

## For Mova

Chapters: [27. Negation](https://md.mova.elamur.ai/free/en/docs/languages/en/27-negation/) (negative concord and inversion), [13. Verbs](https://md.mova.elamur.ai/free/en/docs/languages/en/13-verbs/) (inflection, completive and remote markers), [25. Relative clauses](https://md.mova.elamur.ai/free/en/docs/languages/en/25-relative-clauses/) (relativiser deletion), [37. Register and variation](https://md.mova.elamur.ai/free/en/docs/languages/en/37-register-and-variation/). Code: `en/src/morph.rs` for the rules, `en/src/lin.rs` for word order after a rule. Measures: per-rule flip rate with sample size on `world` school tests; the SAE accuracy before and after each rule; the threshold gate written in advance. Hold out a gold set and use it once per release.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2022.acl-long.258](https://doi.org/10.18653/v1/2022.acl-long.258)

```
@inproceedings{ziems2022value,
  title = {{VALUE: Understanding Dialect Disparity in NLU}},
  author = {Ziems, Caleb and Chen, Jiaao and Harris, Camille and Anderson, Jessica and Yang, Diyi},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.258},
  url = {https://doi.org/10.18653/v1/2022.acl-long.258}
}
```

- [← previous Anatomy of OntoGUM—Adapting GUM to the OntoNotes Scheme to Evaluate Robustness of SOTA Coreference Algorithms](https://md.mova.elamur.ai/free/en/docs/papers/zhu-2021-anatomy-ontogum-adapting-gum-ontonotes/)
- [next → Multi-VALUE: A Framework for Cross-Dialectal English NLP](https://md.mova.elamur.ai/free/en/docs/papers/ziems-2023-multi-value-framework-cross-dialectal/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

73 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Alvarez-Melis and Tommi Jaakkola. 2017. A causal framework for explaining the predictions of black-box sequence-to-sequence models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 412– 421, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Alvarez%2DMelis+and+Tommi+Jaakkola%2E+2017%2E+A+causal+framework+for+explaining+the+predictions+of+black%2Dbox+sequence%2Dto%2Dsequence+models%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+)
2. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Moustafa+Alzantot%2C+Yash+Sharma%2C+Ahmed+Elgohary%2C+Bo%2DJhang+Ho%2C+Mani+Srivastava%2C+and+Kai%2DWei+Chang%2E+2018%2E+Generating+natural+language+adversarial+examples%2E+In+Proceedings+of+the+2018+Conference+on+Empiri)
3. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+Yonatan+Bisk%2E+2018%2E+Synthetic+and+natural+noise+both+break+neural+machine+translation%2E+In+6th+International+Conference+on+Learning+Representations%2C+ICLR+2018%2C+Vancouver%2C+BC%2C+Canad)
4. Emily M Bender. 2000. Syntactic variation and linguistic competence: The case of AAVE copula absence. stanford university Stanford, California. [find](https://scholar.google.com/scholar?q=Emily+M+Bender%2E+2000%2E+Syntactic+variation+and+linguistic+competence%3A+The+case+of+AAVE+copula+absence%2E+stanford+university+Stanford%2C+California%2E)
5. Rakesh M Bhatt. 2008. Indian english: Syntax. In A handbook of varieties of English, pages 2208–2222. De Gruyter Mouton. [find](https://scholar.google.com/scholar?q=Rakesh+M+Bhatt%2E+2008%2E+Indian+english%3A+Syntax%2E+In+A+handbook+of+varieties+of+English%2C+pages+2208%E2%80%932222%2E+De+Gruyter+Mouton%2E)
6. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Su+Lin+Blodgett%2C+Lisa+Green%2C+and+Brendan+O%E2%80%99Connor%2E+2016%2E+Demographic+dialectal+variation+in+social+media%3A+A+case+study+of+African%2DAmerican+English%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+M)
7. Su Lin Blodgett and Brendan O’Connor. 2017. Racial disparity in natural language processing: A case study of social media african-american english. CoRR, abs/1707.00061. [↗](https://arxiv.org/abs/1707.00061)
8. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2018. Twitter Universal Dependency parsing for African-American and mainstream American English. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Su+Lin+Blodgett%2C+Johnny+Wei%2C+and+Brendan+O%E2%80%99Connor%2E+2018%2E+Twitter+Universal+Dependency+parsing+for+African%2DAmerican+and+mainstream+American+English%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Ass) [our card](https://md.mova.elamur.ai/free/en/docs/papers/ud-2018-blodgett-twitter-universal-dependency-parsing-african/)
9. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4349–4357. [find](https://scholar.google.com/scholar?q=Tolga+Bolukbasi%2C+Kai%2DWei+Chang%2C+James+Y%2E+Zou%2C+Venkatesh+Saligrama%2C+and+Adam+Tauman+Kalai%2E+2016%2E+Man+is+to+computer+programmer+as+woman+is+to+homemaker%3F+debiasing+word+embeddings%2E+In+Advances+in+Neural)
10. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
11. Mary Bucholtz and Kira Hall. 2005. Identity and interaction: A sociocultural linguistic approach. Discourse studies, 7(4-5):585–614. [find](https://scholar.google.com/scholar?q=Mary+Bucholtz+and+Kira+Hall%2E+2005%2E+Identity+and+interaction%3A+A+sociocultural+linguistic+approach%2E+Discourse+studies%2C+7%284%2D5%29%3A585%E2%80%93614%2E)
12. Chris Collins, Simanique Moody, and Paul M Postal. 2008. An aae camouflage construction. Language, pages 29–68. [find](https://scholar.google.com/scholar?q=Chris+Collins%2C+Simanique+Moody%2C+and+Paul+M+Postal%2E+2008%2E+An+aae+camouflage+construction%2E+Language%2C+pages+29%E2%80%9368%2E)
13. Nikolas Coupland. 2007. Style: Language variation and identity. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Nikolas+Coupland%2E+2007%2E+Style%3A+Language+variation+and+identity%2E+Cambridge+University+Press%2E)
14. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. ArXiv preprint, abs/1703.04009. [↗](https://arxiv.org/abs/1703.04009)
15. Dorottya Demszky, Devyani Sharma, Jonathan Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. 2021a. Learning to recognize dialect features. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2315–2338, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dorottya+Demszky%2C+Devyani+Sharma%2C+Jonathan+Clark%2C+Vinodkumar+Prabhakaran%2C+and+Jacob+Eisenstein%2E+2021a%2E+Learning+to+recognize+dialect+features%2E+In+Proceedings+of+the+2021+Conference+of+the+North+Americ)
16. Dorottya Demszky, Devyani Sharma, Jonathan Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. 2021b. Learning to recognize dialect features. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2315–2338, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dorottya+Demszky%2C+Devyani+Sharma%2C+Jonathan+Clark%2C+Vinodkumar+Prabhakaran%2C+and+Jacob+Eisenstein%2E+2021b%2E+Learning+to+recognize+dialect+features%2E+In+Proceedings+of+the+2021+Conference+of+the+North+Americ)
17. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). [find](https://scholar.google.com/scholar?q=William+B%2E+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphrasing+%28IWP2005%29%2E)
18. Rachel Dorn. 2019. Dialect-specific models for automatic speech recognition of African American Vernacular English. In Proceedings of the Student Research Workshop Associated with RANLP 2019, pages 16–20, Varna, Bulgaria. INCOMA Ltd. [find](https://scholar.google.com/scholar?q=Rachel+Dorn%2E+2019%2E+Dialect%2Dspecific+models+for+automatic+speech+recognition+of+African+American+Vernacular+English%2E+In+Proceedings+of+the+Student+Research+Workshop+Associated+with+RANLP+2019%2C+pages+16)
19. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31–36, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Javid+Ebrahimi%2C+Anyi+Rao%2C+Daniel+Lowd%2C+and+Dejing+Dou%2E+2018%2E+HotFlip%3A+White%2Dbox+adversarial+examples+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computati)
20. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1307–1323, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Yoav+Artzi%2C+Victoria+Basmov%2C+Jonathan+Berant%2C+Ben+Bogin%2C+Sihao+Chen%2C+Pradeep+Dasigi%2C+Dheeru+Dua%2C+Yanai+Elazar%2C+Ananth+Gottumukkala%2C+Nitish+Gupta%2C+Hannaneh+Hajishirzi%2C+Gabriel+Ilharco%2C+Da) [our card](https://md.mova.elamur.ai/free/en/docs/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
21. Lisa J Green. 2002. African American English: a linguistic introduction. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Lisa+J+Green%2E+2002%2E+African+American+English%3A+a+linguistic+introduction%2E+Cambridge+University+Press%2E)
22. Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. Investigating African-American Vernacular English in transformer-based text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5877–5883, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sophie+Groenwold%2C+Lily+Ou%2C+Aesha+Parekh%2C+Samhita+Honnavalli%2C+Sharon+Levy%2C+Diba+Mirza%2C+and+William+Yang+Wang%2E+2020%2E+Investigating+African%2DAmerican+Vernacular+English+in+transformer%2Dbased+text+generatio)
23. Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard. 2021. Mitigating racial biases in toxic language detection with an equity-based ensemble framework. In Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1–11. [find](https://scholar.google.com/scholar?q=Matan+Halevy%2C+Camille+Harris%2C+Amy+Bruckman%2C+Diyi+Yang%2C+and+Ayanna+Howard%2E+2021%2E+Mitigating+racial+biases+in+toxic+language+detection+with+an+equity%2Dbased+ensemble+framework%2E+In+Equity+and+Access+in+Al)
24. Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dirk+Hovy+and+Shannon+L%2E+Spruit%2E+2016%2E+The+social+impact+of+natural+language+processing%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short+Pap)
25. Alyssa Hwang, William R. Frey, and Kathleen McKeown. 2020a. Towards augmenting lexical resources for slang and African American English. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 160–172, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL). [find](https://scholar.google.com/scholar?q=Alyssa+Hwang%2C+William+R%2E+Frey%2C+and+Kathleen+McKeown%2E+2020a%2E+Towards+augmenting+lexical+resources+for+slang+and+African+American+English%2E+In+Proceedings+of+the+7th+Workshop+on+NLP+for+Similar+Languages)
26. Alyssa Hwang, William R. Frey, and Kathleen McKeown. 2020b. Towards augmenting lexical resources for slang and African American English. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 160–172, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL). [find](https://scholar.google.com/scholar?q=Alyssa+Hwang%2C+William+R%2E+Frey%2C+and+Kathleen+McKeown%2E+2020b%2E+Towards+augmenting+lexical+resources+for+slang+and+African+American+English%2E+In+Proceedings+of+the+7th+Workshop+on+NLP+for+Similar+Languages)
27. Christian Ilbury. 2020. “sassy queens”: Stylistic orthographic variation in twitter and the enregisterment of aave. Journal of Sociolinguistics, 24(2):245–264. [find](https://scholar.google.com/scholar?q=Christian+Ilbury%2E+2020%2E+%E2%80%9Csassy+queens%E2%80%9D%3A+Stylistic+orthographic+variation+in+twitter+and+the+enregisterment+of+aave%2E+Journal+of+Sociolinguistics%2C+24%282%29%3A245%E2%80%93264%2E)
28. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875–1885, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mohit+Iyyer%2C+John+Wieting%2C+Kevin+Gimpel%2C+and+Luke+Zettlemoyer%2E+2018%2E+Adversarial+example+generation+with+syntactically+controlled+paraphrase+networks%2E+In+Proceedings+of+the+2018+Conference+of+the+Nort)
29. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Robin+Jia+and+Percy+Liang%2E+2017%2E+Adversarial+examples+for+evaluating+reading+comprehension+systems%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+202) [our card](https://md.mova.elamur.ai/free/en/docs/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
30. Erik Jones, Robin Jia, Aditi Raghunathan, and Percy Liang. 2020. Robust encodings: A framework for combating adversarial typos. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2752–2765, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Erik+Jones%2C+Robin+Jia%2C+Aditi+Raghunathan%2C+and+Percy+Liang%2E+2020%2E+Robust+encodings%3A+A+framework+for+combating+adversarial+typos%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computa)
31. Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2016. Learning a POS tagger for AAVE-like language. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1115–1120, San Diego, California. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anna+J%C3%B8rgensen%2C+Dirk+Hovy%2C+and+Anders+S%C3%B8gaard%2E+2016%2E+Learning+a+POS+tagger+for+AAVE%2Dlike+language%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+for+Computatio) [our card](https://md.mova.elamur.ai/free/en/docs/papers/jorgensen-2016-learning-pos-tagger-aave-like/)
32. David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating dialectal variability for socially equitable language identification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 51–57, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Jurgens%2C+Yulia+Tsvetkov%2C+and+Dan+Jurafsky%2E+2017%2E+Incorporating+dialectal+variability+for+socially+equitable+language+identification%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+)
33. Divyansh Kaushik, Eduard H. Hovy, and Zachary Chase Lipton. 2020. Learning the difference that makes A difference with counterfactually-augmented data. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Divyansh+Kaushik%2C+Eduard+H%2E+Hovy%2C+and+Zachary+Chase+Lipton%2E+2020%2E+Learning+the+difference+that+makes+A+difference+with+counterfactually%2Daugmented+data%2E+In+8th+International+Conference+on+Learning+Repr)
34. Christopher Kennedy. 2007. Vagueness and grammar: The semantics of relative and absolute gradable adjectives. Linguistics and philosophy, 30(1):1–45. [find](https://scholar.google.com/scholar?q=Christopher+Kennedy%2E+2007%2E+Vagueness+and+grammar%3A+The+semantics+of+relative+and+absolute+gradable+adjectives%2E+Linguistics+and+philosophy%2C+30%281%29%3A1%E2%80%9345%2E)
35. Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43–53, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Svetlana+Kiritchenko+and+Saif+Mohammad%2E+2018%2E+Examining+gender+and+race+bias+in+two+hundred+sentiment+analysis+systems%2E+In+Proceedings+of+the+Seventh+Joint+Conference+on+Lexical+and+Computational+Sema)
36. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689. [find](https://scholar.google.com/scholar?q=Allison+Koenecke%2C+Andrew+Nam%2C+Emily+Lake%2C+Joe+Nudell%2C+Minnie+Quartey%2C+Zion+Mengesha%2C+Connor+Toups%2C+John+R+Rickford%2C+Dan+Jurafsky%2C+and+Sharad+Goel%2E+2020%2E+Racial+disparities+in+automated+speech+recognit)
37. Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737–762, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kalpesh+Krishna%2C+John+Wieting%2C+and+Mohit+Iyyer%2E+2020%2E+Reformulating+unsupervised+style+transfer+as+paraphrase+generation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language) [our card](https://md.mova.elamur.ai/free/en/docs/papers/krishna-2020-reformulating-unsupervised-style-transfer-as/)
38. William Labov. 1972. Language in the inner city: Studies in the Black English vernacular. 3. University of Pennsylvania Press. [find](https://scholar.google.com/scholar?q=William+Labov%2E+1972%2E+Language+in+the+inner+city%3A+Studies+in+the+Black+English+vernacular%2E+3%2E+University+of+Pennsylvania+Press%2E)
39. William Labov. 1995. The case of the missing copula: The interpretation of zeros in african-american english. Language, 1:25–54. [find](https://scholar.google.com/scholar?q=William+Labov%2E+1995%2E+The+case+of+the+missing+copula%3A+The+interpretation+of+zeros+in+african%2Damerican+english%2E+Language%2C+1%3A25%E2%80%9354%2E)
40. William Labov et al. 1998. Co-existent systems in african-american vernacular english. African-American English: Structure, History and Use, pages 110–153. [find](https://scholar.google.com/scholar?q=William+Labov+et+al%2E+1998%2E+Co%2Dexistent+systems+in+african%2Damerican+vernacular+english%2E+African%2DAmerican+English%3A+Structure%2C+History+and+Use%2C+pages+110%E2%80%93153%2E)
41. Guillaume Lample, Sandeep Subramanian, Eric Michael Smith, Ludovic Denoyer, Marc’Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Sandeep+Subramanian%2C+Eric+Michael+Smith%2C+Ludovic+Denoyer%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Y%2DLan+Boureau%2E+2019%2E+Multiple%2Dattribute+text+rewriting%2E+In+7th+International+Conference+on+Learnin)
42. Claudia Lange. 2012. The syntax of spoken Indian English, volume 45. John Benjamins Publishing. [find](https://scholar.google.com/scholar?q=Claudia+Lange%2E+2012%2E+The+syntax+of+spoken+Indian+English%2C+volume+45%2E+John+Benjamins+Publishing%2E)
43. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t)
44. Rosina Lippi-Green. 1997. What we talk about when we talk about ebonics: Why definitions matter. The Black Scholar, 27(2):7–11. [find](https://scholar.google.com/scholar?q=Rosina+Lippi%2DGreen%2E+1997%2E+What+we+talk+about+when+we+talk+about+ebonics%3A+Why+definitions+matter%2E+The+Black+Scholar%2C+27%282%29%3A7%E2%80%9311%2E)
45. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E)
46. Stefan Martin, Walt Wolfram, et al. 1998. The sentence in african-american vernacular english. African American English: structure, history, and use, pages 11–36. [find](https://scholar.google.com/scholar?q=Stefan+Martin%2C+Walt+Wolfram%2C+et+al%2E+1998%2E+The+sentence+in+african%2Damerican+vernacular+english%2E+African+American+English%3A+structure%2C+history%2C+and+use%2C+pages+11%E2%80%9336%2E)
47. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tom+McCoy%2C+Ellie+Pavlick%2C+and+Tal+Linzen%2E+2019%2E+Right+for+the+wrong+reasons%3A+Diagnosing+syntactic+heuristics+in+natural+language+inference%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association) [our card](https://md.mova.elamur.ai/free/en/docs/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)
48. Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 3111– 3119. [find](https://scholar.google.com/scholar?q=Tom%C3%A1s+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Gregory+S%2E+Corrado%2C+and+Jeffrey+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+Neural+Information+Proc)
49. Dong Nguyen, A Seza Doğruöz, Carolyn P Rosé, and Franciska De Jong. 2016. Computational sociolinguistics: A survey. Computational linguistics, 42(3):537–593. [find](https://scholar.google.com/scholar?q=Dong+Nguyen%2C+A+Seza+Dog%CC%86ru%C3%B6z%2C+Carolyn+P+Ros%C3%A9%2C+and+Franciska+De+Jong%2E+2016%2E+Computational+sociolinguistics%3A+A+survey%2E+Computational+linguistics%2C+42%283%29%3A537%E2%80%93593%2E)
50. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2020%2E+Adversarial+NLI%3A+A+new+benchmark+for+natural+language+understanding%2E+In+Proceedings+of+the+58th+Annual+Meetin)
51. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu)
52. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. [find](https://scholar.google.com/scholar?q=Siva+Reddy%2C+Danqi+Chen%2C+and+Christopher+D%2E+Manning%2E+2019%2E+CoQA%3A+A+conversational+question+answering+challenge%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A249%E2%80%93266%2E)
53. Angela Reyes. 2005. Appropriation of african american slang by asian american youth 1. Journal of Sociolinguistics, 9(4):509–532. [find](https://scholar.google.com/scholar?q=Angela+Reyes%2E+2005%2E+Appropriation+of+african+american+slang+by+asian+american+youth+1%2E+Journal+of+Sociolinguistics%2C+9%284%29%3A509%E2%80%93532%2E)
54. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856–865, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2018%2E+Semantically+equivalent+adversarial+rules+for+debugging+NLP+models%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Compu)
55. John R Rickford. 2002. How linguists approach the study of language and dialect. Ms. January. [find](https://scholar.google.com/scholar?q=John+R+Rickford%2E+2002%2E+How+linguists+approach+the+study+of+language+and+dialect%2E+Ms%2E+January%2E)
56. John R Rickford. 2016. Raciolinguistics: How language shapes our ideas about race. Oxford University Press. [find](https://scholar.google.com/scholar?q=John+R+Rickford%2E+2016%2E+Raciolinguistics%3A+How+language+shapes+our+ideas+about+race%2E+Oxford+University+Press%2E)
57. John R Rickford and Sharese King. 2016. Language and linguistics on trial: Hearing rachel jeantel (and other vernacular speakers) in the courtroom and beyond. Language, 92(4):948–988. [find](https://scholar.google.com/scholar?q=John+R+Rickford+and+Sharese+King%2E+2016%2E+Language+and+linguistics+on+trial%3A+Hearing+rachel+jeantel+%28and+other+vernacular+speakers%29+in+the+courtroom+and+beyond%2E+Language%2C+92%284%29%3A948%E2%80%93988%2E)
58. Anthony Rios. 2020. Fuzze: Fuzzy fairness evaluation of offensive language classifiers on african-american english. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 881–889. [find](https://scholar.google.com/scholar?q=Anthony+Rios%2E+2020%2E+Fuzze%3A+Fuzzy+fairness+evaluation+of+offensive+language+classifiers+on+african%2Damerican+english%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+34%2C+pages+8)
59. Douglas Schuler and Aki Namioka. 1993. Participatory design: Principles and practices. CRC Press. [find](https://scholar.google.com/scholar?q=Douglas+Schuler+and+Aki+Namioka%2E+1993%2E+Participatory+design%3A+Principles+and+practices%2E+CRC+Press%2E)
60. Philippa Shoemark, James Kirby, and Sharon Goldwater. 2018. Inducing a lexicon of sociolinguistic variables from code-mixed text. In Proceedings of the 2018 EMNLP Workshop W-NUT: The 4th Workshop on Noisy User-generated Text, pages 1–6, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philippa+Shoemark%2C+James+Kirby%2C+and+Sharon+Goldwater%2E+2018%2E+Inducing+a+lexicon+of+sociolinguistic+variables+from+code%2Dmixed+text%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+W%2DNUT%3A+The+4th+Workshop+on+N)
61. Jack Sidnell. 2002. African american vernacular english (aave) grammar. 1.7. Retrieved April, 19(2009):16. [find](https://scholar.google.com/scholar?q=Jack+Sidnell%2E+2002%2E+African+american+vernacular+english+%28aave%29+grammar%2E+1%2E7%2E+Retrieved+April%2C+19%282009%29%3A16%2E)
62. Arthur K Spears et al. 1998. African-american language use: Ideology and so-called obscenity. African-American English: Structure, history, and use, pages 226–250. [find](https://scholar.google.com/scholar?q=Arthur+K+Spears+et+al%2E+1998%2E+African%2Damerican+language+use%3A+Ideology+and+so%2Dcalled+obscenity%2E+African%2DAmerican+English%3A+Structure%2C+history%2C+and+use%2C+pages+226%E2%80%93250%2E)
63. Ian Stewart. 2014. Now we stronger than ever: African-American English syntax in Twitter. In Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 31– 37, Gothenburg, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ian+Stewart%2E+2014%2E+Now+we+stronger+than+ever%3A+African%2DAmerican+English+syntax+in+Twitter%2E+In+Proceedings+of+the+Student+Research+Workshop+at+the+14th+Conference+of+the+European+Chapter+of+the+Associat)
64. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+7th+Internation)
65. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2E)
66. Lionel Wee. 2008. Singapore english: morphology and syntax. In A handbook of varieties of English, pages 2250–2264. De Gruyter Mouton. [find](https://scholar.google.com/scholar?q=Lionel+Wee%2E+2008%2E+Singapore+english%3A+morphology+and+syntax%2E+In+A+handbook+of+varieties+of+English%2C+pages+2250%E2%80%932264%2E+De+Gruyter+Mouton%2E)
67. Walt Wolfram and Natalie Schilling. 2015. American English: dialects and variation. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=Walt+Wolfram+and+Natalie+Schilling%2E+2015%2E+American+English%3A+dialects+and+variation%2E+John+Wiley+%26+Sons%2E)
68. Nasser Zalmout, Alexander Erdmann, and Nizar Habash. 2018. Noise-robust morphological disambiguation for dialectal Arabic. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 953–964, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nasser+Zalmout%2C+Alexander+Erdmann%2C+and+Nizar+Habash%2E+2018%2E+Noise%2Drobust+morphological+disambiguation+for+dialectal+Arabic%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+As) [our card](https://md.mova.elamur.ai/free/en/docs/papers/naacl-2018-zalmout-noise-robust-morphological-disambiguation-dialectal/)
69. Marcos Zampieri, Liling Tan, Nikola Ljubešić, and Jörg Tiedemann. 2014. A report on the DSL shared task 2014. In Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, pages 58–67, Dublin, Ireland. Association for Computational Linguistics and Dublin City University. [find](https://scholar.google.com/scholar?q=Marcos+Zampieri%2C+Liling+Tan%2C+Nikola+Ljube%C5%A1ic%CC%81%2C+and+J%C3%B6rg+Tiedemann%2E+2014%2E+A+report+on+the+DSL+shared+task+2014%2E+In+Proceedings+of+the+First+Workshop+on+Applying+NLP+Tools+to+Similar+Languages%2C+Varietie)
70. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yuan+Zhang%2C+Jason+Baldridge%2C+and+Luheng+He%2E+2019%2E+PAWS%3A+Paraphrase+adversaries+from+word+scrambling%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computat)
71. Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. 2020. It’s morphin’ time! Combating linguistic discrimination with inflectional perturbations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2920–2935, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samson+Tan%2C+Shafiq+Joty%2C+Min%2DYen+Kan%2C+and+Richard+Socher%2E+2020%2E+It%E2%80%99s+morphin%E2%80%99+time%21+Combating+linguistic+discrimination+with+inflectional+perturbations%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+th)
72. Deanna Thompson. 2016. The morpho-syntax of aspectual stay in aave. [find](https://scholar.google.com/scholar?q=Deanna+Thompson%2E+2016%2E+The+morpho%2Dsyntax+of+aspectual+stay+in+aave%2E)
73. Joe Trotta and Oleg Blyahher. 2011. Game done changed: A look at selected aave features in the tv series the wire. Moderna språk, 105(1):15–42. [find](https://scholar.google.com/scholar?q=Joe+Trotta+and+Oleg+Blyahher%2E+2011%2E+Game+done+changed%3A+A+look+at+selected+aave+features+in+the+tv+series+the+wire%2E+Moderna+spr%C3%A5k%2C+105%281%29%3A15%E2%80%9342%2E)