[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/jorgensen-2016-learning-pos-tagger-aave-like/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Learning a POS tagger for AAVE-like language

# Learning a POS tagger for AAVE-like language

**Authors:** Anna Jørgensen, Dirk Hovy and [Anders Søgaard](https://md.mova.elamur.ai/free/en/docs/papers/authors/anders-sogaard/) · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) **Link:** [aclanthology.org/N16-1130](https://aclanthology.org/N16-1130/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**34** citations](https://openalex.org/works?filter=cites:W2460907386) openalex · 2026-10-18[**25** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/jorgensen-2016-learning-pos-tagger-aave-like/) of the 140673 papers whose bibliographies we read[Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://aclanthology.org/volumes/N16-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/N16-1130/)
- [PDF](https://aclanthology.org/N16-1130.pdf)
- [DOI 10.18653/v1/n16-1130](https://doi.org/10.18653/v1/n16-1130)
- [OpenAlex](https://openalex.org/W2460907386)

**Authors (3)**

1. Anna Jørgensen
2. [Dirk Hovy](https://openalex.org/A5084505122) [ORCID](https://orcid.org/0000-0002-4618-3127) · h 41 · 7453 citations · 198 works
3. [Anders Søgaard](https://openalex.org/A5018138946) [ORCID](https://orcid.org/0000-0001-5250-4276) · h 43 · 8552 citations · 440 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

Newswire POS taggers do poorly on subtitles, hip-hop lyrics and tweets in African-American Vernacular English. The authors train on 1,827 annotated tweets plus about 4.5 million unlabelled sentences. They partly label the unlabelled text with tag dictionaries mined from Wiktionary, a slang glossary, a list of names and Urban Dictionary. Two methods use the dictionaries: ambiguous supervision in a cost-sensitive structured perceptron, where the mined tags carry no cost, and self-training with the dictionary as type constraints. The best system reaches 89% accuracy on subtitles, 85% on lyrics and 83% on tweets. The abstract reports up to 55% error reduction over a newswire tagger and 15 to 25% over a Twitter tagger. Most of the gain is on unseen words (ambiguous supervision cuts their error by 14% on average), while known words gain about 1%.

## Ideas

- I1. `en/src/ptag.rs`: an ambiguous-supervision update for the existing perceptron. If the predicted tag is inside a word's mined tag set, make no update; otherwise update with cost 1. Measure on EWT dev and on a small hand-annotated AAVE-like set, run twice.
- I2. Type constraints at tagging time: for known words, allow only the tags in the lexicon of `en/src/dict.rs`. Report accuracy on known and unseen words apart, because the paper's gains are almost all on unseen words.
- I3. The sources matter: Urban Dictionary scraping is not usable under our rules. A Wiktionary dump is CC BY-SA, so quote only and do not carry it into a release.
- I4. Stop self-training by a rule the paper uses: stop when dev accuracy falls over three iterations. Keep the stopping point in the log.

## For Mova

The tagger chapters are [10. Word classes](https://md.mova.elamur.ai/free/en/docs/languages/en/10-word-classes/) and [37. Register and variation](https://md.mova.elamur.ai/free/en/docs/languages/en/37-register-and-variation/). Spelling variants such as g-dropping belong in [06. Spelling rules: inflection, derivation, compounds and national standards](https://md.mova.elamur.ai/free/en/docs/languages/en/06-spelling-rules/), and slang in [19. The lexicon: vocabulary, multiword expressions and register](https://md.mova.elamur.ai/free/en/docs/languages/en/19-lexicon-mwe-register/). The code is `en/src/ptag.rs`, `en/src/tag.rs` and `en/src/dict.rs`. Measures: accuracy split into known and unseen words, the run-twice check, and the gain from the ambiguous update over the plain perceptron on the same data.

Topics: [Part-of-speech tagging](https://md.mova.elamur.ai/free/en/docs/papers/topics/part-of-speech-tagging/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/n16-1130](https://doi.org/10.18653/v1/n16-1130)

```
@inproceedings{jorgensen2016learning,
  title = {{Learning a POS tagger for AAVE-like language}},
  author = {Jørgensen, Anna and Hovy, Dirk and Søgaard, Anders},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1130},
  url = {https://doi.org/10.18653/v1/n16-1130}
}
```

- [← previous Automatic Dialect Density Estimation for African American English](https://md.mova.elamur.ai/free/en/docs/papers/johnson-2022-automatic-dialect-density-estimation-african/)
- [next → How Challenging is Sarcasm versus Irony Classification?: A Study With a Dataset from English Literature](https://md.mova.elamur.ai/free/en/docs/papers/joshi-2016-how-challenging-is-sarcasm-versus/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hanan Aldarmaki and Mona Diab. 2015. Robust part-of-speech tagging of Arabic text. In Proceedings of the Second Workshop on Arabic Natural Language Processing, Beijing, China. [find](https://scholar.google.com/scholar?q=Hanan+Aldarmaki+and+Mona+Diab%2E+2015%2E+Robust+part%2Dof%2Dspeech+tagging+of+Arabic+text%2E+In+Proceedings+of+the+Second+Workshop+on+Arabic+Natural+Language+Processing%2C+Beijing%2C+China%2E)
2. Delphine Bernhard and Anne-Laure Ligozat. 2013. Hassle-free pos-tagging for the Alsatian dialects. In Marcos Zampieri and Sascha Diwersy, editors, Non-Standard Data Sources in Corpus Based-Research, pages 85–92. ZSM Studien. [find](https://scholar.google.com/scholar?q=Delphine+Bernhard+and+Anne%2DLaure+Ligozat%2E+2013%2E+Hassle%2Dfree+pos%2Dtagging+for+the+Alsatian+dialects%2E+In+Marcos+Zampieri+and+Sascha+Diwersy%2C+editors%2C+Non%2DStandard+Data+Sources+in+Corpus+Based%2DResearch%2C+p)
3. Rahma Boujelbane, Meriem Ellouze Khemekhem, and Lamia Hadrich Belguith. 2013. Mapping rules for building a Tunisian dialect lexicon and generating corpora. In International Joint Conference on Natural Language Processing, pages 419–428. [find](https://scholar.google.com/scholar?q=Rahma+Boujelbane%2C+Meriem+Ellouze+Khemekhem%2C+and+Lamia+Hadrich+Belguith%2E+2013%2E+Mapping+rules+for+building+a+Tunisian+dialect+lexicon+and+generating+corpora%2E+In+International+Joint+Conference+on+Natural)
4. Michael Collins. 2002. Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms. In EMNLP. [find](https://scholar.google.com/scholar?q=Michael+Collins%2E+2002%2E+Discriminative+Training+Methods+for+Hidden+Markov+Models%3A+Theory+and+Experiments+with+Perceptron+Algorithms%2E+In+EMNLP%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/collins-2002-discriminative-training-methods-hidden-markov/)
5. Dipanjan Das and Slav Petrov. 2011. Unsupervised part-of-speech tagging with bilingual graph-based projections. In ACL, pages 256–263. [find](https://scholar.google.com/scholar?q=Dipanjan+Das+and+Slav+Petrov%2E+2011%2E+Unsupervised+part%2Dof%2Dspeech+tagging+with+bilingual+graph%2Dbased+projections%2E+In+ACL%2C+pages+256%E2%80%93263%2E)
6. Kevin Duh and Katrin Kirchhoff. 2005. Pos tagging of dialectal Arabic: A minimally supervised approach. In Proceedings in the ACL Workshop on Computational Approaches to Semitic Languages. [find](https://scholar.google.com/scholar?q=Kevin+Duh+and+Katrin+Kirchhoff%2E+2005%2E+Pos+tagging+of+dialectal+Arabic%3A+A+minimally+supervised+approach%2E+In+Proceedings+in+the+ACL+Workshop+on+Computational+Approaches+to+Semitic+Languages%2E)
7. Dan Garrette and Jason Baldridge. 2013. Learning a part-of-speech tagger from two hours of annotation. In Proceedings of NAACL-HLT, pages 138–147. [find](https://scholar.google.com/scholar?q=Dan+Garrette+and+Jason+Baldridge%2E+2013%2E+Learning+a+part%2Dof%2Dspeech+tagger+from+two+hours+of+annotation%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+138%E2%80%93147%2E)
8. Kevin Gimpel, Nathan Schneider, Brendan OConnor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In ACL. [find](https://scholar.google.com/scholar?q=Kevin+Gimpel%2C+Nathan+Schneider%2C+Brendan+OConnor%2C+Dipanjan+Das%2C+Daniel+Mills%2C+Jacob+Eisenstein%2C+Michael+Heilman%2C+Dani+Yogatama%2C+Jeffrey+Flanigan%2C+and+Noah+A%2E+Smith%2E+2011%2E+Part%2Dof%2Dspeech+tagging+for+twi)
9. Stephen Gouws and Anders Søgaard. 2015. Simple task-specific bilingual word embeddings. In Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1386– 1390. [find](https://scholar.google.com/scholar?q=Stephen+Gouws+and+Anders+S%C3%B8gaard%2E+2015%2E+Simple+task%2Dspecific+bilingual+word+embeddings%2E+In+Human+Language+Technologies%3A+The+2015+Annual+Conference+of+the+North+American+Chapter+of+the+ACL%2C+pages+1386%E2%80%93)
10. Bo Han and Timothy Baldwin. 2011. Lexical Normalisation of Short Text Messages: Makn Sens a #twitter. In ACL. [find](https://scholar.google.com/scholar?q=Bo+Han+and+Timothy+Baldwin%2E+2011%2E+Lexical+Normalisation+of+Short+Text+Messages%3A+Makn+Sens+a+%23twitter%2E+In+ACL%2E)
11. Dirk Hovy and Eduard Hovy. 2012. Exploiting partial annotations with em training. In Proceedings of the NAACL-HLT Workshop on the Induction of Linguistic Structure, pages 31–38. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dirk+Hovy+and+Eduard+Hovy%2E+2012%2E+Exploiting+partial+annotations+with+em+training%2E+In+Proceedings+of+the+NAACL%2DHLT+Workshop+on+the+Induction+of+Linguistic+Structure%2C+pages+31%E2%80%9338%2E+Association+for+Comput)
12. Dirk Hovy, Barbara Plank, Héctor Martı́nez Alonso, and Anders Søgaard. 2015a. Mining for unambiguous instances to adapt part-of-speech taggers to new domains. In Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1256–1261. [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Barbara+Plank%2C+He%CC%81ctor+Mart%C4%B1%CC%81nez+Alonso%2C+and+Anders+S%C3%B8gaard%2E+2015a%2E+Mining+for+unambiguous+instances+to+adapt+part%2Dof%2Dspeech+taggers+to+new+domains%2E+In+Human+Language+Technologies%3A+The+2015)
13. Dirk Hovy, Barbara Plank, Héctor Martı́nez Alonso, and Anders Søgaard. 2015b. Mining for unambiguous instances to adapt pos taggers to new domains. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Barbara+Plank%2C+He%CC%81ctor+Mart%C4%B1%CC%81nez+Alonso%2C+and+Anders+S%C3%B8gaard%2E+2015b%2E+Mining+for+unambiguous+instances+to+adapt+pos+taggers+to+new+domains%2E+In+NAACL%2DHLT%2E)
14. Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2015. Challenges of studying and processing dialects in social media. In Proceedings of the ACL Workshop on Noisy User-generated Text. [find](https://scholar.google.com/scholar?q=Anna+J%C3%B8rgensen%2C+Dirk+Hovy%2C+and+Anders+S%C3%B8gaard%2E+2015%2E+Challenges+of+studying+and+processing+dialects+in+social+media%2E+In+Proceedings+of+the+ACL+Workshop+on+Noisy+User%2Dgenerated+Text%2E)
15. Shen Li, João V. Graça, and Ben Taskar. 2012. Wikily supervised part-of-speech tagging. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. [find](https://scholar.google.com/scholar?q=Shen+Li%2C+Joa%CC%83o+V%2E+Grac%CC%A7a%2C+and+Ben+Taskar%2E+2012%2E+Wikily+supervised+part%2Dof%2Dspeech+tagging%2E+In+Proceedings+of+the+2012+Joint+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+Computatio)
16. T. Mikolov, K. Chen, G Corrado, and J. Dean. 2013a. Efficient estimation of word representations in vector space. ArXiv e-prints. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2C+K%2E+Chen%2C+G+Corrado%2C+and+J%2E+Dean%2E+2013a%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+ArXiv+e%2Dprints%2E)
17. T. Mikolov, W.T. Yih, and G. Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of NAACL-HLT. [find](https://scholar.google.com/scholar?q=T%2E+Mikolov%2C+W%2ET%2E+Yih%2C+and+G%2E+Zweig%2E+2013b%2E+Linguistic+regularities+in+continuous+space+word+representations%2E+In+Proceedings+of+NAACL%2DHLT%2E)
18. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In Proceedings of NAACL-HLT. [find](https://scholar.google.com/scholar?q=Olutobi+Owoputi%2C+Brendan+O%E2%80%99Connor%2C+Chris+Dyer%2C+Kevin+Gimpel%2C+Nathan+Schneider%2C+and+Noah+A%2E+Smith%2E+2013%2E+Improved+part%2Dof%2Dspeech+tagging+for+online+conversational+text+with+word+clusters%2E+In+Proceeding)
19. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Slav+Petrov%2C+Dipanjan+Das%2C+and+Ryan+McDonald%2E+2011%2E+A+universal+part%2Dof%2Dspeech+tagset%2E+In+Proceedings+of+LREC%2E)
20. Barbara Plank, Dirk Hovy, Ryan McDonald, and Anders Søgaard. 2014a. Adapting taggers to twitter with not-so-distant supervision. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1783– 1792. [find](https://scholar.google.com/scholar?q=Barbara+Plank%2C+Dirk+Hovy%2C+Ryan+McDonald%2C+and+Anders+S%C3%B8gaard%2E+2014a%2E+Adapting+taggers+to+twitter+with+not%2Dso%2Ddistant+supervision%2E+In+Proceedings+of+COLING+2014%2C+the+25th+International+Conference+on+Com)
21. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014b. Learning part-of-speech taggers with inter-annotator agreement loss. In EACL. [find](https://scholar.google.com/scholar?q=Barbara+Plank%2C+Dirk+Hovy%2C+and+Anders+S%C3%B8gaard%2E+2014b%2E+Learning+part%2Dof%2Dspeech+taggers+with+inter%2Dannotator+agreement+loss%2E+In+EACL%2E)
22. Anders Søgaard. 2013. Semi-supervised learning and domain adaptation for NLP. Morgan & Claypool. [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard%2E+2013%2E+Semi%2Dsupervised+learning+and+domain+adaptation+for+NLP%2E+Morgan+%26+Claypool%2E)
23. Ian Stewart. 2014. Now We Stronger Than Ever: African-American syntax on Twitter. In Proceedings of the Student Research Workshop to the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 26–30, Gothenburg, Sweden, April. [find](https://scholar.google.com/scholar?q=Ian+Stewart%2E+2014%2E+Now+We+Stronger+Than+Ever%3A+African%2DAmerican+syntax+on+Twitter%2E+In+Proceedings+of+the+Student+Research+Workshop+to+the+14th+Conference+of+the+European+Chapter+of+the+Association+for+)
24. Oscar Täckström, Dipanjan Das, Slav Petrov, Ryan Mc-Donald, and Joakim Nivre, 2013. Token and Type Constraints for Cross-Lingual Part-of-Speech Tagging, pages 1–12. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Oscar+Ta%CC%88ckstro%CC%88m%2C+Dipanjan+Das%2C+Slav+Petrov%2C+Ryan+Mc%2DDonald%2C+and+Joakim+Nivre%2C+2013%2E+Token+and+Type+Constraints+for+Cross%2DLingual+Part%2Dof%2DSpeech+Tagging%2C+pages+1%E2%80%9312%2E+Association+for+Computational+Lin)
25. Guillaume Wisniewski, Nicolas Pécheux, Sophir Gahbiche-Braham, and François Yvon. 2014. Cross-lingual part-of-speech tagging through ambiguous learning. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Guillaume+Wisniewski%2C+Nicolas+Pe%CC%81cheux%2C+Sophir+Gahbiche%2DBraham%2C+and+Franc%CC%A7ois+Yvon%2E+2014%2E+Cross%2Dlingual+part%2Dof%2Dspeech+tagging+through+ambiguous+learning%2E+In+Proceedings+of+the+2014+Conference+on+Empi)