[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2016-hashemi-evaluation-parser-robustness-ungrammatical-sentences/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / An Evaluation of Parser Robustness for Ungrammatical Sentences

# An Evaluation of Parser Robustness for Ungrammatical Sentences

**Authors:** Homa B. Hashemi, Rebecca Hwa · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing **Link:** [aclanthology.org/D16-1182](https://aclanthology.org/D16-1182/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**12** citations](https://openalex.org/works?filter=cites:W2561981731) openalex · 2026-10-18[**8** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2016-hashemi-evaluation-parser-robustness-ungrammatical-sentences/) of the 140673 papers whose bibliographies we read[Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing](https://aclanthology.org/volumes/D16-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/D16-1182/)
- [PDF](https://aclanthology.org/D16-1182.pdf)
- [DOI 10.18653/v1/d16-1182](https://doi.org/10.18653/v1/d16-1182)
- [OpenAlex](https://openalex.org/W2561981731)

**Authors (2)**

1. [Homa B. Hashemi](https://openalex.org/A5043472268) h 8 · 315 citations · 15 works
2. [Rebecca Hwa](https://openalex.org/A5070135550) [ORCID](https://orcid.org/0000-0003-1158-7014) · h 25 · 3228 citations · 96 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The question is whether a dependency parser that is accurate on clean text also keeps its structure when the input is ungrammatical. The authors compare eight parsers, trained on the Penn Treebank and on Tweebank, on 10,000 learner sentences (FCE) and 10,000 machine-translation outputs, each paired with its corrected version. Robustness is scored without a gold tree: the parse of the corrected sentence is the reference, and dependencies that touch an extra or missing word are ignored. The best ESL robustness F1 rises from 93.72 (Turbo, PTB) to 94.36 (Malt, Tweebank), and robustness falls faster with the number of errors on MT than on ESL. Missing-word errors are the hardest type, replacement errors the easiest, and errors on closed-class words (prepositions, determiners) hurt more than errors on open-class words in most settings.

## Ideas

- I1. `en parse-eval --robust`: parse the ungrammatical sentence and its corrected version with `en`, then score shared dependencies over the unmatched words as the paper defines; measure the robustness F1 on learner text that we write or license openly, and on FCE as test only.
- I2. `en/seeds/errors/`: check the missing-word rules (missing determiner, auxiliary, preposition) against the robustness analysis; measure the flag recall for missing versus unnecessary words on the same learner sample.
- I3. `en/src/ptag.rs` and `en/src/parse.rs`: a closed-class lexicon feature, so that a wrong preposition or determiner is not allowed to cascade into the head choice; measure robustness F1 on closed-class errors before and after.
- I4. `en beam-curve` extended: plot robustness F1 against the number of errors per sentence for the greedy and the beam-8 parser; measure the slope for each, as the paper does for its eight parsers.

## For Mova

Touches `languages/en/12-pronouns-determiners-numerals`, `23-complementation` and `27-negation`, the `en` crate (`parse.rs`, `ptag.rs`), `en/seeds/errors/` and the `en parse-eval` command. FCE is research-only: test only. Measure first the robustness F1 of the current `en` parser on a small set of sentences we can release.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/d16-1182](https://doi.org/10.18653/v1/d16-1182)

```
@inproceedings{hashemi2016evaluation,
  title = {{An Evaluation of Parser Robustness for Ungrammatical Sentences}},
  author = {Hashemi, Homa B. and Hwa, Rebecca},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1182},
  url = {https://doi.org/10.18653/v1/d16-1182}
}
```

- [← previous English Resource Semantics](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2016-flickinger-english-resource-semantics/)
- [next → LSTM CCG Parsing](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2016-lewis-lstm-ccg-parsing/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. arXiv preprint arXiv:1603.06042. [↗](https://arxiv.org/abs/1603.06042)
2. Johnny Bigert, Jonas Sjöbergh, Ola Knutsson, and Magnus Sahlgren. 2005. Unsupervised evaluation of parser robustness. In Computational Linguistics and Intelligent Text Processing, pages 142–154. [find](https://scholar.google.com/scholar?q=Johnny+Bigert%2C+Jonas+Sjo%CC%88bergh%2C+Ola+Knutsson%2C+and+Magnus+Sahlgren%2E+2005%2E+Unsupervised+evaluation+of+parser+robustness%2E+In+Computational+Linguistics+and+Intelligent+Text+Processing%2C+pages+142%E2%80%93154%2E)
3. E. Black, S. Abney, S. Flickenger, C. Gdaniec, C. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski. 1991. A procedure for quantitatively comparing the syntactic coverage of English grammars. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 306– 311. [find](https://scholar.google.com/scholar?q=E%2E+Black%2C+S%2E+Abney%2C+S%2E+Flickenger%2C+C%2E+Gdaniec%2C+C%2E+Grishman%2C+P%2E+Harrison%2C+D%2E+Hindle%2C+R%2E+Ingria%2C+F%2E+Jelinek%2C+J%2E+Klavans%2C+M%2E+Liberman%2C+M%2E+Marcus%2C+S%2E+Roukos%2C+B%2E+Santorini%2C+and+T%2E+Strzalkowski%2E+1991%2E+A+pro)
4. Bernd Bohnet. 2010. Very high accuracy and fast dependency parsing is not a contradiction. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 89–97. [find](https://scholar.google.com/scholar?q=Bernd+Bohnet%2E+2010%2E+Very+high+accuracy+and+fast+dependency+parsing+is+not+a+contradiction%2E+In+Proceedings+of+the+23rd+International+Conference+on+Computational+Linguistics%2C+pages+89%E2%80%9397%2E)
5. Aoife Cahill. 2015. Parsing learner text: to shoehorn or not to shoehorn. In Proceedings of LAW IX - The 9th Linguistic Annotation Workshop, page 144. [find](https://scholar.google.com/scholar?q=Aoife+Cahill%2E+2015%2E+Parsing+learner+text%3A+to+shoehorn+or+not+to+shoehorn%2E+In+Proceedings+of+LAW+IX+%2D+The+9th+Linguistic+Annotation+Workshop%2C+page+144%2E)
6. Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750. [find](https://scholar.google.com/scholar?q=Danqi+Chen+and+Christopher+D+Manning%2E+2014%2E+A+fast+and+accurate+dependency+parser+using+neural+networks%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNL) [our card](https://md.mova.elamur.ai/free/en/docs/papers/missing-2014-chen-fast-accurate-dependency-parser-neural/)
7. Jinho D Choi, Joel Tetreault, and Amanda Stent. 2015. It depends: Dependency parser comparison using a web-based evaluation tool. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 26–31. [find](https://scholar.google.com/scholar?q=Jinho+D+Choi%2C+Joel+Tetreault%2C+and+Amanda+Stent%2E+2015%2E+It+depends%3A+Dependency+parser+comparison+using+a+web%2Dbased+evaluation+tool%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Compu) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2015-choi-it-depends-dependency-parser-comparison/)
8. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493– 2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+The+Journal+of+Machine+Learning+Research%2C+12%3A2)
9. Joachim Daiber and Rob van der Goot. 2016. The denoised web treebank: Evaluating dependency parsing under noisy input conditions. In LREC. [find](https://scholar.google.com/scholar?q=Joachim+Daiber+and+Rob+van+der+Goot%2E+2016%2E+The+denoised+web+treebank%3A+Evaluating+dependency+parsing+under+noisy+input+conditions%2E+In+LREC%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2016-daiber-denoised-web-treebank-evaluating-dependency/)
10. Marie-Catherine De Marneffe, Bill MacCartney, Christopher D Manning, et al. 2006. Generating typed dependency parses from phrase structure parses. In LREC, number 2006, pages 449–454. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+De+Marneffe%2C+Bill+MacCartney%2C+Christopher+D+Manning%2C+et+al%2E+2006%2E+Generating+typed+dependency+parses+from+phrase+structure+parses%2E+In+LREC%2C+number+2006%2C+pages+449%E2%80%93454%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/map-2006-marneffe-generating-typed-dependency-parses-phrase/)
11. Jennifer Foster, Özlem Çetinoglu, Joachim Wagner, Joseph Le Roux, Stephen Hogan, Joakim Nivre, Deirdre Hogan, Josef Van Genabith, et al. 2011. # hardtoparse: POS tagging and parsing the twitterverse. In proceedings of the Workshop On Analyzing Microtext (AAAI 2011), pages 20–25. [find](https://scholar.google.com/scholar?q=Jennifer+Foster%2C+O%CC%88zlem+C%CC%A7etinoglu%2C+Joachim+Wagner%2C+Joseph+Le+Roux%2C+Stephen+Hogan%2C+Joakim+Nivre%2C+Deirdre+Hogan%2C+Josef+Van+Genabith%2C+et+al%2E+2011%2E+%23+hardtoparse%3A+POS+tagging+and+parsing+the+twitterverse)
12. Jennifer Foster. 2004. Parsing ungrammatical input: an evaluation procedure. In LREC. [find](https://scholar.google.com/scholar?q=Jennifer+Foster%2E+2004%2E+Parsing+ungrammatical+input%3A+an+evaluation+procedure%2E+In+LREC%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2004-foster-parsing-ungrammatical-input-evaluation-procedure/)
13. Jennifer Foster. 2007. Treebanks gone bad. International Journal of Document Analysis and Recognition, 10(3-4):129–145. [find](https://scholar.google.com/scholar?q=Jennifer+Foster%2E+2007%2E+Treebanks+gone+bad%2E+International+Journal+of+Document+Analysis+and+Recognition%2C+10%283%2D4%29%3A129%E2%80%93145%2E)
14. Jennifer Foster. 2010. “cba to check the spelling” investigating parser performance on discussion forum posts. In The Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 381–384. [find](https://scholar.google.com/scholar?q=Jennifer+Foster%2E+2010%2E+%E2%80%9Ccba+to+check+the+spelling%E2%80%9D+investigating+parser+performance+on+discussion+forum+posts%2E+In+The+Annual+Conference+of+the+North+American+Chapter+of+the+Association+for+Computation) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2010-foster-cba-check-spelling-investigating-parser/)
15. Jeroen Geertzen, Theodora Alexopoulou, and Anna Korhonen. 2013. Automatic linguistic annotation of large scale l2 databases: the EF-Cambridge open language database (EFCamDat). In Proceedings of the 31st Second Language Research Forum. [find](https://scholar.google.com/scholar?q=Jeroen+Geertzen%2C+Theodora+Alexopoulou%2C+and+Anna+Korhonen%2E+2013%2E+Automatic+linguistic+annotation+of+large+scale+l2+databases%3A+the+EF%2DCambridge+open+language+database+%28EFCamDat%29%2E+In+Proceedings+of+the+3)
16. Daniel Gildea. 2001. Corpus variation and parser performance. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 167–202. [find](https://scholar.google.com/scholar?q=Daniel+Gildea%2E+2001%2E+Corpus+variation+and+parser+performance%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+167%E2%80%93202%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2001-gildea-corpus-variation-parser-performance/)
17. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In ACL-HLT, pages 42–47. [find](https://scholar.google.com/scholar?q=Kevin+Gimpel%2C+Nathan+Schneider%2C+Brendan+O%E2%80%99Connor%2C+Dipanjan+Das%2C+Daniel+Mills%2C+Jacob+Eisenstein%2C+Michael+Heilman%2C+Dani+Yogatama%2C+Jeffrey+Flanigan%2C+and+Noah+A+Smith%2E+2011%2E+Part%2Dof%2Dspeech+tagging+for+Twi)
18. Tuomo Kakkonen. 2007. Robustness evaluation of two CCG, a PCFG and a link grammar parsers. Proceedings of the 3rd Language & Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics. [find](https://scholar.google.com/scholar?q=Tuomo+Kakkonen%2E+2007%2E+Robustness+evaluation+of+two+CCG%2C+a+PCFG+and+a+link+grammar+parsers%2E+Proceedings+of+the+3rd+Language+%26+Technology+Conference%3A+Human+Language+Technologies+as+a+Challenge+for+Compu)
19. Lingpeng Kong and Noah A Smith. 2014. An empirical comparison of parsing methods for stanford dependencies. arXiv preprint arXiv:1404.4314. [↗](https://arxiv.org/abs/1404.4314) [our card](https://md.mova.elamur.ai/free/en/docs/papers/enud-2014-kong-empirical-comparison-parsing-methods-stanford/)
20. Lingpeng Kong, Nathan Schneider, Swabha Swayamdipta, Archna Bhatia, Chris Dyer, and Noah A Smith. 2014. A dependency parser for tweets. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Lingpeng+Kong%2C+Nathan+Schneider%2C+Swabha+Swayamdipta%2C+Archna+Bhatia%2C+Chris+Dyer%2C+and+Noah+A+Smith%2E+2014%2E+A+dependency+parser+for+tweets%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2014-kong-dependency-parser-tweets/)
21. Jonathan K Kummerfeld, David Hall, James R Curran, and Dan Klein. 2012. Parser showdown at the wall street corral: An empirical investigation of error types in parser output. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1048–1059. [find](https://scholar.google.com/scholar?q=Jonathan+K+Kummerfeld%2C+David+Hall%2C+James+R+Curran%2C+and+Dan+Klein%2E+2012%2E+Parser+showdown+at+the+wall+street+corral%3A+An+empirical+investigation+of+error+types+in+parser+output%2E+In+Proceedings+of+the+201) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2012-kummerfeld-parser-showdown-wall-street-corral/)
22. André FT Martins, Miguel Almeida, and Noah A Smith. 2013. Turning on the turbo: Fast third-order non-projective turbo parsers. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 617–622. [find](https://scholar.google.com/scholar?q=Andre%CC%81+FT+Martins%2C+Miguel+Almeida%2C+and+Noah+A+Smith%2E+2013%2E+Turning+on+the+turbo%3A+Fast+third%2Dorder+non%2Dprojective+turbo+parsers%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computa) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2013-martins-turning-turbo-fast-third-order/)
23. David McClosky, Eugene Charniak, and Mark Johnson. 2010. Automatic domain adaptation for parsing. In The Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 28–36. [find](https://scholar.google.com/scholar?q=David+McClosky%2C+Eugene+Charniak%2C+and+Mark+Johnson%2E+2010%2E+Automatic+domain+adaptation+for+parsing%2E+In+The+Annual+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistic)
24. Ryan McDonald and Joakim Nivre. 2011. Analyzing and integrating dependency parsers. Computational Linguistics, 37(1):197–230. [find](https://scholar.google.com/scholar?q=Ryan+McDonald+and+Joakim+Nivre%2E+2011%2E+Analyzing+and+integrating+dependency+parsers%2E+Computational+Linguistics%2C+37%281%29%3A197%E2%80%93230%2E)
25. Ryan T McDonald and Fernando CN Pereira. 2006. Online learning of approximate dependency parsing algorithms. In EACL. [find](https://scholar.google.com/scholar?q=Ryan+T+McDonald+and+Fernando+CN+Pereira%2E+2006%2E+Online+learning+of+approximate+dependency+parsing+algorithms%2E+In+EACL%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/continual-2006-mcdonald-online-learning-approximate-dependency-parsing/)
26. Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gülsen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007. Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(02):95–135. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Johan+Hall%2C+Jens+Nilsson%2C+Atanas+Chanev%2C+Gu%CC%88lsen+Eryigit%2C+Sandra+Ku%CC%88bler%2C+Svetoslav+Marinov%2C+and+Erwin+Marsi%2E+2007%2E+Maltparser%3A+A+language%2Dindependent+system+for+data%2Ddriven+dependency+p)
27. Niels Ott and Ramon Ziai. 2010. Evaluating dependency parsing performance on german learner language. Proceedings of the Ninth Workshop on Treebanks and Linguistic Theories (TLT-9), 9:175–186. [find](https://scholar.google.com/scholar?q=Niels+Ott+and+Ramon+Ziai%2E+2010%2E+Evaluating+dependency+parsing+performance+on+german+learner+language%2E+Proceedings+of+the+Ninth+Workshop+on+Treebanks+and+Linguistic+Theories+%28TLT%2D9%29%2C+9%3A175%E2%80%93186%2E)
28. Slav Petrov, Pi-Chuan Chang, Michael Ringgaard, and Hiyan Alshawi. 2010. Uptraining for accurate deterministic question parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 705–713. [find](https://scholar.google.com/scholar?q=Slav+Petrov%2C+Pi%2DChuan+Chang%2C+Michael+Ringgaard%2C+and+Hiyan+Alshawi%2E+2010%2E+Uptraining+for+accurate+deterministic+question+parsing%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Langua)
29. Marion Potet, Emmanuelle Esperança-Rodier, Laurent Besacier, and Hervé Blanchon. 2012. Collection of a large database of French-English SMT output corrections. In LREC, pages 4043–4048. [find](https://scholar.google.com/scholar?q=Marion+Potet%2C+Emmanuelle+Esperanc%CC%A7a%2DRodier%2C+Laurent+Besacier%2C+and+Herve%CC%81+Blanchon%2E+2012%2E+Collection+of+a+large+database+of+French%2DEnglish+SMT+output+corrections%2E+In+LREC%2C+pages+4043%E2%80%934048%2E)
30. Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics, 34(2):257–287. [find](https://scholar.google.com/scholar?q=Vasin+Punyakanok%2C+Dan+Roth%2C+and+Wen%2Dtau+Yih%2E+2008%2E+The+importance+of+syntactic+parsing+and+inference+in+semantic+role+labeling%2E+Computational+Linguistics%2C+34%282%29%3A257%E2%80%93287%2E)
31. Marwa Ragheb and Markus Dickinson. 2012. Defining syntax for learner language annotation. In COLING (Posters), pages 965–974. [find](https://scholar.google.com/scholar?q=Marwa+Ragheb+and+Markus+Dickinson%2E+2012%2E+Defining+syntax+for+learner+language+annotation%2E+In+COLING+%28Posters%29%2C+pages+965%E2%80%93974%2E)
32. Mohammad Sadegh Rasooli and Joel Tetreault. 2015. Yara parser: A fast and accurate dependency parser. arXiv preprint arXiv:1503.06733. [↗](https://arxiv.org/abs/1503.06733)
33. Brian Roark, Mary Harper, Eugene Charniak, Bonnie Dorr, Mark Johnson, Jeremy G Kahn, Yang Liu, Mari Ostendorf, John Hale, Anna Krasnyanskaya, et al. 2006. Sparseval: Evaluation metrics for parsing speech. In LREC. [find](https://scholar.google.com/scholar?q=Brian+Roark%2C+Mary+Harper%2C+Eugene+Charniak%2C+Bonnie+Dorr%2C+Mark+Johnson%2C+Jeremy+G+Kahn%2C+Yang+Liu%2C+Mari+Ostendorf%2C+John+Hale%2C+Anna+Krasnyanskaya%2C+et+al%2E+2006%2E+Sparseval%3A+Evaluation+metrics+for+parsing+spe)
34. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In NAACL, pages 173–180. [find](https://scholar.google.com/scholar?q=Kristina+Toutanova%2C+Dan+Klein%2C+Christopher+D+Manning%2C+and+Yoram+Singer%2E+2003%2E+Feature%2Drich+part%2Dof%2Dspeech+tagging+with+a+cyclic+dependency+network%2E+In+NAACL%2C+pages+173%E2%80%93180%2E)
35. Huichao Xue and Rebecca Hwa. 2014. Redundancy detection in esl writings. In EACL, pages 683–691. [find](https://scholar.google.com/scholar?q=Huichao+Xue+and+Rebecca+Hwa%2E+2014%2E+Redundancy+detection+in+esl+writings%2E+In+EACL%2C+pages+683%E2%80%93691%2E)
36. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 180–189. [find](https://scholar.google.com/scholar?q=Helen+Yannakoudakis%2C+Ted+Briscoe%2C+and+Ben+Medlock%2E+2011%2E+A+new+dataset+and+method+for+automatically+grading+ESOL+texts%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+L)