[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/lingbulk-2004-foster-parsing-ungrammatical-input-evaluation-procedure/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Parsing Ungrammatical Input: An Evaluation Procedure

# Parsing Ungrammatical Input: An Evaluation Procedure

**Authors:** [Jennifer Foster](https://md.mova.elamur.ai/free/en/docs/papers/authors/jennifer-foster/) · **Year:** 2004 · **Venue:** Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004) **Link:** [aclanthology.org/L04-1452](https://aclanthology.org/L04-1452/) · **Licence:** copyrighted (ELRA, all rights reserved; listed as [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) by the ACL Anthology) — summary only

[**17** citations](https://openalex.org/works?filter=cites:W2400296909) openalex · 2026-10-19[**5** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/lingbulk-2004-foster-parsing-ungrammatical-input-evaluation-procedure/) of the 140673 papers whose bibliographies we read[Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)](https://aclanthology.org/volumes/L04-1/)[licence](https://creativecommons.org/licenses/by-nc/4.0/)

- [Publisher's page](https://aclanthology.org/L04-1452/)
- [PDF](https://aclanthology.org/L04-1452.pdf)
- [DOI 10.63317/3vjm8zsqfw8p](https://doi.org/10.63317/3vjm8zsqfw8p)
- [OpenAlex](https://openalex.org/W2400296909)

**Authors (1)**

1. [Foster, Jennifer](https://openalex.org/A5066497689) [ORCID](https://orcid.org/0000-0002-7789-4853) · [Semantic Scholar](https://www.semanticscholar.org/author/144116340) · h 23 · 2298 citations · 125 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

The question is how well a parser handles an ungrammatical sentence. The method needs two parallel corpora: ungrammatical sentences and their corrected versions. Each ungrammatical sentence is parsed, and its parse is scored against the parse of each correction produced by the same parser; the best score counts. The source is a 20,000-word corpus of ungrammatical English from newspapers, email, academic papers and websites, where 20% of sentences have more than one correction. Charniak's 2000 parser reaches 91% precision and 91% recall, with 32% complete matches against a correction and 16% of sentences below 75%. Incorrect word forms make up 44% of the corpus, missing words 28% and extra words 15%.

## Ideas

- I1. `en parse-eval`: for a sentence with several corrections, score it against the best one and print the chosen correction. Measure on a small set of sentences that we write and correct ourselves, with two or more corrections for some of them.
- I2. Error-type breakdown: tag each test sentence as word form, missing word, extra word, composite or multiple errors, then report LAS and complete-match rate per type. Measure the complete-match rate per type on our own set.
- I3. Extra-word rule in `en/src/expert.rs`: flag a `to` that follows a verb and does not head an infinitive, with the reason written by `en explain`. Measure the precision of the flag and the LAS change on the own set.
- I4. Negative control: score the same parser on a clean set and on its corrupted copy. If the drop is zero, the gate is broken and the test must be fixed before any claim.

## For Mova

The chapters are [22. Phrases: noun, adjective, adverb and prepositional phrases](https://md.mova.elamur.ai/free/en/docs/languages/en/22-phrases/) and `37-register-and-variation.md`. The code is `en/src/parse.rs`, `en/src/expert.rs`, `en/src/explain.rs` and the `en parse-eval` command. The paper's corpus is not in Mova's data, and its licence is NC, so it cannot be built into anything. Any test set must be written by us and used for measurement only, and the paper's figures belong to a different parser.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.63317/3vjm8zsqfw8p](https://doi.org/10.63317/3vjm8zsqfw8p)

```
@inproceedings{foster2004parsing,
  title = {{Parsing Ungrammatical Input: An Evaluation Procedure}},
  author = {Foster, Jennifer},
  booktitle = {Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)},
  year = {2004},
  doi = {10.63317/3vjm8zsqfw8p},
  url = {https://doi.org/10.63317/3vjm8zsqfw8p}
}
```

- [← previous Multiword expressions: linguistic precision and reusability](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2002-copestake-multiword-expressions-linguistic-precision-reusability/)
- [next → A Labelled Corpus for Prepositional Phrase Attachment](https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2004-mitchell-labelled-corpus-prepositional-phrase-attachment/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

7 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Black, E., S. Abney, D. Flickinger, C. Gdaniec, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J.Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski, 1991. A procedure for quantitatively comparing the syntactic coverage of english grammars. In Proceedings of th… [find](https://scholar.google.com/scholar?q=Black%2C+E%2E%2C+S%2E+Abney%2C+D%2E+Flickinger%2C+C%2E+Gdaniec%2C+R%2E+Grishman%2C+P%2E+Harrison%2C+D%2E+Hindle%2C+R%2E+Ingria%2C+F%2E+Jelinek%2C+J%2EKlavans%2C+M%2E+Liberman%2C+M%2E+Marcus%2C+S%2E+Roukos%2C+B%2E+Santorini%2C+and+T%2E+Strzalkowski%2C+1991%2E+A+pro)
2. Charniak, 2000. A maximum-entropy-inspired parser. In Proceedings of the NAACL-2000. [find](https://scholar.google.com/scholar?q=Charniak%2C+2000%2E+A+maximum%2Dentropy%2Dinspired+parser%2E+In+Proceedings+of+the+NAACL%2D2000%2E)
3. Collins, Michael, 1997. Three generative lexicalized models for statistical parsing. In Proceedings of the 35th ACL. Madrid. [find](https://scholar.google.com/scholar?q=Collins%2C+Michael%2C+1997%2E+Three+generative+lexicalized+models+for+statistical+parsing%2E+In+Proceedings+of+the+35th+ACL%2E+Madrid%2E)
4. Copestake, Ann, 2002. Implementing Typed Feature Structure Grammars. CSLI Lecture Notes. Cambridge: Cambridge University Press. [find](https://scholar.google.com/scholar?q=Copestake%2C+Ann%2C+2002%2E+Implementing+Typed+Feature+Structure+Grammars%2E+CSLI+Lecture+Notes%2E+Cambridge%3A+Cambridge+University+Press%2E)
5. Foster, Jennifer and Carl Vogel, 2004. Good reasons for noting bad grammar: Constructing a corpus of ungrammatical language. In Stephan Kepser and Marga Reis (eds.), Pre-Proceedings of the International Conference on Linguistic Evidence: Empirical, Theoretical and Computational Perspectives. Tübing… [find](https://scholar.google.com/scholar?q=Foster%2C+Jennifer+and+Carl+Vogel%2C+2004%2E+Good+reasons+for+noting+bad+grammar%3A+Constructing+a+corpus+of+ungrammatical+language%2E+In+Stephan+Kepser+and+Marga+Reis+%28eds%2E%29%2C+Pre%2DProceedings+of+the+Internation)
6. Marcus, Mitchell P., Beatrice Santorini, and Mary Ann Marcinkiewicz, 1993. Building a large annotated corpus of english: the penn treebank. Computational Linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=Marcus%2C+Mitchell+P%2E%2C+Beatrice+Santorini%2C+and+Mary+Ann+Marcinkiewicz%2C+1993%2E+Building+a+large+annotated+corpus+of+english%3A+the+penn+treebank%2E+Computational+Linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
7. Musillo, Gabriele and Khalil Sima’an, 2002. Towards comparing parsers from different linguistic frameworks: An information theoretic approach. In Proceedings of the ”Beyond Parseval - Towards Improved Evaluation Measures for Parsing Systems” Workshop, 3rd LREC. Las Palmas, Gran Canaria. [find](https://scholar.google.com/scholar?q=Musillo%2C+Gabriele+and+Khalil+Sima%E2%80%99an%2C+2002%2E+Towards+comparing+parsers+from+different+linguistic+frameworks%3A+An+information+theoretic+approach%2E+In+Proceedings+of+the+%E2%80%9DBeyond+Parseval+%2D+Towards+Improved)