[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2005-flickinger-dimensions-deep-grammar-validation/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Dimensions of Deep Grammar Validation

# Dimensions of Deep Grammar Validation

**Authors:** [Dan Flickinger](https://md.mova.elamur.ai/free/en/docs/papers/authors/dan-flickinger/) · **Year:** 2005 · **Venue:** Proceedings of the Australasian Language Technology Workshop 2005 **Link:** [aclanthology.org/U05-1001](https://aclanthology.org/U05-1001/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**0** citations](https://openalex.org/works?filter=cites:W2251010909) openalex · 2026-10-19[Proceedings of the Australasian Language Technology Workshop 2005](https://aclanthology.org/volumes/U05-1/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/U05-1001/)
- [PDF](https://aclanthology.org/U05-1001.pdf)
- [OpenAlex](https://openalex.org/W2251010909)

**Authors (1)**

1. [Flickinger, Dan](https://openalex.org/A5089106912) [ORCID](https://orcid.org/0000-0001-7959-999X) · [Semantic Scholar](https://www.semanticscholar.org/author/3209288) · h 32 · 5154 citations · 155 works

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

The paper sets out a method for validating a large deep grammar at each release, using the English Resource Grammar as the example. It names ten dimensions. Four are the main tensions: robustness, precision, efficiency and stability of the mappings on test suites. The others cover several processing systems, bi-directional use, grammar configurations, disambiguation, applications, and the external interface specifications, such as the list of semantic predicates. The tools are growing test suites, both hand-built and corpus-based, and a profiler that stores competence and performance data for each item. Gold treebanks from the suites measure each new revision. The paper gives no numbers; it describes one release cycle.

## Ideas

- I1. `en/src/udreg.rs`: extend the release gate with one line per dimension: coverage on EWT test, LAS, seconds, the number of changed analyses since the last release, and the version of the tag inventory. The gate must turn red on a seeded regression.
- I2. Stability metric: count how many EWT dev trees change between two `en` builds, and print it beside LAS in `en parse-eval`.
- I3. Configuration check: run `en ud-check` with and without `en/src/ctx.rs` and report the difference in LAS on EWT dev, as a configuration dimension.

## For Mova

The code is `en/src/udreg.rs` and the commands are `en ud-check` and `en parse-eval`; the chapter is [37. Register and variation](https://md.mova.elamur.ai/free/en/docs/languages/en/37-register-and-variation/). The paper has no numbers and describes an HPSG grammar. Borrow the checklist only, and measure on EWT.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{flickinger2005dimensions,
  title = {{Dimensions of Deep Grammar Validation}},
  author = {Flickinger, Dan},
  booktitle = {Proceedings of the Australasian Language Technology Workshop 2005},
  year = {2005},
  url = {https://aclanthology.org/U05-1001/}
}
```

- [← previous Supersense Tagging of Unknown Nouns using Semantic Similarity](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-curran-supersense-tagging-unknown-nouns-semantic/)
- [next → Automatic Interpretation of Noun Compounds Using WordNet Similarity](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-kim-automatic-interpretation-noun-compounds-wordnet/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

15 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Valerie Barr and Judith Klavans. 2001. Verification and validation of language processing systems: Is it evaluation? Proceedings of the Workshop on Evaluation Methodologies for Language and Dialogue Systems, ACL2001. [find](https://scholar.google.com/scholar?q=Valerie+Barr+and+Judith+Klavans%2E+2001%2E+Verification+and+validation+of+language+processing+systems%3A+Is+it+evaluation%3F+Proceedings+of+the+Workshop+on+Evaluation+Methodologies+for+Language+and+Dialogue+S)
2. Valerie Barr and Ellen Siefring. 2004. Verification of lexicalized tree adjoining grammars. Online Proceedings of the Seventh International Workshop on TAG and Related Formalisms. [find](https://scholar.google.com/scholar?q=Valerie+Barr+and+Ellen+Siefring%2E+2004%2E+Verification+of+lexicalized+tree+adjoining+grammars%2E+Online+Proceedings+of+the+Seventh+International+Workshop+on+TAG+and+Related+Formalisms%2E)
3. Norbert Broeker. 2000. The use of instrumentation in grammar engineering. Proceedings of COLING 2000. [find](https://scholar.google.com/scholar?q=Norbert+Broeker%2E+2000%2E+The+use+of+instrumentation+in+grammar+engineering%2E+Proceedings+of+COLING+2000%2E)
4. Ulrich Callmeier. 2000. PET — A platform for experimentation with efficient HPSG processing techniques. Natural Language Engineering, 6 (1) (Special Issue on Efficient Processing with HPSG):99 – 108. [find](https://scholar.google.com/scholar?q=Ulrich+Callmeier%2E+2000%2E+PET+%E2%80%94+A+platform+for+experimentation+with+efficient+HPSG+processing+techniques%2E+Natural+Language+Engineering%2C+6+%281%29+%28Special+Issue+on+Efficient+Processing+with+HPSG%29%3A99+%E2%80%93+108%2E)
5. Ann Copestake, Dan Flickinger, Ivan A. Sag, and Carl Pollard. 2006. Minimal Recursion Semantics. An introduction. Research on Language and Computation. (to appear). [find](https://scholar.google.com/scholar?q=Ann+Copestake%2C+Dan+Flickinger%2C+Ivan+A%2E+Sag%2C+and+Carl+Pollard%2E+2006%2E+Minimal+Recursion+Semantics%2E+An+introduction%2E+Research+on+Language+and+Computation%2E+%28to+appear%29%2E)
6. Ann Copestake. 2002. Implementing Typed Feature Structure Grammars. CSLI Publications, Stanford, CA. [find](https://scholar.google.com/scholar?q=Ann+Copestake%2E+2002%2E+Implementing+Typed+Feature+Structure+Grammars%2E+CSLI+Publications%2C+Stanford%2C+CA%2E)
7. Marc Dymetman and Pierre Isabelle. 1988. Reversible logic grammars for machine translation. Proceedings of the 2nd International Conference on Theoretical and Methodological Issues in the Machine Translation of Natural Languages. [find](https://scholar.google.com/scholar?q=Marc+Dymetman+and+Pierre+Isabelle%2E+1988%2E+Reversible+logic+grammars+for+machine+translation%2E+Proceedings+of+the+2nd+International+Conference+on+Theoretical+and+Methodological+Issues+in+the+Machine+Tran)
8. EAGLES. 1996. Evaluation of Natural Language Processing Systems. Final Report EAG-EWG-PR.2. EU Report. [find](https://scholar.google.com/scholar?q=EAGLES%2E+1996%2E+Evaluation+of+Natural+Language+Processing+Systems%2E+Final+Report+EAG%2DEWG%2DPR%2E2%2E+EU+Report%2E)
9. Dan Flickinger, Jan Tore Lønning, Helge Dyvik, Stephan Oepen, and Francis Bond. 2005. SEM-I rational MT: Enriching deep grammars with a semantic interface for scalable machine translation. In Machine Translation Summit X, Phuket. (to appear). [find](https://scholar.google.com/scholar?q=Dan+Flickinger%2C+Jan+Tore+L%C3%B8nning%2C+Helge+Dyvik%2C+Stephan+Oepen%2C+and+Francis+Bond%2E+2005%2E+SEM%2DI+rational+MT%3A+Enriching+deep+grammars+with+a+semantic+interface+for+scalable+machine+translation%2E+In+Machine+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/flickinger-2005-sem-i-rational-mt-enriching/)
10. Dan Flickinger. 2000. On building a more efficient grammar by exploiting types. Natural Language Engineering, 6 (1):15 – 28. [find](https://scholar.google.com/scholar?q=Dan+Flickinger%2E+2000%2E+On+building+a+more+efficient+grammar+by+exploiting+types%2E+Natural+Language+Engineering%2C+6+%281%29%3A15+%E2%80%93+28%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/parsers-2000-flickinger-building-more-efficient-grammar/)
11. Lynette Hirschman and Henry Thompson. 1998. Overview of evaluation in speech and natural language processing. In G. Varile and A. Zampolli, editors, Survey of the State of the Art in Human Language Technology. Cambridge University Press, New York. [find](https://scholar.google.com/scholar?q=Lynette+Hirschman+and+Henry+Thompson%2E+1998%2E+Overview+of+evaluation+in+speech+and+natural+language+processing%2E+In+G%2E+Varile+and+A%2E+Zampolli%2C+editors%2C+Survey+of+the+State+of+the+Art+in+Human+Language+Te)
12. Jan Tore Lønning, Stephan Oepen, Dorothee Beermann, Lars Hellan, John Carroll, Helge Dyvik, Dan Flickinger, Janne Bondi Johannessen, Paul Meurer, Torbjørn Nordgård, Victoria Rosén, and Erik Velldal. 2004. LO-GON. A Norwegian MT effort. In Proceedings of the Workshop in Recent Advances in Scandinav… [find](https://scholar.google.com/scholar?q=Jan+Tore+L%C3%B8nning%2C+Stephan+Oepen%2C+Dorothee+Beermann%2C+Lars+Hellan%2C+John+Carroll%2C+Helge+Dyvik%2C+Dan+Flickinger%2C+Janne+Bondi+Johannessen%2C+Paul+Meurer%2C+Torbj%C3%B8rn+Nordga%CC%8Ard%2C+Victoria+Rose%CC%81n%2C+and+Erik+Velldal%2E)
13. Stephan Oepen and Dan Flickinger. 1998. Towards systematic grammar profiling. Test suite technology ten years after. Journal of Computer Speech and Language, 12 (4):411 – 436. [find](https://scholar.google.com/scholar?q=Stephan+Oepen+and+Dan+Flickinger%2E+1998%2E+Towards+systematic+grammar+profiling%2E+Test+suite+technology+ten+years+after%2E+Journal+of+Computer+Speech+and+Language%2C+12+%284%29%3A411+%E2%80%93+436%2E)
14. Stephan Oepen, Daniel Flickinger, Kristina Toutanova, and Christopher D. Manning. 2004. LinGO Redwoods. A rich and dynamic treebank for HPSG. Journal of Language and Computation. [find](https://scholar.google.com/scholar?q=Stephan+Oepen%2C+Daniel+Flickinger%2C+Kristina+Toutanova%2C+and+Christopher+D%2E+Manning%2E+2004%2E+LinGO+Redwoods%2E+A+rich+and+dynamic+treebank+for+HPSG%2E+Journal+of+Language+and+Computation%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/enprag-2004-oepen-lingo-redwoods-rich-dynamic-treebank/)
15. Karen Sparck Jones and Julia Galliers. 1998. Evaluating Natural Language Processing Systems. Springer-Verlag, Berlin. [find](https://scholar.google.com/scholar?q=Karen+Sparck+Jones+and+Julia+Galliers%2E+1998%2E+Evaluating+Natural+Language+Processing+Systems%2E+Springer%2DVerlag%2C+Berlin%2E)