[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/aclbulk-2007-clark-formalism-independent-parser-evaluation-ccg/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [English](https://md.mova.elamur.ai/free/en/docs/papers/en/) / Formalism-Independent Parser Evaluation with CCG and DepBank

# Formalism-Independent Parser Evaluation with CCG and DepBank

**Authors:** [Stephen Clark](https://md.mova.elamur.ai/free/en/docs/papers/authors/stephen-clark/), James R. Curran · **Year:** 2007 · **Venue:** Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics **Link:** [aclanthology.org/P07-1032](https://aclanthology.org/P07-1032/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**55** citations](https://openalex.org/works?filter=cites:W2139117344) openalex · 2026-10-19[**24** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/aclbulk-2007-clark-formalism-independent-parser-evaluation-ccg/) of the 140673 papers whose bibliographies we read[Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics](https://aclanthology.org/volumes/P07-1/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/P07-1032/)
- [PDF](https://aclanthology.org/P07-1032.pdf)
- [OpenAlex](https://openalex.org/W2139117344)

**Authors (2)**

1. [Stephen Clark](https://openalex.org/A5057392864) [ORCID](https://orcid.org/0000-0002-2193-4705) · [Semantic Scholar](https://www.semanticscholar.org/author/144523372) · h 55 · 10787 citations · 406 works
2. James Curran [ORCID](https://orcid.org/0000-0001-9637-0484) · [Semantic Scholar](https://www.semanticscholar.org/author/1733593)

Citations and author metrics from openalex (2026-10-19); authors with the same name are not always told apart.

## What it is about

Parsers that use different formalisms produce different output, so their scores cannot be compared directly. The authors evaluate a CCG parser on DepBank, a set of grammatical relations that does not match the parser's own output. The main difficulty is the conversion from CCG output to DepBank relations, so they measure the conversion itself: parse the gold analyses, convert them, and compare. This gives an upper bound on what the parser can score. The CCG parser reaches an F-score of 81.9% on labelled dependencies against an upper bound of 84.8%, and beats the RASP parser by over 5% overall. The paper's message is to evaluate on a resource that was not used to develop the parser, and to report the conversion ceiling next to the score.

## Ideas

- I1. `en/src/convert.rs`: measure Mova's conversion ceiling by converting gold EWT trees to the `mova` dialect and back to UD and scoring against the original; the gap from 100 is conversion loss, and should be reported next to `en ud-eval` (two runs, byte-identical).
- I2. `tools/parsecmp`: report each witness score as a fraction of its own conversion ceiling, so that a converter loss is not counted as a parser error.
- I3. Independent resource: score `en` against ERG-derived dependencies from the ACE setup listed in [English parsers — local collection](https://md.mova.elamur.ai/free/en/docs/docs/parsers/) on 200 held-out sentences (test only), and compare the agreement with the EWT LAS.

## For Mova

Touches [20. Syntax: the Universal Dependencies frame](https://md.mova.elamur.ai/free/en/docs/languages/en/20-syntax-overview/). The conversion ceiling is a cheap check that the `mova` dialect does not lose information, which the converters must keep. Measure I1 first; it is the quickest of the three and needs no new data.

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{clark2007formalism,
  title = {{Formalism-Independent Parser Evaluation with CCG and DepBank}},
  author = {Clark, Stephen and Curran, James},
  booktitle = {Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics},
  year = {2007},
  url = {https://aclanthology.org/P07-1032/}
}
```

- [← previous How Bad is the Problem of PP-Attachment? A Comparison of English, German and Swedish](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-volk-how-bad-is-problem-pp/)
- [next → Semantic Composition with (Robust) Minimal Recursion Semantics](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2007-copestake-semantic-composition-robust-minimal-recursion/)
- [↑ back to English](https://md.mova.elamur.ai/free/en/docs/papers/en/)

## References

22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rens Bod. 2003. An efficient implementation of a new DOP model. In Proceedings of the 10th Meeting of the EACL, pages 19–26, Budapest, Hungary. [find](https://scholar.google.com/scholar?q=Rens+Bod%2E+2003%2E+An+efficient+implementation+of+a+new+DOP+model%2E+In+Proceedings+of+the+10th+Meeting+of+the+EACL%2C+pages+19%E2%80%9326%2C+Budapest%2C+Hungary%2E)
2. Ted Briscoe and John Carroll. 2006. Evaluating the accuracy of an unlexicalized statistical parser on the PARC DepBank. In Proceedings of the Poster Session of COLING/ACL-06, Sydney, Australia. [find](https://scholar.google.com/scholar?q=Ted+Briscoe+and+John+Carroll%2E+2006%2E+Evaluating+the+accuracy+of+an+unlexicalized+statistical+parser+on+the+PARC+DepBank%2E+In+Proceedings+of+the+Poster+Session+of+COLING%2FACL%2D06%2C+Sydney%2C+Australia%2E)
3. Ted Briscoe, John Carroll, and Rebecca Watson. 2006. The second release of the RASP system. In Proceedings of the Interactive Demo Session of COLING/ACL-06, Sydney, Australia. [find](https://scholar.google.com/scholar?q=Ted+Briscoe%2C+John+Carroll%2C+and+Rebecca+Watson%2E+2006%2E+The+second+release+of+the+RASP+system%2E+In+Proceedings+of+the+Interactive+Demo+Session+of+COLING%2FACL%2D06%2C+Sydney%2C+Australia%2E)
4. Sabine Buchholz, Jorn Veenstra, and Walter Daelemans. 1999. Cascaded grammatical relation assignment. In Proceedings of EMNLP/VLC-99, pages 239–246, University of Maryland, June 21-22. [find](https://scholar.google.com/scholar?q=Sabine+Buchholz%2C+Jorn+Veenstra%2C+and+Walter+Daelemans%2E+1999%2E+Cascaded+grammatical+relation+assignment%2E+In+Proceedings+of+EMNLP%2FVLC%2D99%2C+pages+239%E2%80%93246%2C+University+of+Maryland%2C+June+21%2D22%2E)
5. A. Cahill, M. Burke, R. O’Donovan, J. van Genabith, and A. Way. 2004. Long-distance dependency resolution in automatically acquired wide-coverage PCFG-based LFG approximations. In Proceedings of the 42nd Meeting of the ACL, pages 320–327, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=A%2E+Cahill%2C+M%2E+Burke%2C+R%2E+O%E2%80%99Donovan%2C+J%2E+van+Genabith%2C+and+A%2E+Way%2E+2004%2E+Long%2Ddistance+dependency+resolution+in+automatically+acquired+wide%2Dcoverage+PCFG%2Dbased+LFG+approximations%2E+In+Proceedings+of+the+4)
6. John Carroll, Ted Briscoe, and Antonio Sanfilippo. 1998. Parser evaluation: a survey and a new proposal. In Proceedings of the 1st LREC Conference, pages 447–454, Granada, Spain. [find](https://scholar.google.com/scholar?q=John+Carroll%2C+Ted+Briscoe%2C+and+Antonio+Sanfilippo%2E+1998%2E+Parser+evaluation%3A+a+survey+and+a+new+proposal%2E+In+Proceedings+of+the+1st+LREC+Conference%2C+pages+447%E2%80%93454%2C+Granada%2C+Spain%2E)
7. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine nbest parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the ACL, University of Michigan, Ann Arbor. [find](https://scholar.google.com/scholar?q=Eugene+Charniak+and+Mark+Johnson%2E+2005%2E+Coarse%2Dto%2Dfine+nbest+parsing+and+maxent+discriminative+reranking%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+ACL%2C+University+of+Michigan%2C+Ann+Arbor%2E)
8. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Meeting of the NAACL, pages 132– 139, Seattle, WA. [find](https://scholar.google.com/scholar?q=Eugene+Charniak%2E+2000%2E+A+maximum%2Dentropy%2Dinspired+parser%2E+In+Proceedings+of+the+1st+Meeting+of+the+NAACL%2C+pages+132%E2%80%93+139%2C+Seattle%2C+WA%2E)
9. Stephen Clark and James R. Curran. 2004a. The importance of supertagging for wide-coverage CCG parsing. In Proceedings of COLING-04, pages 282–288, Geneva, Switzerland. [find](https://scholar.google.com/scholar?q=Stephen+Clark+and+James+R%2E+Curran%2E+2004a%2E+The+importance+of+supertagging+for+wide%2Dcoverage+CCG+parsing%2E+In+Proceedings+of+COLING%2D04%2C+pages+282%E2%80%93288%2C+Geneva%2C+Switzerland%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2004-clark-importance-supertagging-wide-coverage-ccg/)
10. Stephen Clark and James R. Curran. 2004b. Parsing the WSJ using CCG and log-linear models. In Proceedings of the 42nd Meeting of the ACL, pages 104–111, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Stephen+Clark+and+James+R%2E+Curran%2E+2004b%2E+Parsing+the+WSJ+using+CCG+and+log%2Dlinear+models%2E+In+Proceedings+of+the+42nd+Meeting+of+the+ACL%2C+pages+104%E2%80%93111%2C+Barcelona%2C+Spain%2E)
11. Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational Linguistics, 29(4):589–637. [find](https://scholar.google.com/scholar?q=Michael+Collins%2E+2003%2E+Head%2Ddriven+statistical+models+for+natural+language+parsing%2E+Computational+Linguistics%2C+29%284%29%3A589%E2%80%93637%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/cl-2003-collins-head-driven-statistical-models-natural/)
12. James R. Curran and Stephen Clark. 2003. Investigating GIS and smoothing for maximum entropy taggers. In Proceedings of the 10th Meeting of the EACL, pages 91–98, Budapest, Hungary. [find](https://scholar.google.com/scholar?q=James+R%2E+Curran+and+Stephen+Clark%2E+2003%2E+Investigating+GIS+and+smoothing+for+maximum+entropy+taggers%2E+In+Proceedings+of+the+10th+Meeting+of+the+EACL%2C+pages+91%E2%80%9398%2C+Budapest%2C+Hungary%2E)
13. Julia Hockenmaier and Mark Steedman. 2002. Generative models for statistical parsing with Combinatory Categorial Grammar. In Proceedings of the 40th Meeting of the ACL, pages 335–342, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=Julia+Hockenmaier+and+Mark+Steedman%2E+2002%2E+Generative+models+for+statistical+parsing+with+Combinatory+Categorial+Grammar%2E+In+Proceedings+of+the+40th+Meeting+of+the+ACL%2C+pages+335%E2%80%93342%2C+Philadelphia%2C+PA)
14. Julia Hockenmaier. 2003. Data and Models for Statistical Parsing with Combinatory Categorial Grammar. Ph.D. thesis, University of Edinburgh. [find](https://scholar.google.com/scholar?q=Julia+Hockenmaier%2E+2003%2E+Data+and+Models+for+Statistical+Parsing+with+Combinatory+Categorial+Grammar%2E+Ph%2ED%2E+thesis%2C+University+of+Edinburgh%2E)
15. Ron Kaplan, Stefan Riezler, Tracy H. King, John T. Maxwell III, Alexander Vasserman, and Richard Crouch. 2004. Speed and accuracy in shallow and deep stochastic parsing. In Proceedings of the HLT Conference and the 4th NAACL Meeting (HLT-NAACL’04), Boston, MA. [find](https://scholar.google.com/scholar?q=Ron+Kaplan%2C+Stefan+Riezler%2C+Tracy+H%2E+King%2C+John+T%2E+Maxwell+III%2C+Alexander+Vasserman%2C+and+Richard+Crouch%2E+2004%2E+Speed+and+accuracy+in+shallow+and+deep+stochastic+parsing%2E+In+Proceedings+of+the+HLT+Conf)
16. Tracy H. King, Richard Crouch, Stefan Riezler, Mary Dalrymple, and Ronald M. Kaplan. 2003. The PARC 700 Dependency Bank. In Proceedings of the LINC-03 Workshop, Budapest, Hungary. [find](https://scholar.google.com/scholar?q=Tracy+H%2E+King%2C+Richard+Crouch%2C+Stefan+Riezler%2C+Mary+Dalrymple%2C+and+Ronald+M%2E+Kaplan%2E+2003%2E+The+PARC+700+Dependency+Bank%2E+In+Proceedings+of+the+LINC%2D03+Workshop%2C+Budapest%2C+Hungary%2E)
17. Robert Malouf and Gertjan van Noord. 2004. Wide coverage parsing with stochastic attribute value grammars. In Proceedings of the IJCNLP-04 Workshop: Beyond shallow analyses - Formalisms and statistical modeling for deep analyses, Hainan Island, China. [find](https://scholar.google.com/scholar?q=Robert+Malouf+and+Gertjan+van+Noord%2E+2004%2E+Wide+coverage+parsing+with+stochastic+attribute+value+grammars%2E+In+Proceedings+of+the+IJCNLP%2D04+Workshop%3A+Beyond+shallow+analyses+%2D+Formalisms+and+statistica)
18. Joakim Nivre and Mario Scholz. 2004. Deterministic dependency parsing of English text. In Proceedings of COLING-2004, pages 64–70, Geneva, Switzerland. [find](https://scholar.google.com/scholar?q=Joakim+Nivre+and+Mario+Scholz%2E+2004%2E+Deterministic+dependency+parsing+of+English+text%2E+In+Proceedings+of+COLING%2D2004%2C+pages+64%E2%80%9370%2C+Geneva%2C+Switzerland%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2004-nivre-deterministic-dependency-parsing-english-text/)
19. Judita Preiss. 2003. Using grammatical relations to compare parsers. In Proceedings of the 10th Meeting of the EACL, pages 291–298, Budapest, Hungary. [find](https://scholar.google.com/scholar?q=Judita+Preiss%2E+2003%2E+Using+grammatical+relations+to+compare+parsers%2E+In+Proceedings+of+the+10th+Meeting+of+the+EACL%2C+pages+291%E2%80%93298%2C+Budapest%2C+Hungary%2E)
20. Anoop Sarkar and Aravind Joshi. 2003. Tree-adjoining grammars and its application to statistical parsing. In Rens Bod, Remko Scha, and Khalil Sima’an, editors, Data-oriented parsing. CSLI. [find](https://scholar.google.com/scholar?q=Anoop+Sarkar+and+Aravind+Joshi%2E+2003%2E+Tree%2Dadjoining+grammars+and+its+application+to+statistical+parsing%2E+In+Rens+Bod%2C+Remko+Scha%2C+and+Khalil+Sima%E2%80%99an%2C+editors%2C+Data%2Doriented+parsing%2E+CSLI%2E)
21. Mark Steedman. 2000. The Syntactic Process. The MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Mark+Steedman%2E+2000%2E+The+Syntactic+Process%2E+The+MIT+Press%2C+Cambridge%2C+MA%2E)
22. Kristina Toutanova, Christopher Manning, Stuart Shieber, Dan Flickinger, and Stephan Oepen. 2002. Parse disambiguation for a rich HPSG grammar. In Proceedings of the First Workshop on Treebanks and Linguistic Theories, pages 253–263, Sozopol, Bulgaria. [find](https://scholar.google.com/scholar?q=Kristina+Toutanova%2C+Christopher+Manning%2C+Stuart+Shieber%2C+Dan+Flickinger%2C+and+Stephan+Oepen%2E+2002%2E+Parse+disambiguation+for+a+rich+HPSG+grammar%2E+In+Proceedings+of+the+First+Workshop+on+Treebanks+and+Li)