[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/isabelle-2017-challenge-set-approach-evaluating-machine/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) / A Challenge Set Approach to Evaluating Machine Translation

# A Challenge Set Approach to Evaluating Machine Translation

**Authors:** [Pierre Isabelle](https://md.mova.elamur.ai/free/en/docs/papers/authors/pierre-isabelle/), [Colin Cherry](https://md.mova.elamur.ai/free/en/docs/papers/authors/colin-cherry/), [George Foster](https://md.mova.elamur.ai/free/en/docs/papers/authors/george-foster/) · **Year:** 2017 · **Venue:** Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing **Link:** [aclanthology.org/D17-1263](https://aclanthology.org/D17-1263/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**161** citations](https://openalex.org/works?filter=cites:W2608615602) openalex · 2026-10-18[**111** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/isabelle-2017-challenge-set-approach-evaluating-machine/) of the 140673 papers whose bibliographies we read[Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing](https://aclanthology.org/volumes/D17-1/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/D17-1263/)
- [PDF](https://aclanthology.org/D17-1263.pdf)
- [DOI 10.18653/v1/d17-1263](https://doi.org/10.18653/v1/d17-1263)
- [OpenAlex](https://openalex.org/W2608615602)

**Authors (3)**

1. [Pierre Isabelle](https://openalex.org/A5109464669) h 19 · 3543 citations · 62 works
2. [Colin Cherry](https://openalex.org/A5048302740) h 44 · 8948 citations · 208 works
3. [George Foster](https://openalex.org/A5090694771) h 31 · 5041 citations · 187 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The paper proposes challenge sets: small, hand-written sets of English to French sentences, each built to test one structural divergence between the two languages, for example morphology that exposes agreement, stranded prepositions, double-object verbs, subcategorisation frames, idioms and WH-movement. Each item has a yes or no question for a human judge. The authors compare phrase-based and neural systems of 2017. Neural systems beat phrase-based ones on almost every category, idioms being the exception. Some rules are learned by the neural system in simple cases and still fail in longer sentences, which the authors report as brittle generalisation. The method is manual, the sets are small, and native speakers are needed to repeat it.

## Ideas

- I1. A challenge set for the Bible translator: one set per divergence between Hebrew and English (verb-initial order, construct chains, *thou* and *ye* by number and person, *hath*, negation particles, idioms). Each item has a fixed target output, and the score is the exact-match rate per class, not BLEU.
- I2. Turn the yes or no questions into rules a script can check on the output (for example, the subject precedes the verb). The set then needs no human judge, and it runs in the test suite.
- I3. A brittleness item: a pattern that passes in the short form and fails in a longer verse, to test the frame and encoder path rather than the lexicon.

## For Mova

The places are `tools/translate` test data, `tools/macula` for the source trees and `en/src/morph.rs` for the forms. The items are small and can live in the repo as data. The set is a dev instrument; any score on it must be read against the held-out rule and the two-run check.

Topics: [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/d17-1263](https://doi.org/10.18653/v1/d17-1263)

```
@inproceedings{isabelle2017challenge,
  title = {{A Challenge Set Approach to Evaluating Machine Translation}},
  author = {Isabelle, Pierre and Cherry, Colin and Foster, George},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1263},
  url = {https://doi.org/10.18653/v1/d17-1263}
}
```

- [← previous Towards Reversible MT Systems](https://md.mova.elamur.ai/free/en/docs/papers/isabelle-1989-reversible-mt-systems/)
- [next → T-STAR: Truthful Style Transfer using AMR Graph as Intermediate Representation](https://md.mova.elamur.ai/free/en/docs/papers/jangra-2022-t-star-truthful-style-transfer/)
- [↑ back to Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/)

## References

16 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the Third International Conference on Learning Representations (ICLR). San Diego, USA. http://arxiv.org/abs/1409.0473. [↗](https://arxiv.org/abs/1409.0473)
2. Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus phrase-based machine translation quality: a case study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 257–267. https://aclweb.org/anthology/D16-1025. [↗](https://aclanthology.org/D16-1025/)
3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 131–198. http://www.aclweb.org/anthology/W16-2301. [↗](https://aclanthology.org/W16-2301/)
4. Colin Cherry. 2013. Improved reordering for phrase-based translation using sparse features. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Atlanta, Georgia, pages 22–31. http://www.aclweb.org/anthology/N13-1003. [↗](https://aclanthology.org/N13-1003/)
5. Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Montréal, Canada, pages 427–436. http://www.aclweb.org/anthology/N12-1047. [↗](https://aclanthology.org/N12-1047/)
6. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 1724–1734. http://www.aclweb.org/anthology/D14-1179. [↗](https://aclanthology.org/D14-1179/)
7. Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Baltimore, Maryland, pages 1370–1380. http://www.aclweb.org/anthology/P14-1129. [↗](https://aclanthology.org/P14-1129/)
8. Bonnie J. Dorr. 1994. Machine translation divergences: a formal description and proposed solution. Computational Linguistics 20:4. http://aclweb.org/anthology/J/J94/J94-4004.pdf. [↗](https://aclanthology.org/J94-4004/) [our card](https://md.mova.elamur.ai/free/en/docs/papers/dorr-1994-machine-translation-divergences-formal-description/)
9. Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and Stephan Vogel. 2016. QCRI machine translation systems for IWSLT 16. In Proceedings of the 13th International Workshop on Spoken Language Translation (IWSLT). Seattle, Washington. [https://workshop2016.iwslt.org/downloads/qcri-machine-translation.pdf](https://workshop2016.iwslt.org/downloads/qcri-machine-translation.pdf). [↗](https://workshop2016.iwslt.org/downloads/qcri-machine-translation.pdf)
10. Michel Galley and Christopher D. Manning. 2008. A simple and effective hierarchical phrase reordering model. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, Hawaii, pages 848–856. http://www.aclweb.org/anthology/D08-1089. [↗](https://aclanthology.org/D08-1089/)
11. Liang Huang and David Chiang. 2007. Forest rescoring: Faster decoding with integrated language models. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. Association for Computational Linguistics, Prague, Czech Republic, pages 144–151. http://www.aclweb.org/anthology/P07-1019. [↗](https://aclanthology.org/P07-1019/)
12. Marcin Junczys-Dowmunt, Tomasz Dwojak, and Hieu Hoang. 2016a. Is neural machine translation ready for deployment? a case study on 30 translation directions. In Proceedings of the 13th International Workshop on Spoken Language Translation (IWSLT). Seattle, Washington. [find](https://scholar.google.com/scholar?q=Marcin+Junczys%2DDowmunt%2C+Tomasz+Dwojak%2C+and+Hieu+Hoang%2E+2016a%2E+Is+neural+machine+translation+ready+for+deployment%3F+a+case+study+on+30+translation+directions%2E+In+Proceedings+of+the+13th+International+Wo)
13. Marcin Junczys-Dowmunt, Tomasz Dwojak, and Rico Sennrich. 2016b. The amu-uedin submission to the wmt16 news translation task: Attention-based nmt models as feature functions in phrase-based smt. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 319–325. http://www.aclweb.org/anthology/W16-2316. [↗](https://aclanthology.org/W16-2316/)
14. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 1700–1709. http://www.aclweb.org/anthology/D13-1176. [↗](https://aclanthology.org/D13-1176/)
15. Margaret King and Kirsten Falkedal. 1990. Using test suites in evaluation of machine translation systems. In Proceedings of the 1990 Conference on Computational Linguistics. Association for Computational Linguistics, Helsinki, Finland. http://aclweb.org/anthology/C/C90/C90-2037.pdf. [↗](https://aclanthology.org/C90-2037/) [our card](https://md.mova.elamur.ai/free/en/docs/papers/king-1990-test-suites-evaluation-machine-translation/)
16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pages 311–318. https://doi.org/10.3115/1073083.1073135. [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)