[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/vilar-2006-error-analysis-statistical-machine-translation/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) / Error Analysis of Statistical Machine Translation Output

# Error Analysis of Statistical Machine Translation Output

**Authors:** David Vilar, Jia Xu, Luis Fernando D'Haro, [Hermann Ney](https://md.mova.elamur.ai/free/en/docs/papers/authors/hermann-ney/) · **Year:** 2006 · **Venue:** Fifth International Conference on Language Resources and Evaluation (LREC 2006) **Link:** [aclanthology.org/L06-1244](https://aclanthology.org/L06-1244/) · **Licence:** [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — summary only

[**83** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/vilar-2006-error-analysis-statistical-machine-translation/) of the 140673 papers whose bibliographies we read[Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)](https://aclanthology.org/volumes/L06-1/)[licence](https://creativecommons.org/licenses/by-nc/4.0/)

- [Publisher's page](https://aclanthology.org/L06-1244/)
- [PDF](https://aclanthology.org/L06-1244.pdf)

**Authors (4)**

1. David Vilar
2. Jia Xu
3. Luis Fernando D'Haro
4. Hermann Ney

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

The paper gives a hand-built error typology for statistical MT output and applies it to the RWTH system in the first TC-STAR evaluation, for Spanish-to-English, English-to-Spanish and Chinese-to-English, on speech-derived data (clean text, verbatim transcripts and recogniser output). The top level has five classes: missing words, word order, incorrect words, unknown words and punctuation. Missing words are split into content and filler words, word order into word and phrase level, local and long range, and incorrect words into wrong lexical choice, wrong disambiguation, wrong form, extra words, style and idioms. The authors observe that automatic metrics such as WER, PER, BLEU and NIST cannot say what the main problems are. For English-to-Spanish, verb tense errors made up about 15 percent of the errors, and noun-adjective reordering was prominent. For Spanish-to-English, lexical choice and disambiguation were about 28 percent, and most missing words were filler words that did not change the meaning. The typology is the basis for a later automatic classifier.

## Ideas

- I1. Use this typology as the label set for the translator's residual: missing versus extra words, local and long-range order, word and phrase order, wrong lexical choice, wrong form, idiom. Most residual classes already match, so the change is in the names.
- I2. Count missing filler words apart from missing content words in `tools/translate/src/measure.rs`. This is open problem 4 in another form: function words and boundaries should be features of meta tokens, and their loss needs its own number.
- I3. Treat the be-type disambiguation as a sense class. The paper's example, two Spanish verbs for one English verb, is the analogue of merging two concepts that differ in use, such as *be* and *have* in the current lexicon (open problem 2).
- I4. Report verb tense and person as a separate error class for every target language with a verbal system. This is a morphology measure that `en/src/morph.rs` could supply.

## For Mova

The code is the residual labels in `tools/translate` and the morphology side in `en/src/morph.rs`. The typology gives the translator a human-readable error list that matches its own classes, so that residual numbers can be read in the same terms as earlier error studies. The paper's finding that filler words account for much of the missing material supports the open problem on function words.

Topics: [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{vilar2006error,
  title = {{Error Analysis of Statistical Machine Translation Output}},
  author = {Vilar, David and Xu, Jia and D'Haro, Luis Fernando and Ney, Hermann},
  booktitle = {Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)},
  year = {2006},
  url = {https://aclanthology.org/L06-1244/}
}
```

- [← previous Translation Techniques and Interpretative Phenomena in the Greek Version of the Hebrew Bible: A Study of the Figurative Use of the Noun צֵל 'Shadow'](https://md.mova.elamur.ai/free/en/docs/papers/vergari-2015-translation-techniques-interpretative-phenomena-greek/)
- [next → Developing an interlingual translation lexicon using WordNets and Grammatical Framework](https://md.mova.elamur.ai/free/en/docs/papers/virk-2014-developing-interlingual-translation-lexicon-wordnets/)
- [↑ back to Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/)

## References

5 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proc. ARPA Workshop on Human Language Technology. [find](https://scholar.google.com/scholar?q=George+Doddington%2E+2002%2E+Automatic+evaluation+of+machine+translation+quality+using+n%2Dgram+co%2Doccurrence+statistics%2E+In+Proc%2E+ARPA+Workshop+on+Human+Language+Technology%2E)
2. LDC. 2005. Linguistic data consortium chinese training data resources. [http://www.ldc.upenn.edu/Projects/TIDES/mt2005cn.htm](http://www.ldc.upenn.edu/Projects/TIDES/mt2005cn.htm). [↗](http://www.ldc.upenn.edu/Projects/TIDES/mt2005cn.htm)
3. Ariadna Font Llitjós, Jaime G. Carbonell, and Alon Lavie. 2005. A framework for interactive and automatic refinement of transfer-based machine translation. In Proc. of the 10th Annual Conf. of the European Association for Machine Translation (EAMT), Budapest, Hungary, May. [find](https://scholar.google.com/scholar?q=Ariadna+Font+Llitjo%CC%81s%2C+Jaime+G%2E+Carbonell%2C+and+Alon+Lavie%2E+2005%2E+A+framework+for+interactive+and+automatic+refinement+of+transfer%2Dbased+machine+translation%2E+In+Proc%2E+of+the+10th+Annual+Conf%2E+of+the+Eu)
4. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, PA, July. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proc%2E+of+the+40th+Annual+Meeting+of+the+Association+for+Computatio) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
5. David Vilar, Evgeny Matusov, Saša Hasan, Richard Zens, and Hermann Ney. 2005. Statistical Machine Translation of European Parliamentary Speeches. In Proceedings of MT Summit X, pages 259–266, Phuket, Thailand, September. Asia-Pacific Association for Machine Translation (AAMT). [find](https://scholar.google.com/scholar?q=David+Vilar%2C+Evgeny+Matusov%2C+Sas%CC%8Ca+Hasan%2C+Richard+Zens%2C+and+Hermann+Ney%2E+2005%2E+Statistical+Machine+Translation+of+European+Parliamentary+Speeches%2E+In+Proceedings+of+MT+Summit+X%2C+pages+259%E2%80%93266%2C+Phuket%2C)