[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/popovic-2006-morpho-syntactic-information-automatic-error/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) / Morpho-syntactic Information for Automatic Error Analysis of Statistical Machine Translation Output

# Morpho-syntactic Information for Automatic Error Analysis of Statistical Machine Translation Output

**Authors:** [Maja Popović](https://md.mova.elamur.ai/free/en/docs/papers/authors/maja-popovic/), Adrià de Gispert, Deepa Gupta, Patrik Lambert, [Hermann Ney](https://md.mova.elamur.ai/free/en/docs/papers/authors/hermann-ney/), José B. Mariño, Marcello Federico, Rafael Banchs · **Year:** 2006 · **Venue:** Workshop on Statistical Machine Translation (WMT 2006) **Link:** [aclanthology.org/W06-3101](https://aclanthology.org/W06-3101/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only

[**11** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/popovic-2006-morpho-syntactic-information-automatic-error/) of the 140673 papers whose bibliographies we read[Proceedings on the Workshop on Statistical Machine Translation](https://aclanthology.org/volumes/W06-31/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/)

- [Publisher's page](https://aclanthology.org/W06-3101/)
- [PDF](https://aclanthology.org/W06-3101.pdf)

**Authors (8)**

1. Maja Popović
2. Adrià de Gispert
3. Deepa Gupta
4. Patrik Lambert
5. Hermann Ney
6. José B. Mariño
7. Marcello Federico
8. Rafael Banchs

Citations and author metrics from OpenAlex (); authors with the same name are not always told apart.

## What it is about

The paper uses word classes to find out where an SMT system goes wrong, for Spanish and English in the TC-STAR parliament data. It computes WER and PER separately for nouns, adjectives and verbs, and for full forms and base forms. The gap between PER and one minus WER, measured on nouns and adjectives, indicates reordering: words are translated correctly but in the wrong order. Spanish puts most adjectives after the noun, so the gap is an indicator of the noun-adjective problem in both directions. A large difference between full-form and base-form PER for Spanish indicates inflection errors. The authors also test a pre-reordering step that moves each noun past its adjective group on the source side. On a small 13,000-sentence subset this raised Spanish-to-English BLEU from 43 to about 49, but on the full corpus the gain was about 2 points, from 55 to 56, so the benefit depends on data size. Their error analysis of the full system showed the reordering gap was above 20 percent for both directions.

## Ideas

- I1. Compute WER and PER per word class in `tools/translate/src/measure.rs`, with classes taken from the UD part-of-speech tags of the stack output (`tools/roundtrip` trees). The PER-minus-WER gap per class is an order signal.
- I2. Use full-form against base-form PER as an inflection signal. `en/src/morph.rs` supplies base forms, and a large gap for a target language flags a morphology problem, which open problem 5 (number and person from UD features) would then address.
- I3. Test a pre-reordering rule, such as moving the noun past the adjective, on both a small and the full dev split, and run each twice. The paper's result shows the same rule can help on little data and do nothing on more, so the claim must be made on the size the translator will use.
- I4. Use the per-class rates to locate the Hebrew reader's weakest part (open problem 7). The verb and subject classes are where a verb-subject order normaliser would show up.

## For Mova

The code is `tools/translate/src/measure.rs` for the per-class rates and `en/src/lin.rs` for any order rule. The paper gives a diagnostic that separates reordering from inflection without a reference parser, which is useful while the translator's reader is still the weakest part. The data-size effect in the paper is a warning for any pre-reordering rule the translator considers.

Topics: [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work.

```
@inproceedings{popovic2006morpho,
  title = {{Morpho-syntactic Information for Automatic Error Analysis of Statistical Machine Translation Output}},
  author = {Popović, Maja and Gispert, Adrià de and Gupta, Deepa and Lambert, Patrik and Ney, Hermann and Mariño, José B. and Federico, Marcello and Banchs, Rafael},
  booktitle = {Proceedings on the Workshop on Statistical Machine Translation},
  year = {2006},
  url = {https://aclanthology.org/W06-3101/}
}
```

- [← previous Apertium-fin-eng–Rule-based Shallow Machine Translation for WMT 2019 Shared Task](https://md.mova.elamur.ai/free/en/docs/papers/pirinen-2019-apertium-fin-eng-rule-based/)
- [next → From Human to Automatic Error Classification for Machine Translation Output](https://md.mova.elamur.ai/free/en/docs/papers/popovic-2011-human-automatic-error-classification-machine/)
- [↑ back to Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/)

## References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bogdan Babych and Anthony Hartley. 2004. Extending bleu mt evaluation method with frequency weighting. In Proc. of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), Barcelona, Spain, July. [find](https://scholar.google.com/scholar?q=Bogdan+Babych+and+Anthony+Hartley%2E+2004%2E+Extending+bleu+mt+evaluation+method+with+frequency+weighting%2E+In+Proc%2E+of+the+42nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+Barce)
2. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgements. In 43rd Annual Meeting of the Assoc. for Computational Linguistics: Proc. Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, pages 65–72, Ann Arbor, MI, June. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+Meteor%3A+An+automatic+metric+for+mt+evaluation+with+improved+correlation+with+human+judgements%2E+In+43rd+Annual+Meeting+of+the+Assoc%2E+for+Computational+Linguisti)
3. Adrià de Gispert, José B. Mariño, and Josep M. Crego. 2005. Improving statistical machine translation by classifying and generalizing inflected verb forms. In Proc. of the 9th European Conf. on Speech Communication and Technology (Interspeech), pages 3185– 3188, Lisbon, Portugal, September. [find](https://scholar.google.com/scholar?q=Adria%CC%80+de+Gispert%2C+Jose%CC%81+B%2E+Marin%CC%83o%2C+and+Josep+M%2E+Crego%2E+2005%2E+Improving+statistical+machine+translation+by+classifying+and+generalizing+inflected+verb+forms%2E+In+Proc%2E+of+the+9th+European+Conf%2E+on+Spe)
4. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proc. ARPA Workshop on Human Language Technology, pages 128–132, San Diego. [find](https://scholar.google.com/scholar?q=George+Doddington%2E+2002%2E+Automatic+evaluation+of+machine+translation+quality+using+n%2Dgram+co%2Doccurrence+statistics%2E+In+Proc%2E+ARPA+Workshop+on+Human+Language+Technology%2C+pages+128%E2%80%93132%2C+San+Diego%2E)
5. Sharon Goldwater and David McClosky. 2005. Improving stastistical machine translation through morphological analysis. In Proc. of the Conf. on Empirical Methods for Natural Language Processing (EMNLP), Vancouver, Canada, October. [find](https://scholar.google.com/scholar?q=Sharon+Goldwater+and+David+McClosky%2E+2005%2E+Improving+stastistical+machine+translation+through+morphological+analysis%2E+In+Proc%2E+of+the+Conf%2E+on+Empirical+Methods+for+Natural+Language+Processing+%28EMNLP%29)
6. Young-suk Lee. 2004. Morphological analysis for statistical machine translation. In Proc. 2004 Meeting of the North American chapter of the Association for Computational Linguistics (HLT-NAACL), Boston, MA, May. [find](https://scholar.google.com/scholar?q=Young%2Dsuk+Lee%2E+2004%2E+Morphological+analysis+for+statistical+machine+translation%2E+In+Proc%2E+2004+Meeting+of+the+North+American+chapter+of+the+Association+for+Computational+Linguistics+%28HLT%2DNAACL%29%2C+Bosto)
7. Evgeny Matusov, Gregor Leusch, Oliver Bender, and Hermann Ney. 2005. Evaluating machine translation output with automatic sentence segmentation. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), pages 148–154, Pittsburgh, PA, October. [find](https://scholar.google.com/scholar?q=Evgeny+Matusov%2C+Gregor+Leusch%2C+Oliver+Bender%2C+and+Hermann+Ney%2E+2005%2E+Evaluating+machine+translation+output+with+automatic+sentence+segmentation%2E+In+Proceedings+of+the+International+Workshop+on+Spoken+)
8. Sonja Nießen and Hermann Ney. 2000. Improving SMT quality with morpho-syntactic analysis. In COLING ’00: The 18th Int. Conf. on Computational Linguistics, pages 1081–1085, Saarbrücken, Germany, July. [find](https://scholar.google.com/scholar?q=Sonja+Nie%C3%9Fen+and+Hermann+Ney%2E+2000%2E+Improving+SMT+quality+with+morpho%2Dsyntactic+analysis%2E+In+COLING+%E2%80%9900%3A+The+18th+Int%2E+Conf%2E+on+Computational+Linguistics%2C+pages+1081%E2%80%931085%2C+Saarbru%CC%88cken%2C+Germany%2C+July%2E)
9. Sonja Nießen and Hermann Ney. 2001a. Morphosyntactic analysis for reordering in statistical machine translation. In Proc. MT Summit VIII, pages 247–252, Santiago de Compostela, Galicia, Spain, September. [find](https://scholar.google.com/scholar?q=Sonja+Nie%C3%9Fen+and+Hermann+Ney%2E+2001a%2E+Morphosyntactic+analysis+for+reordering+in+statistical+machine+translation%2E+In+Proc%2E+MT+Summit+VIII%2C+pages+247%E2%80%93252%2C+Santiago+de+Compostela%2C+Galicia%2C+Spain%2C+Septemb)
10. Sonja Nießen and Hermann Ney. 2001b. Toward hierarchical models for statistical machine translation of inflected languages. In Data-Driven Machine Translation Workshop, pages 47–54, Toulouse, France, July. [find](https://scholar.google.com/scholar?q=Sonja+Nie%C3%9Fen+and+Hermann+Ney%2E+2001b%2E+Toward+hierarchical+models+for+statistical+machine+translation+of+inflected+languages%2E+In+Data%2DDriven+Machine+Translation+Workshop%2C+pages+47%E2%80%9354%2C+Toulouse%2C+France%2C+)
11. Sonja Nießen, Franz J. Och, Gregor Leusch, and Hermann Ney. 2000. An evaluation tool for machine translation: Fast evaluation for mt research. In Proc. Second Int. Conf. on Language Resources and Evaluation (LREC), pages 39–45, Athens, Greece, May. [find](https://scholar.google.com/scholar?q=Sonja+Nie%C3%9Fen%2C+Franz+J%2E+Och%2C+Gregor+Leusch%2C+and+Hermann+Ney%2E+2000%2E+An+evaluation+tool+for+machine+translation%3A+Fast+evaluation+for+mt+research%2E+In+Proc%2E+Second+Int%2E+Conf%2E+on+Language+Resources+and+Eval)
12. Kishore Papineni, Salim Roukos, Todd Ward, and Wie-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, PA, July. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wie%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proc%2E+of+the+40th+Annual+Meeting+of+the+Association+for+Computatio) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
13. Maja Popović and Hermann Ney. 2004. Towards the use of word stems & suffixes for statistical machine translation. In Proc. 4th Int. Conf. on Language Resources and Evaluation (LREC), pages 1585–1588, Lissabon, Portugal, May. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81+and+Hermann+Ney%2E+2004%2E+Towards+the+use+of+word+stems+%26+suffixes+for+statistical+machine+translation%2E+In+Proc%2E+4th+Int%2E+Conf%2E+on+Language+Resources+and+Evaluation+%28LREC%29%2C+pages+1585%E2%80%931588%2C)
14. Maja Popović and Hermann Ney. 2006. POS-based word reorderings for statistical machine translation. In Proc. of the Fifth Int. Conf. on Language Resources and Evaluation (LREC), Genova, Italy, May. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81+and+Hermann+Ney%2E+2006%2E+POS%2Dbased+word+reorderings+for+statistical+machine+translation%2E+In+Proc%2E+of+the+Fifth+Int%2E+Conf%2E+on+Language+Resources+and+Evaluation+%28LREC%29%2C+Genova%2C+Italy%2C+May%2E)
15. Maja Popović, David Vilar, Hermann Ney, Slobodan Jovičić, and Zoran Šarić. 2005. Augmenting a small parallel text with morpho-syntactic language resources for Serbian–English statistical machine translation. In 43rd Annual Meeting of the Assoc. for Computational Linguistics: Proc. Workshop on Building and Using Parallel Texts: Data-Driven Machine Translation and Beyond, pages 41–48, Ann Arbor, MI, June. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2C+David+Vilar%2C+Hermann+Ney%2C+Slobodan+Jovic%CC%8Cic%CC%81%2C+and+Zoran+S%CC%8Caric%CC%81%2E+2005%2E+Augmenting+a+small+parallel+text+with+morpho%2Dsyntactic+language+resources+for+Serbian%E2%80%93English+statistical+machine+)
16. David Vilar, Evgeny Matusov, Saša Hasan, Richard Zens, and Hermann Ney. 2005. Statistical machine translation of european parliamentary speeches. In Proc. MT Summit X, pages 259–266, Phuket, Thailand, September. [find](https://scholar.google.com/scholar?q=David+Vilar%2C+Evgeny+Matusov%2C+Sas%CC%8Ca+Hasan%2C+Richard+Zens%2C+and+Hermann+Ney%2E+2005%2E+Statistical+machine+translation+of+european+parliamentary+speeches%2E+In+Proc%2E+MT+Summit+X%2C+pages+259%E2%80%93266%2C+Phuket%2C+Thailand)
17. David Vilar, Jia Xu, Luis Fernando D’Haro, and Hermann Ney. 2006. Error analysis of statistical machine translation output. In Proc. of the Fifth Int. Conf. on Language Resources and Evaluation (LREC), page to appear, Genova, Italy, May. [find](https://scholar.google.com/scholar?q=David+Vilar%2C+Jia+Xu%2C+Luis+Fernando+D%E2%80%99Haro%2C+and+Hermann+Ney%2E+2006%2E+Error+analysis+of+statistical+machine+translation+output%2E+In+Proc%2E+of+the+Fifth+Int%2E+Conf%2E+on+Language+Resources+and+Evaluation+%28LREC%29) [our card](https://md.mova.elamur.ai/free/en/docs/papers/vilar-2006-error-analysis-statistical-machine-translation/)
18. Richard Zens, Oliver Bender, Saša Hasan, Shahram Khadivi, Evgeny Matusov, Jia Xu, Yuqi Zhang, and Hermann Ney. 2005. The RWTH phrase-based statistical machine translation system. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), pages 155–162, Pittsburgh, PA, October. [find](https://scholar.google.com/scholar?q=Richard+Zens%2C+Oliver+Bender%2C+Sas%CC%8Ca+Hasan%2C+Shahram+Khadivi%2C+Evgeny+Matusov%2C+Jia+Xu%2C+Yuqi+Zhang%2C+and+Hermann+Ney%2E+2005%2E+The+RWTH+phrase%2Dbased+statistical+machine+translation+system%2E+In+Proceedings+of+th)