[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/)

[home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) / Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics

# Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics

**Authors:** [Nitika Mathur](https://md.mova.elamur.ai/free/en/docs/papers/authors/nitika-mathur/), [Timothy Baldwin](https://md.mova.elamur.ai/free/en/docs/papers/authors/timothy-baldwin/), Trevor Cohn · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the ACL (ACL 2020) **Link:** [aclanthology.org/2020.acl-main.448](https://aclanthology.org/2020.acl-main.448/) · **Licence:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)

[**188** citations](https://openalex.org/works?filter=cites:W3034269545) openalex · 2026-10-18[**203** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/mathur-2020-tangled-up-bleu-reevaluating-evaluation/) of the 140673 papers whose bibliographies we read[Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://aclanthology.org/volumes/2020.acl-main/)[licence](https://creativecommons.org/licenses/by/4.0/)

- [Publisher's page](https://aclanthology.org/2020.acl-main.448/)
- [PDF](https://aclanthology.org/2020.acl-main.448.pdf)
- [DOI 10.18653/v1/2020.acl-main.448](https://doi.org/10.18653/v1/2020.acl-main.448)
- [arXiv 2006.06264](https://arxiv.org/abs/2006.06264)
- [OpenAlex](https://openalex.org/W3034269545)

**Authors (3)**

1. [Nitika Mathur](https://openalex.org/A5007765266) h 10 · 516 citations · 17 works
2. [Timothy Baldwin](https://openalex.org/A5103085805) [ORCID](https://orcid.org/0000-0002-4445-1386) · h 64 · 18251 citations · 880 works
3. [Trevor Cohn](https://openalex.org/A5078530959) [ORCID](https://orcid.org/0000-0003-4363-1673) · h 49 · 8963 citations · 305 works

Citations and author metrics from openalex (2026-10-18); authors with the same name are not always told apart.

## What it is about

The paper checks how metrics are judged. The usual test is the Pearson correlation between a metric's system scores and human direct-assessment scores, computed on the WMT 2019 systems. The authors show that this correlation is fragile: a single outlier system, far worse than the rest, can lift the correlation of all metrics above 0.97 when it is in a sample, and removing outliers lowers the correlations for BLEU, chrF, YiSi and ESIM in several language pairs. Outliers are found with the median and the median absolute deviation, with a cutoff of 2.5. When only the top four systems are kept, correlations in some pairs fall to near zero or below zero, and rolling-window tests show that Pearson's r is unstable for small samples of close systems. The second part turns to pairwise ranking. It sets a threshold on the metric's gain that trades type I errors (accepting differences humans do not see) against type II errors (rejecting differences humans do see), and reports how often each occurs at a given threshold.

## Ideas

- I1. Before any claim that a Mova measure agrees with a gold, rerun the check with and without outlier systems, using the MAD rule with cutoff 2.5, and report both r values. Apply it to the translator's own comparisons of readers, such as meta-world against word-by-word.
- I2. Decide author deltas with a paired test over verses instead of a count rule. The current rule (at least three more tokens fixed, in at least half the context) has no error rate attached; a bootstrap over the same verses gives one.
- I3. Report the spread of a score over random subsets of verses, drawn with replacement as the paper does for rankings. Two systems whose difference lies inside that spread should be called equal.
- I4. Test the metrics on close variants separately. The paper shows that metrics lose validity among near-equal systems; the meta-world ties close Bible authors for the same reason, so the same check applies there.

## For Mova

The code is the evaluation harness in `tools/translate` and the measures in `tools/translate/src/measure.rs`. The paper matters for the translator's release gate, which runs once per release on held-out data. A gate on a single number is fragile in the way the paper describes, so the release report should carry the subset spread and the outlier check. The threshold analysis is the right tool for choosing between a delta and no delta.

Topics: [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/)

## Cite

The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.18653/v1/2020.acl-main.448](https://doi.org/10.18653/v1/2020.acl-main.448) · [arXiv 2006.06264](https://arxiv.org/abs/2006.06264)

```
@inproceedings{mathur2020tangled,
  title = {{Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics}},
  author = {Mathur, Nitika and Baldwin, Timothy and Cohn, Trevor},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.448},
  eprint = {2006.06264},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.448}
}
```

- [← previous Grammar-Based Concept Alignment for Domain-Specific Machine Translation](https://md.mova.elamur.ai/free/en/docs/papers/masciolini-2021-grammar-based-concept-alignment-domain/)
- [next → DTED: Evaluation of Machine Translation Structure Using Dependency Parsing and Tree Edit Distance](https://md.mova.elamur.ai/free/en/docs/papers/mccaffery-2016-dted-evaluation-machine-translation-structure/)
- [↑ back to Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/)

## References

30 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Francis J Anscombe. 1973. Graphs in statistical analysis. The American Statistician, 27(1):17–21. [find](https://scholar.google.com/scholar?q=Francis+J+Anscombe%2E+1973%2E+Graphs+in+statistical+analysis%2E+The+American+Statistician%2C+27%281%29%3A17%E2%80%9321%2E)
2. Vic Barnett and Toby Lewis. 1974. Outliers in Statistical Data. Wiley. [find](https://scholar.google.com/scholar?q=Vic+Barnett+and+Toby+Lewis%2E+1974%2E+Outliers+in+Statistical+Data%2E+Wiley%2E)
3. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. [find](https://scholar.google.com/scholar?q=Lo%C3%AFc+Barrault%2C+Ondr%CC%8Cej+Bojar%2C+Marta+R%2E+Costa%2Djuss%C3%A0%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Philipp+Koehn%2C+Shervin+Malmasi%2C+Christof+Monz%2C+Mathias+M%C3%BCller%2C+Santanu)
4. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Findings+of+the+2019+conference+on+machine+translation+%28WMT19%29%2E+In+Proceedings+of+the+Fourth+Conference+on+Machine+Translation+%28Volume+2%3A+Shared+Task+Papers%2C+Day+1%29%2C+pages+1%E2%80%9361%2C+Florence%2C+Italy%2E+Assoc)
5. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12–58, Baltimore, Maryland, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Christian+Buck%2C+Christian+Federmann%2C+Barry+Haddow%2C+Philipp+Koehn%2C+Johannes+Leveling%2C+Christof+Monz%2C+Pavel+Pecina%2C+Matt+Post%2C+Herve+Saint%2DAmand%2C+Radu+Soricut%2C+Lucia+Specia%2C+and+Ale%C5%A1+Tamc)
6. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131–198, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Antonio+Jimeno+Yepes%2C+Philipp+Koehn%2C+Varvara+Logacheva%2C+Christof+Monz%2C+Matteo+Negri%2C+Aur%C3%A9lie+N%C3%A9v%C3%A9ol%2C+M)
7. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of Bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Miles+Osborne%2C+and+Philipp+Koehn%2E+2006%2E+Re%2Devaluating+the+role+of+Bleu+in+machine+translation+research%2E+In+11th+Conference+of+the+European+Chapter+of+the+Association+for+Computat)
8. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+LSTM+for+natural+language+inference%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computa)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, USA. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
10. Marina Fomicheva and Lucia Specia. 2019. Taking MT evaluation metrics to extremes: Beyond correlation with human judgments. Computational Linguistics, 45(3):515–558. [find](https://scholar.google.com/scholar?q=Marina+Fomicheva+and+Lucia+Specia%2E+2019%2E+Taking+MT+evaluation+metrics+to+extremes%3A+Beyond+correlation+with+human+judgments%2E+Computational+Linguistics%2C+45%283%29%3A515%E2%80%93558%2E)
11. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, Doha, Qatar. [find](https://scholar.google.com/scholar?q=Yvette+Graham+and+Timothy+Baldwin%2E+2014%2E+Testing+for+significance+of+increased+correlation+with+human+judgment%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
12. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone? Natural Language Engineering, 23(1):3–30. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2017%2E+Can+machine+translation+systems+be+evaluated+by+the+crowd+alone%3F+Natural+Language+Engineering%2C+23%281%29%3A3%E2%80%9330%2E)
13. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 266–274, Baltimore, Maryland, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Nitika+Mathur%2C+and+Timothy+Baldwin%2E+2014%2E+Randomized+significance+tests+in+machine+translation%2E+In+Proceedings+of+the+Ninth+Workshop+on+Statistical+Machine+Translation%2C+pages+266%E2%80%93274%2C+B)
14. Boris Iglewicz and David Caster Hoaglin. 1993. How to detect and handle outliers, volume 16. Asq Press. [find](https://scholar.google.com/scholar?q=Boris+Iglewicz+and+David+Caster+Hoaglin%2E+1993%2E+How+to+detect+and+handle+outliers%2C+volume+16%2E+Asq+Press%2E)
15. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2004%2E+Statistical+significance+tests+for+machine+translation+evaluation%2E+In+Proceedings+of+the+2004+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+388%E2%80%93395%2C+Barcel)
16. Christophe Leys, Christophe Ley, Olivier Klein, Philippe Bernard, and Laurent Licata. 2013. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49(4):764–766. [find](https://scholar.google.com/scholar?q=Christophe+Leys%2C+Christophe+Ley%2C+Olivier+Klein%2C+Philippe+Bernard%2C+and+Laurent+Licata%2E+2013%2E+Detecting+outliers%3A+Do+not+use+standard+deviation+around+the+mean%2C+use+absolute+deviation+around+the+median%2E)
17. Chi-kiu Lo. 2019. YiSi — a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507–513, Florence, Italy. [find](https://scholar.google.com/scholar?q=Chi%2Dkiu+Lo%2E+2019%2E+YiSi+%E2%80%94+a+unified+semantic+MT+quality+evaluation+and+estimation+metric+for+languages+with+different+levels+of+available+resources%2E+In+Proceedings+of+the+Fourth+Conference+on+Machine+T)
18. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. [find](https://scholar.google.com/scholar?q=Qingsong+Ma%2C+Johnny+Wei%2C+Ondr%CC%8Cej+Bojar%2C+and+Yvette+Graham%2E+2019%2E+Results+of+the+WMT19+metrics+shared+task%3A+Segment%2Dlevel+and+strong+MT+systems+pose+big+challenges%2E+In+Proceedings+of+the+Fourth+Confere)
19. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799–2808, Florence, Italy. [find](https://scholar.google.com/scholar?q=Nitika+Mathur%2C+Timothy+Baldwin%2C+and+Trevor+Cohn%2E+2019%2E+Putting+evaluation+in+context%3A+Contextual+embeddings+improve+machine+translation+evaluation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Ass)
20. Jason W Osborne and Amy Overbay. 2004. The power of outliers (and why researchers should always check for them). Practical Assessment, Research & Evaluation, 9(6):1–12. [find](https://scholar.google.com/scholar?q=Jason+W+Osborne+and+Amy+Overbay%2E+2004%2E+The+power+of+outliers+%28and+why+researchers+should+always+check+for+them%29%2E+Practical+Assessment%2C+Research+%26+Evaluation%2C+9%286%29%3A1%E2%80%9312%2E)
21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002a. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, USA. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002a%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Com) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002b. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, USA. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002b%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+for+Com) [our card](https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
23. Maja Popović. 2015. chrF: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2015%2E+chrF%3A+character+n%2Dgram+f%2Dscore+for+automatic+MT+evaluation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+392%E2%80%93395%2C+Lisbon%2C+Portugal%2E+Association+f) [our card](https://md.mova.elamur.ai/free/en/docs/papers/popovic-2015-chrf-character-n-gram-f/)
24. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matt+Post%2E+2018%2E+A+call+for+clarity+in+reporting+BLEU+scores%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%3A+Research+Papers%2C+pages+186%E2%80%93+191%2C+Belgium%2C+Brussels%2E+Association+for+Computa) [our card](https://md.mova.elamur.ai/free/en/docs/papers/post-2018-call-clarity-reporting-bleu-scores/)
25. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393– 401. [find](https://scholar.google.com/scholar?q=Ehud+Reiter%2E+2018%2E+A+structured+review+of+the+validity+of+BLEU%2E+Computational+Linguistics%2C+44%283%29%3A393%E2%80%93+401%2E) [our card](https://md.mova.elamur.ai/free/en/docs/papers/reiter-2018-structured-review-validity-bleu/)
26. Peter J Rousseeuw and Mia Hubert. 2011. Robust statistics for outlier detection. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1(1):73–79. [find](https://scholar.google.com/scholar?q=Peter+J+Rousseeuw+and+Mia+Hubert%2E+2011%2E+Robust+statistics+for+outlier+detection%2E+Wiley+Interdisciplinary+Reviews%3A+Data+Mining+and+Knowledge+Discovery%2C+1%281%29%3A73%E2%80%9379%2E)
27. Aaron Smith, Christian Hardmeier, and Joerg Tiedemann. 2016. Climbing mont BLEU: The strange world of reachable high-BLEU translations. In Proceedings of the 19th Annual Conference of the European Association for Machine Translation, pages 269–281. [find](https://scholar.google.com/scholar?q=Aaron+Smith%2C+Christian+Hardmeier%2C+and+Joerg+Tiedemann%2E+2016%2E+Climbing+mont+BLEU%3A+The+strange+world+of+reachable+high%2DBLEU+translations%2E+In+Proceedings+of+the+19th+Annual+Conference+of+the+European+Ass)
28. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the Association for Machine Transaltion in the Americas, pages 223–231. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+Proceedings+of+the+Association+for+Machine+) [our card](https://md.mova.elamur.ai/free/en/docs/papers/snover-2006-study-translation-edit-rate-targeted/)
29. Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In Computational Linguistics and Intelligent Text Processing, pages 341– 351, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Amanda+Stent%2C+Matthew+Marge%2C+and+Mohit+Singhai%2E+2005%2E+Evaluating+evaluation+methods+for+generation+in+the+presence+of+variation%2E+In+Computational+Linguistics+and+Intelligent+Text+Processing%2C+pages+341)
30. Rand Wilcox. 2004. Inferences based on a skipped correlation coefficient. Journal of Applied Statistics, 31(2):131–143. [find](https://scholar.google.com/scholar?q=Rand+Wilcox%2E+2004%2E+Inferences+based+on+a+skipped+correlation+coefficient%2E+Journal+of+Applied+Statistics%2C+31%282%29%3A131%E2%80%93143%2E)