[Mova](https://md.mova.elamur.ai/free/en/) · [this page in HTML](https://mova.elamur.ai/free/en/docs/papers/lommel-2013-multidimensional-quality-metrics-flexible-system/) [home](https://md.mova.elamur.ai/free/en/) / [docs](https://md.mova.elamur.ai/free/en/docs/) / [papers](https://md.mova.elamur.ai/free/en/docs/papers/) / [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) / Multidimensional quality metrics: a flexible system for assessing translation quality # Multidimensional quality metrics: a flexible system for assessing translation quality **Authors:** Arle Richard Lommel, [Aljoscha Burchardt](https://md.mova.elamur.ai/free/en/docs/papers/authors/aljoscha-burchardt/), [Hans Uszkoreit](https://md.mova.elamur.ai/free/en/docs/papers/authors/hans-uszkoreit/) · **Year:** 2013 · **Venue:** Proceedings of Translating and the Computer 35 **Link:** [aclanthology.org/2013.tc-1.6](https://aclanthology.org/2013.tc-1.6/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only [**87** citations in our library](https://md.mova.elamur.ai/free/en/docs/papers/cited-by/lommel-2013-multidimensional-quality-metrics-flexible-system/) of the 140673 papers whose bibliographies we read[Proceedings of Translating and the Computer 35](https://aclanthology.org/volumes/2013.tc-1/)[licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/2013.tc-1.6/) - [PDF](https://aclanthology.org/2013.tc-1.6.pdf) **Authors (3)** 1. Arle Richard Lommel 2. Aljoscha Burchardt 3. Hans Uszkoreit Citations and author metrics from OpenAlex (); authors with the same name are not always told apart. ## What it is about The paper describes Multidimensional Quality Metrics (MQM), a framework from the QTLaunchPad project for building translation quality metrics out of a shared catalogue of issue types. The catalogue had about 108 issue types in 2013, arranged in a hierarchy and drawn from more than twenty existing metrics and checking tools, with BLEU and error-analysis tools included as sources. The issue types sit under five branches: fluency, accuracy, verity (conformity with the real world), design and internationalisation. A concrete metric chooses a small subset, usually no more than about twelve types, guided by twelve dimensions from the translation-specification standard, such as audience, purpose and register. The scoring rule starts from 100 and subtracts weighted penalties per word, with severity weights of 1 for minor, 5 for major and 10 for critical, taken from the LISA QA model. Penalties for source-side errors are added back, so a translator who repairs the source is not punished. The paper argues that one fixed metric cannot serve all translation tasks, and that the framework is free to implement. ## Ideas - I1. Map the translator's residual classes (lexical, construction, order, function words, punctuation and case, morphology) to MQM issue types in a table under `docs/`. The labels then match other metrics, and the translator's residual becomes portable. - I2. Give each residual token a severity: minor for punctuation and case, major for lexical choice and missing content, critical for a changed meaning. Report a weighted penalty per verse next to the bit count, and keep the bit count as the gate. - I3. Credit source repairs. A reader that corrects a source-side problem, for example a wrong tree in the Hebrew reader, should be counted apart from output loss, as the paper's source credit intends. - I4. Select the measure per author or style class, using a small set of dimensions, as the paper recommends. This suits the author deltas and the style-class deltas of open problem 8. ## For Mova The code is the residual report in `tools/translate` and the label table in `docs/`. The paper gives the translator a shared vocabulary for its errors and a way to weight them, so that a verse with one wrong word and a verse with a changed meaning are not counted alike. It does not give a better gate than the residual in bits, and should not replace it. Topics: [Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/topics/machine-translation/) ## Cite The paper itself, not this card. BibTeX from the library's record of the work. ``` @inproceedings{lommel2013multidimensional, title = {{Multidimensional quality metrics: a flexible system for assessing translation quality}}, author = {Lommel, Arle Richard and Burchardt, Aljoscha and Uszkoreit, Hans}, booktitle = {Proceedings of Translating and the Computer 35}, year = {2013}, url = {https://aclanthology.org/2013.tc-1.6/} } ``` - [← previous MEANT 2.0: Accurate semantic MT evaluation for any output language](https://md.mova.elamur.ai/free/en/docs/papers/lo-2017-meant-2-0-accurate-semantic/) - [next → Linguistically Motivated Evaluation of the 2022 State-of-the-art Machine Translation Systems for Three Language Directions](https://md.mova.elamur.ai/free/en/docs/papers/macketanz-2022-linguistically-motivated-evaluation-2022-state/) - [↑ back to Machine translation](https://md.mova.elamur.ai/free/en/docs/papers/mtrans/) ## References 6 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Garvin, David. 1984. “What does ‘product quality’ really mean?” Sloan Management Review, fall, pp. 25–45. [find](https://scholar.google.com/scholar?q=Garvin%2C+David%2E+1984%2E+%E2%80%9CWhat+does+%E2%80%98product+quality%E2%80%99+really+mean%3F%E2%80%9D+Sloan+Management+Review%2C+fall%2C+pp%2E+25%E2%80%9345%2E) 2. ISO. 2012. ISO/TS 11669:2012: Translation projects -- General guidance. Geneva: ISO. [find](https://scholar.google.com/scholar?q=ISO%2E+2012%2E+ISO%2FTS+11669%3A2012%3A+Translation+projects+%2D%2D+General+guidance%2E+Geneva%3A+ISO%2E) 3. LISA. 2006. “LISA QA Model 3.1: Assisting the localization development, production and quality control processes for global product distribution” (press release). Romainmôtier: LISA. [find](https://scholar.google.com/scholar?q=LISA%2E+2006%2E+%E2%80%9CLISA+QA+Model+3%2E1%3A+Assisting+the+localization+development%2C+production+and+quality+control+processes+for+global+product+distribution%E2%80%9D+%28press+release%29%2E+Romainm%C3%B4tier%3A+LISA%2E) 4. Lommel, Arle R. 2013. “Measuring translation quality in today’s automated lifecycle”. Presentation at tcworld 2013, Wiesbaden, Germany, November 8. Available from [http://tagungen.tekom.de/fileadmin/tx_doccon/slides/453_A_Unified_Model_for_Document_and_Translation_Quality_Assurance.pdf](http://tagungen.tekom.de/fileadmin/tx_doccon/slides/453_A_Unified_Model_for_Document_and_Translation_Quality_Assurance.pdf) (accessed November 14, 2013). [↗](http://tagungen.tekom.de/fileadmin/tx_doccon/slides/453_A_Unified_Model_for_Document_and_Translation_Quality_Assurance.pdf) 5. QTLaunchPad. 2013a. Multidimensional Quality Metric quality issue types, version 2.5.5. [https://docs.google.com/document/d/1E8IR1-8bR_M7VouHQhogUPpP2htpprwMMx1Mk9KPBTI/pub](https://docs.google.com/document/d/1E8IR1-8bR_M7VouHQhogUPpP2htpprwMMx1Mk9KPBTI/pub) (accessed November 14, 2013). —. 2013b. Error analysis in near miss translations. http://www.qt21.eu/launchpad/deliverable/error-analysis-near-miss-translations (accessed November 14, 2013). [↗](https://docs.google.com/document/d/1E8IR1-8bR_M7VouHQhogUPpP2htpprwMMx1Mk9KPBTI/pub) 6. SAE International. 2001. SAE J2450: Translation Quality Metric. [find](https://scholar.google.com/scholar?q=SAE+International%2E+2001%2E+SAE+J2450%3A+Translation+Quality+Metric%2E)