What are the metrics for evaluating a translation?
We have established again and again that the ‘quality’ of a translation is very hard to measure absolutely. Beyond certain objective indices that everyone can agree on (linguistic mistakes, correct use of industry terminology, correct use of terminology specified by the client) it very often comes down to a question of (subjective) preference. There are, however, several ways of evaluating the quality and impact of a translation, beyond the text itself.
They fall into four broad categories:
1. Does it meet your specifications?
A key point is whether the translation, and the translator, responds to your specifications. These may be fundamentally your subjective choices, but in the moment that you communicate them clearly to your language service provider (LSP) they become objective!
Translation specifications can include:
- Tailoring the copy for the intended (target) audience
- The purpose of the content — eg. it’s advertising or a technical document
- Intended usage: will it only be used internally or will customers see it?
- Specific terminology, such as product names or taglines, that must be translated in a pre-agreed way
- The target location (country or region) of the content
Most importantly, you should be sure that your LSP is working following the glossaries and style guides provided in your localisation kit and/or brand book. This also makes it easier to compare the work of different LSPs and translators.
Read more: What can I do internally to Streamline the Translation Process?
2. Translator benchmarks
The translation industry is cost-driven and unregulated. If an LSP actively seeks external validation, this can be taken as a good sign. You should thus check which translators’ associations they belong to. A very important consideration is the ISO: 9001 certification, which speaks volumes for about an LSP’s commitment to providing outstanding service.
ISO 9001 certification cannot guarantee the quality of a specific translation. However, it will evaluate the Translation Agency itself, and guarantee that you receive a high-quality, professional service, managed and measured in an objective manner. Furthermore, it ensures that the Agency procedures focus attention on correcting mistakes, and preventing error recurrence.
For example, ISO 9001 requires that some form of measuring customer satisfaction is directly incorporated into the company’s core processes; as well as procedures for dealing with customer feedback. Also, the quality of the suppliers (the translators) must be objectively measured on a regular basis.
Read more: How to evaluate a Translation Agency
3. Vertical industry metrics
In recent years many industries have promulgating their own global standards for translation and globalisation. For example, The Society for Automotive Engineers developed the J2450 “Quality metric for language translation of service information“; and General Motors specifies J2450 in its contracts.
The Multidimensional Quality Metrics (MQM) framework provides a vocabulary for categorising translation quality issues. The project is headed by the European Commission and aims to improve mass machine translation of important documents.
MQM provides a set of categories for translation quality issues, including:
- Accuracy
- Local Conventions
- Style
- Terminology
- Fluency
But the best method of evaluating a translation?
4. Test it on humans!
This is the ultimate metric of evaluating a translation: does it achieve was it was supposed to achieve. In the case of marketing collateral, show it to an unbiased member of the target demographic. If it is a technical manual, run it past an engineer.
This is also a good way to streamline the future translation process, as it will enable you to communicate better what your needs are to your LSP. Ideally, there should be a continual feedback loop between the two parties (LSP and client). Good communication is the best way to guarantee a productive relationship.
