Andrei Gerasimov, PhD, member of ATA, ITI and UTR (Russia), www.erussiantranslations.com

Review of Translation Quality Assurance Software*

Introduction
Translation Quality Assurance software (hereinafter referred to as TQA tools) compares source and target segments of bilingual texts (saved in .doc, .rtf, and .ttx files) in order to detect translation errors. Such errors include: inconsistencies; terms, which have not been translated in accordance with a project glossary; omissions; target segments, which are identical to source segments; punctuation, capitalization and number value/formatting errors; and incorrect untranslatables and tags.
The aim of this study is to compare three of the most popular TQA tools in order to find out their strengths and weaknesses, and therefore help translators, project managers and proofreaders to select the optimal TQA tool for any particular job.

Intrinsic Limitations of TQA Tools
There are a number of intrinsic limitations with TQA tools, some of which are listed below.
· TQA tools cannot detect mistakes arising from an incorrect (or incomplete) understanding of the source text, poor stylistics or an inappropriate choice of language register.
· When TQA tools check terminology, they are limited by the glossary being used for the check.
· TQA tools often detect false errors because they do not ‘understand’ that source and target languages may have different grammatical rules (for example, punctuation and capitalization). As will be seen below, the only TQA tool which has different language settings is QA Distiller.
· Comparison tools expect the source text to be correct, which is not always the case. If the translator rectifies a mistake in a source sentence (such as incorrect initial capitalization or punctuation), this may result in a false error being detected by the TQA tool.
· TQA tools work on the logic that all inconsistencies are equally bad. However, IMHO only special terminology should be translated consistently while general phrases which are identical in the source text may be translated in different ways in order to improve readability and style. Identical phrases may even require different translations depending on context.

General Description and Features of Three TQA Tools
The three TQA tools tested in this study were: SDL Trados Terminology Verifier and QA Checker; Wordfast Quality Check feature; and QA Distiller (hereinafter referred to as Trados, WF and QAD, respectively). General information about these three tools is contained in Table 1 below and a comparative list of their main features is given in Table 2.

Table 1
Trados Developed by SDL.Plug-ins integrated in Trados TagEditor.Files that can be checked directly: ttx.User interface used: TagEditor.Protection: soft key license file.
WF Developed by Yves Champollion.A feature integrated in WF. Files that can be checked directly: doc, rtf.User interface used: MS Word.Protection: license code.
QAD Developed by Yamagata Europe (Belgium).A stand-alone application. Requires installation of Trados. Files that can be checked directly: rtf, ttx, tmx.User interface used: proprietary (QAD UI).Protection: license code (requires Internet protection and the license only works for eight hours after disconnecting from the Internet).


Table 2
X means that a feature is provided.
0 means that a feature is not provided.

Name Details and explanation of the check carried out Trados WF QAD
Terminology Target terms used are identical to those specified in your glossary. X X X
Segment Forgotten and empty translations.Identical source and target text.Target segments that are shorter or longer than the source by a specified percentage.Target segments that contain more than a specified number of characters.Target segments that contain forbidden characters. X 0 X
Inconsistency Repeated phrases translated inconsistently. X 0 X
Punctuation Different end of sentence punctuation in source and target segments.Spaces before punctuation.Double spaces.Double dots. X Only double spaces X
Capitalization Capitalization of initial words. X 0 X
Numbers Are numbers identical in source and target segments. X X X
Tags Are tags identical in source and target segments. X X 0
Untranslatables Automatically detects untranslatables (even those not included in your glossary) and checks whether they are identical in source and target segments. 0 X 0
Bookmarks Source and target texts contain an identical number of bookmarks. 0 X 0
Other Features Trados WF QAD
TQA check settings can be customized. X X X
Customized TQA settings can be saved to file. X X X
The results of a TQA check can be save in a log file. X X X
Checks are performed in real time during the translation session (not after translation is completed). 0 X 0
Batch mode (the TQA tool can check multiple files during a single operation). 0 X X
Indication of segment with detected error X X X
Possibility to add your own TQA checks (macros) 0 X 0
Fuzzy terminology checks (the TQA takes into account during the terminology check that words may have various forms (case endings, for example)). Х X X
Language dependent settings. 0 0 X
License price (price of one user license). $895.00 From €90.00 $1000.00
Technical support from the developers. X X X

Detection of formal errors
In order to test these TQA tools, I created a test .doc file (1,373 words) containing a sample source text from a real client (Volvo Cars), and translated it with Trados in both MS Word and TagEditor. As a result, I had two identical bilingual target files (1,071 words) saved in .rtf and .ttx formats.
At the first stage (check of formal errors only) I added seven typical formal errors to both files:
1. One sentence was kept in English (identical source and target segments).
2. A double space.
3. One end of sentence punctuation different from that in the source sentence.
4. Repeated phrases translated inconsistently.
5. Incorrect untranslatable (Volvo S60 in the source segment changed to Volvo S60R in the target segment).
6. Incorrect number (350 in the source segment changed to 360 in the target segment).
7. One closing round bracket ")" missing in the target segment.

All special terminology in the target file was translated in accordance with my Volvo glossary (although I did not perform a terminology check at this stage of the study).

The settings in the three TQA tools were optimized experimentally to ensure detection of the maximum number of real errors and the minimum number of false errors (maximum ‘signal to noise’ ratio).

The results of the TQA formal error check are given in Table 3 below.

Table 3
Total number of errors detected Number of real errors detected Number of false error reports Number of real errors not detected
Trados 11 6 of 7 (all except #5) 5 (mostly such as ‘100’ translated as ‘сто’) 1 (#5, untranslatable)
WF 11 3 of 7 (##2,5,6) 8 (mostly such as ‘110km/h’ translated as ‘110 км/час’) 4 (## 1, 3, 4, 7)
QAD 20 6 of 7 (all except #5) 14 (all were number errors) 1 (#5, untranslatable)

As a result of carrying out this formal error check, the conclusions listed below in Table 4 can be drawn.

Table 4

Tool Strengths Weaknesses
Trados Detects the majority of real formal errors (6 of 7). Checks only bilingual .ttx files (does not check .doc and .rtf files directly).Does not detect errors in untranslatables (if they have not been included manually into the glossary).Not user-friendly.Learning curve is long.High number of false errors mostly associated with numbers translated by words (‘100’ translated as ‘сто’)
WF Highest user-friendliness.Learning curve is very short.The only TQA tool which automatically detects incorrect untranslables not included in the glossary. Detected only 3 of 7 errors.Does not detect real errors such as ## 1, 3, 4, 7.
QAD Detects the majority of real formal errors (6 of 7).Batch mode enables translation companies to check many files at a click. Failed to install on my desktop with Russian version of Windows XP (license code field was not displayed), but did install successfully on my notebook with the same OS.Verifies its own license code via an Internet connection and only works for eight hours after disconnecting from the Internet.Detects many false errors (14, mostly number values and formatting).Does not detect errors in untranslatables (if they have not been included manually into the glossary).Not user-friendly.Learning curve is long (compared to WF).

Detection of Terminology Errors
In order to test the terminology check features, I added four terminology errors to the test translation. First, I translated ‘simulator’ as ‘имитатор’, rather than ‘симулятор’, then I created glossaries containing one record only (simulator > симулятор) in the formats required by each TQA tool.
Note: Russian is an inflected language and my test translation contained various forms of the word ‘имитатор’.

The results of terminology check were as follows:

Table 5
Total number of errors detected Number of real errors detected Number of false errors reports Number of real errors not detected
Trados 6 4 2 0
WF 4 of 4 4 0 0
QAD no data no data no data no data

Comments on the data received:
Trados - The false errors detected by Trados were caused by fuzzy matches. On both occasions, Trados suggested the use of the glossary term ‘simulator/симулятор’ for the verb ‘simulate’. The user has no control over such situations. The only option is to ignore such false errors.
WF - This proved to be the most simple, accurate, user-friendly and controllable terminology checker. The user can set the level of fuzziness by using wildcards.
QAD - The copy of QAD installed on my notebook failed to perform the terminology check. During the Analyze step, the application returned the following error message: “A program exception occurred”.


Are TQA Tools Necessary for an Experienced and Diligent Translator?
As a freelance English-Russian translator with 27 years of experience, I always take pride in my human quality assurance methods. I proofread all my translations at least twice before delivery and frequently hire a proofreader or a technical expert to check my translations. Further information about my human quality assurance methods can be found on my website at www.erussiantranslations.com/Article9.htm.
Since 2000, I have translated about 700,000 words per year, and in the ten years before that I translated 56 novels. My sample translations were checked and approved by ATA, ITI and UTR. My clients are always happy with the quality of my translations.
However, are experience and human quality assurance methods enough to avoid formal and terminology mistakes? To find the answer I checked a 10,000-word translation I did in 2005, before I started to use TQA tools. I found two terminology and eight formal errors, which is enough to suggest that TQA tools may be as useful for experienced translators as they are for beginners.

Conclusions
1. TQA tools do not replace human editors/proofreaders, but only help them. First and foremost they help translators.

2. Each of the three TQA tools has its own strengths and weaknesses, as well as its preferable area of use.
· Trados is a good choice when you need to check .ttx files. Besides due to aggressive marketing Trados is a de facto industry standard.
· WF provides the best check of terminology and untranslatables. Furthermore, the WF developer offers the best technical support to users. The program is, in my opinion, the optimum choice for price-sensitive freelancers who do not want to spend many hours learning to use a complex software.
· QAD is the only tool enabling you to check Translation Memories saved in .tmx format and to use language dependent settings. Unlike current version of SDL Trados, QAD operates in batch mode (checks many files at a click) which is a big advantage for translation companies/agencies. Therefore QAD is probably the best choice for corporate users.

3. No matter how experienced the translator is and what human quality assurance methods s/he uses, TQA tools are able to decrease the number of mistakes and improve the overall quality of translation.

The results given above were achieved on my two PCs, a desktop and a notebook, both running the Russian version of Windows XP with SP and updates. Were the tests to be run on computers using a different operating system, there might be a slight variance in the results.

I would like to record my special thanks to Nathalie De Sutter for her invaluable contribution to this study.

* Original publication: Multilingual Magazine, January-February 2007, p. 22.