How Good Is Machine Translation Really?

Ready for a day of linguistic mind reading, I opened my email one day to find a request for proofreading of a translation completed by a very strange translator indeed. Whoever it was had quite good grammar and at first seemed to be on the verge of knowing what they were doing. On my journey through texts unknown, however, I soon identified words that didn’t seem quite right. Was the client truly keen on referring to their toaster oven as a “miniature baking device”? Did their product quality scale really consist of the levels “Excellent, Very Good, Good, Courteous, and Penniless”? Where did this new guy dig up all this terminology? It turned out to be a machine translation and a very good machine translation indeed, compared to the texts produced by older translation machines. If it briefly managed to fool even me into thinking it was human writer, what must clients think of this new technology? Are they enamored with its grammatical charm or have they caught on to its fatal flaws? How good are the latest machine translations really? I set out on a yearlong journey of discovery to find out just that and I’m prepared to share my conclusions with you. Right here, right now. I have broken my analysis down to address four important areas of quality that translation machines have struggled with in the past.

Human vs Machine Translation: 4 Quality Indicators

1. Fluency

Good news: 
Machine translations no longer read as if they were written by aliens with severe brain trauma. Advanced translation machines are now producing texts that, in many cases, read very fluently. Clunky, nonsensical machine writing is going the way of the dinosaur.
Bad news: 
Similar to the aftermath of a conversation with an unsavory car salesman, you may be convinced the text is amazing until you take a closer look. It reads so well, the uninitiated run the risk of failing to evaluate translated content as critically as they otherwise would. Due to the many mistakes inherent in machine translation, this could have serious consequences down the road. It is imperative that the text be subjected to thorough bilingual proofreading by a language expert.

2. Writing Style

Good news:
If you asked a rock to write a document for you, machine translation would be the result. People who thrill at the sight of boring, robotic, clinical writing will love the writing style produced by machine translations.
Bad news:
Readers are guaranteed to alternately fall asleep, be bored to tears, or lose interest in your text. This is the kiss of death for marketing and journalistic texts, but it might not be a problem for technical texts. 

Translation machines also tend to directly translate writing styles used frequently in German that are considered bad practice in English writing. One example is the passive voice. English speakers love to read things that are written in a more direct, active, and lively way. German speakers, on the other hand, want to avoid saying who did what as much as possible, because then they would have to choose either the formal or informal version of the word “you”. Keeping it passive is keeping it simple in German.

3. Translation Accuracy

Good news:
On occasion, machine translation will produce a very accurate translation that just needs light editing. Machines seem particularly adept at translating texts with simple sentence structures, although blatant context-related mistakes still plague them. Ironically, the latest machine translation technology is terrible at translating the simplest of linguistic structures: lists. 
Bad news:
Most of the time, machine translation will produce a text that is moderately accurate or not accurate at all, requiring in-depth analysis and heavy editing. It will read as if a human wrote it, so rotten content may come out smelling like flowers. The most frustrating part is that it is almost impossible to predict if the machine will give you something spot-on or way out in left field. Assessing quality requires in-depth analysis of both the original and translation side-by-side.

4. Grammar

Good news:
The latest machine translation technology tends to have great grammar.
Bad news:
Good grammar does nothing for you if the content doesn’t make sense.

Evaluate Machine Translation Quality vs. Value with These 3 Questions

Clearly machine translation has come a long way, but it still is not on par with human translations. When is a good time to use these advanced machine translations? It boils down to one question: Will it deliver the results I need while saving me money? Before diving into the nitty-gritty details let’s acknowledge that any unedited machine translation you receive will contain substantial mistakes requiring correction by a translator. Time is money, as the saying goes, so the more time a translator has to spend modifying the text, the more their services will cost. Many clients have attempted to treat machine translation post-editing as a cheaper version of human translation that can be billed, like many human translations, by the word. Per-word billing might work for human translations, because the quality of the source text is generally an unchanging factor. Due to the sheer unpredictability of machine translation quality, however, a per-word approach to billing machine translation post-editing is woefully misguided. You could find yourself paying far more or far less than fair price. The best approach is to pay by the hour, which means the translation will be cheaper if the translator has to spend less time on it and will cost as much as a traditional translation at most. In order to get a rough idea of how much time a machine translation might save and the quality you might be able to expect, ask yourself the following questions:

1. Does writing style matter? Do you want a text that will engage your readers? Do you want something that sounds like a professional working in your field has written it? If you answered yes to either of the above questions, machine translation will not work for your document. Translating machines can be taught to use specialized terminology, but their writing style is generally very boring and clinical. Changing the writing style of a text requires rewriting the entire thing, so a machine translation is unlikely to save you much time, if any.

2. Do the sentences in my document have a very simple structure or are they more complex? Machine translation does very well with texts containing sentences with a very simple structure. This type of language is often present in step-by-step instructions or technical manuals where sentences such as “Affix the bracket to the back of the frame.” (verb/subject/prepositional phrase) abound. Once you start adding a wealth of adjectives or putting in subordinate clauses, your sentences will become more complex and harder for the machine to understand. If your document is written in a flashy and engaging way, as is often the case with marketing texts, it becomes highly likely that the machine will misunderstand and spit out grammatically correct, fluent-sounding nonsense.

3. Is there a lot of specialized terminology in the document? If you have a lot of specialized terminology in the document, you will have to make sure the machine has been trained to use your specific terminology before getting started. Even if you do train the machine to use your specialized terminology, there is no guarantee it will use it correctly. Take the German word “Sicherung” for example, which can mean “computer data backup, fuse, or security”. If you work at an IT security company, you may make use of all three of these specialized meanings at different times. Machine translations are not yet to the point where they can make refined differentiations based on context.

Contact me for a 15 min free project needs assessment.