ChatGPT and Translation
ChatGPT, the latest chatbot developed by OpenAI is making many headlines at the moment; but what are the possibilities for ChatGPT and translation?
Background
ChatGPT4 (Generative Pre-trained Transformer 4) can be used for tasks such as language translation and speech recognition — although it was not designed for this purpose. ChatGPT is an NLP- (Natural Language Processing) based model which has been trained on large datasets of (mostly) conversations, and other human-generated content. The majority of ChatGPT’s training, around 93%, has been in English. OpenAI refer to their training approach as “unsupervised learning”. This method enables the model to learn a general understanding of the relationships between words, phrases, and sentences. It is capable of learning language structures and the patterns of different languages, as well as learning the context of conversations.
Find out more: Natural Language Processing
Can ChatGPT translate?
Chinese tech company Tencent did some research (Jan 23) into ChatGPT’s translation skills. According to the paper, ChatGPT performs “competitively” with commercial machine translation (MT) systems, but “struggles with low-resource or unrelated language pairs.”
Tencent compared ChatGPTs translations against three competitors: Google Translate, DeepL and their own system, Tencent Transmart. They used BLEU, chrF++, and TER: three automatic metrics for evaluating machine translation (MT) quality. The team said, “obtaining the translation results from ChatGPT is time-consuming since it can only be interacted with manually and can not respond to large batches. Thus, we randomly sample 50 sentences from each set for evaluation.” However, as well as general translation, they also tested ChatGPT on more specialised subjects (legal and biomedical); and by using source text that came mostly from social media, ie. user generated and frequently containing errors!
While ChatGPT performed “competitively” with Google Translate and DeepL for English-German translation, its BLEU score for English-Romanian translation was 46.4% lower than that of Google Translate.
ChatGPT was able to handle the specialised subject matter (the biomedical tests were from German-to-English) and also performed well with the informal source content. From this, we can assume that a lot of social media posts were used during the model’s training. The datasets used were almost certainly crawled from the web.
Over all, Tencent concluded that the four systems are comparable, although it is worth noting that ChatGPT is not trained specifically for translation. ChatGPT scored consistently higher when English was part of the language combination. However, it scored poorly when translating to, or from, languages with fewer online resources to learn from, such as Romanian or Japanese. It would seem that it simply hasn’t had as much exposure or training in these languages.
So it’s better at chatting?!
Over all Tencent concluded that while ChatGPT is still behind the major translation engines, it does perform surprisingly well. Especially on more informal texts, and in English. The authors believe this finding suggests that ChatGPT is “capable of generating more natural spoken languages than these commercial translation systems.” This will, perhaps, suggest a possible future focus.
Naturally, someone has asked ChatGPT’s opinion on this! The model responded “Yes, you can use ChatGPT for translation… However, it’s important to note that ChatGPT’s translation ability may not be as accurate or precise as a specialized translation tool or a professional translator. Additionally, ChatGPT may not be able to translate certain technical or domain-specific terms accurately. Nonetheless, ChatGPT can be a useful tool for basic language translation tasks.”
Implications for translators
Machine translation (MT) has had a great influence on the translation industry. Many translators have been forced to adapt how they work; but it has also provided customers with a wider variety of options to suit their needs. One new service is post-editing machine translation (PEMT). A a blend of machine and human translation, PEMT is more affordable and faster than human translation plus, it is more accurate than MT alone. As you can probably guess from the name, it involves an initial machine translation, followed by post-editing performed by a human translator and/or editor…
