One of the earliest goals for computers was the automatic translation of text from one language to another.
Automatic or machine translation is perhaps one of the most challenging artificial intelligence tasks given the fluidity of human language. Classically, rule-based systems were used for this task, which were replaced in the 1990s with statistical methods. More recently, deep neural network models achieve state-of-the-art results in a field that is aptly named neural machine translation.
In this post, you will discover the challenge of machine translation and the effectiveness of neural machine translation models.
After reading this post, you will know:
Machine translation is challenging given the inherent ambiguity and flexibility of human language.
Statistical machine translation replaces classical rule-based systems with models that learn to translate from examples.
Neural machine translation models fit a single model rather than a pipeline of fine-tuned models and currently achieve state-of-the-art results.
A Gentle Introduction to Neural Machine Translation Photo by Fabio Achilli, some rights reserved.
What is Machine Translation?
Machine translation is the task of automatically converting source text in one language to text in another language.
In a machine translation task, the input already consists of a sequence of symbols in some language, and the computer program must convert this into a sequence of symbols in another language.
Given a sequence of text in a source language, there is no one single best translation of that text to another language. This is because of the natural ambiguity and flexibility of human language. This makes the challenge of automatic machine translation difficult, perhaps one of the most difficult in artificial intelligence:
The fact is that accurate translation requires background knowledge in order to resolve ambiguity and establish the content of the sentence.
Classical machine translation methods often involve rules for converting text in the source language to the target language. The rules are often developed by linguists and may operate at the lexical, syntactic, or semantic level. This focus on rules gives the name to this area of study: Rule-based Machine Translation, or RBMT.
RBMT is characterized with the explicit use and manual creation of linguistically informed rules and representations.
The key limitations of the classical machine translation approaches are both the expertise required to develop the rules, and the vast number of rules and exceptions required.
Need help with Deep Learning for Text Data?
Take my free 7-day email crash course now (with code).
Click to sign-up and also get a free PDF Ebook version of the course.
What is Statistical Machine Translation?
Statistical machine translation, or SMT for short, is the use of statistical models that learn to translate text from a source language to a target language gives a large corpus of examples.
This task of using a statistical model can be stated formally as follows:
Given a sentence T in the target language, we seek the sentence S from which the translator produced T. We know that our chance of error is minimized by choosing that sentence S that is most probable given T. Thus, we wish to choose S so as to maximize Pr(S|T).
This formal specification makes the maximizing of the probability of the output sequence given the input sequence of text explicit. It also makes the notion of there being a suite of candidate translations explicit and the need for a search process or decoder to select the one most likely translation from the model’s output probability distribution.
Given a text in the source language, what is the most probable translation in the target language? […] how should one construct a statistical model that assigns high probabilities to “good” translations and low probabilities to “bad” translations?
The approach is data-driven, requiring only a corpus of examples with both source and target language text. This means linguists are not longer required to specify the rules of translation.
This approach does not need a complex ontology of interlingua concepts, nor does it need handcrafted grammars of the source and target languages, nor a hand-labeled treebank. All it needs is data—sample translations from which a translation model can be learned.
Quickly, the statistical approach to machine translation outperformed the classical rule-based methods to become the de-facto standard set of techniques.
Since the inception of the field at the end of the 1980s, the most popular models for statistical machine translation […] have been sequence-based. In these models, the basic units of translation are words or sequences of words […] These kinds of models are simple and effective, and they work well for man language pairs
The most widely used techniques were phrase-based and focus on translating sub-sequences of the source text piecewise.
Statistical Machine Translation (SMT) has been the dominant translation paradigm for decades. Practical implementations of SMT are generally phrase-based systems (PBMT) which translate sequences of words or phrases where the lengths may differ
Although effective, statistical machine translation methods suffered from a narrow focus on the phrases being translated, losing the broader nature of the target text. The hard focus on data-driven approaches also meant that methods may have ignored important syntax distinctions known by linguists. Finally, the statistical approaches required careful tuning of each module in the translation pipeline.
What is Neural Machine Translation?
Neural machine translation, or NMT for short, is the use of neural network models to learn a statistical model for machine translation.
The key benefit to the approach is that a single system can be trained directly on source and target text, no longer requiring the pipeline of specialized systems used in statistical machine learning.
Unlike the traditional phrase-based translation system which consists of many small sub-components that are tuned separately, neural machine translation attempts to build and train a single, large neural network that reads a sentence and outputs a correct translation.
Multilayer Perceptron neural network models can be used for machine translation, although the models are limited by a fixed-length input sequence where the output must be the same length.
These early models have been greatly improved upon recently through the use of recurrent neural networks organized into an encoder-decoder architecture that allow for variable length input and output sequences.
An encoder neural network reads and encodes a source sentence into a fixed-length vector. A decoder then outputs a translation from the encoded vector. The whole encoder–decoder system, which consists of the encoder and the decoder for a language pair, is jointly trained to maximize the probability of a correct translation given a source sentence.
Key to the encoder-decoder architecture is the ability of the model to encode the source text into an internal fixed-length representation called the context vector. Interestingly, once encoded, different decoding systems could be used, in principle, to translate the context into different languages.
… one model first reads the input sequence and emits a data structure that summarizes the input sequence. We call this summary the “context” C. […] A second mode, usually an RNN, then reads the context C and generates a sentence in the target language.
Although effective, the Encoder-Decoder architecture has problems with long sequences of text to be translated.
The problem stems from the fixed-length internal representation that must be used to decode each word in the output sequence.
The solution is the use of an attention mechanism that allows the model to learn where to place attention on the input sequence as each word of the output sequence is decoded.
Using a fixed-sized representation to capture all the semantic details of a very long sentence […] is very difficult. […] A more efficient approach, however, is to read the whole sentence or paragraph […], then to produce the translated words one at a time, each time focusing on a different part of he input sentence to gather the semantic details required to produce the next output word.
The encoder-decoder recurrent neural network architecture with attention is currently the state-of-the-art on some benchmark problems for machine translation. And this architecture is used in the heart of the Google Neural Machine Translation system, or GNMT, used in their Google Translate service.
https://translate.google.com
… current state-of-the-art machine translation systems are powered by models that employ attention.
Although effective, the neural machine translation systems still suffer some issues, such as scaling to larger vocabularies of words and the slow speed of training the models. There are the current areas of focus for large production neural translation systems, such as the Google system.
Three inherent weaknesses of Neural Machine Translation […]: its slower training and inference speed, ineffectiveness in dealing with rare words, and sometimes failure to translate all words in the source sentence.
Given a database of hundreds of million lines of short sentences with a limited number of 20000 words, do you think it is better to investigate a character-level RNN or a word-based RNN? What your intuition tells you?
Great post Jason, machinelearningmastery.com has become my new home for practical learning as I am starting to get a hold of some ML techniques. A suggestion from me that may help others..
Could you look putting together a simple tutorial developing ‘production’ ready models. For example, once a model has been developed how does one go about updating with new data and using the model for ongoing classification and prediction with new data. Some methods I have come stumbled across are manually updating new inputs into the code, manually updating new inputs into a .CSV file and for bigger datasets updating new data into .H5 file that the model recognises. This would help take the enormous learnings you offer to a level where the models become an ongoing tool for work or research….definitely something I have not yet mastered!
Sir, your post is very informative, and it gives me novel intuitions into this area.
Thank you very much for sharing your knowledge.
I’m completely new in this field.
Actually, I used to translate research papers and articles as my freelance job. So, I know nothing academic in the computer science field.
However, I have been interested in machine learning since 2 years ago. I worked with python and attended in some online courses. The whole field is full of joy, and challenges, of course.
I’m not an English native speaker, as it can be inferred from my english writing skills; sorry for that.
My first language is Persian (Farsi) and Persian has no ASCII representation. We use unicode charset, just like Arabic.
I was wondering if the aforementioned issue (lack of ASCII support), and the special properties of Persian language (e.g., its syntax which is way different from that of English, Spanish, French, or even Arabic) would affect NLP techniques and algorithms used in translation services like Google Translate?
I think google service translates English-Arabic pair so much better than English-Persian pair, and I feel like it has nothing to do with the volume of data (Persian texts, particularly) provided for the engine.
Also, I really like to develope a minimal machine translation project (for my research purposes), but I have no idea in terms of best algorithms, platforms, or techniques.
It would be useful if you share your opinion with us on this particular matter, and I would really appreciate that.
Again, thank you for the intuitive information you post here.
Best Wishes,
Ali from Persia
It is an interesting question and not something I know much about. Off the cuff, I would try to model the problem using unicode instead of chars, but I’d encourage you to read up in the literature how it is addressed generally.
Do you have any thoughts on the usefulness of NNT to the task of Bible translation? I consult for a Bible translation agency and am eager to show the application of NNT to the production of first draft translations in small and threatened languages of the world. FYI: the Hebrew Bible has only about 6,000+ discrete words, the Christian New Testament about the same amount. Many of the small and endangered languages have about the same number of discrete words.
Thanks for the post its very constructive and interesting, and it gives me good understanding but I got some questions on Neural Machine Translation
1 As I understand, In NMT we don’t need a separate language model, so how does a Decoder learns the grammar of the target language during predicting the next word, Or does a Seq2seq model do not need to learn grammar of a language ?
As luck would have it, I’m glad I came across your informative post. It is a good introduction–thanks to your good analysis and gentle approach (your headline got me here).
I have been translating from Japanese to English for about 40 years now, and since the beginning of MT, I do see surprising progress, but it still seems the “attention” or equivalent level of improvement in the Western languages is greater than for the Asian languages, as nuanced in some of the earlier posts to you in this blog. Goofy Google translations (Google Maps) made headlines recently in Japan, in addition to the continued cry for help with Chinese to English translations.
I perceive this is still simply a “cultural issue” and in time this too will improve; sorry for being in the wrong forum. It seems to me NMT providers should at least use qualified human checks before publishing (sometimes perverse) translations. “Well, this too will get better sooner or later.”
Probably one of the issues is that historically most of the investment came from the west (IBM, Google), and it’s easier to collaborate in a language that many understand (English).
However, I think you might want to have a look at DeepL, I have clients in China and Japan that are really happy with it (https://www.deepl.com/en/blog/20200319)
Thank you so much for the comprehensive explanation of how neural machine translation works, I have a question regarding probabilities learning; for commonly used words, pronouns, helping verbs, etc. Are they treated differently than domain-specific terms?
A valuable and well-structured overview of this fascinating field, for which many thanks.
I have great respect for the quantum leaps which neural nets have brought to Speech and Language Technology in general – my own specific interest has been real-time transcription. Nevertheless, I still believe that another very significant quantum leap is still required. That will involve bridging the huge capability gap between the neural net approach and the approach taken by a human being: the human approach is explicitly informed by “meaning”.
Whilst neural nets encode the “meaning-driven” human skill which has created example target texts from given source texts, they have no explicit concept of that meaning. Hence they may still “lose the thread”. An even if working at a sentence level rather than by word or by phrase, even a sentence is not normally an independent entity: sentences are usually part of a self-consistent text which has been created for a purpose – to convey meaning from one human to another.
I’d be interested in your comments on this, and on how the next quantum leap to address the challenge of respecting context and meaning in SLT in general might be taken?
Thank you. I am asked to write a paper and I will mainly be discussing modern translation tools and their impact on the process and product of translation. Also the challenges of machine translation. So, for those two ideas which translation tools fit the ideas to be examined?
In your book “Deep Learning for Natural Language Processing” chapter 15, the predictions seemed not be influenced by the number of epochs. Increasing the number of epochs to 40 still gave me a wrong prediction:
1
2
3
4
5
6
7
8
Vocabulary size:25768
Maximum length:1317
Train Accuracy:100.00
Test Accuracy:90.00
Review:[Everyone will enjoy thisfilm.Ilove it,recommended!]
Sentiment:POSITIVE(72.091%)
Review:[Thisisabad movie.Donotwatch it.It sucks.]
Sentiment:POSITIVE(67.981%)
However increasing the level of detail of the movie review examples gave me a good prediction:
1
2
3
4
5
6
7
8
Vocabulary size:25768
Maximum length:1317
Train Accuracy:100.00
Test Accuracy:90.00
Review:[Everyone will enjoy thisfilm.Ilove it,recommended!.Ienjoy watching thismovie.Iwas very happy during two hours]
Sentiment:POSITIVE(75.594%)
Review:[Thisisabad movie.Donotwatch it.It sucks.Idid notlike the actors,the decoration was very poor.Ilose my time]
Sentiment:NEGATIVE(56.763%)
This is a confirmation of your remark “this may be the two contrived reviews are very short and the model is expecting sequences of 1,000 or more words.”
I am looking for a way to set up machine translation of older versions of languages to their modern equivalents. For example, I would like to be able to translate an Old French text into Modern French. Any help or suggestions would be appreciated.
your post was really very informative.
where can i get the actual code for
sequence to sequence models
encoder decoder model
encoder decoder model with attention
Also any suggestion for hands on neural machine translation course.
Given a database of hundreds of million lines of short sentences with a limited number of 20000 words, do you think it is better to investigate a character-level RNN or a word-based RNN? What your intuition tells you?
Start with words and go to char to see if it can lift skill or simplify the model.
I shared your interesting article on my Fb page European Terminology
Thanks!
Great post Jason, machinelearningmastery.com has become my new home for practical learning as I am starting to get a hold of some ML techniques. A suggestion from me that may help others..
Could you look putting together a simple tutorial developing ‘production’ ready models. For example, once a model has been developed how does one go about updating with new data and using the model for ongoing classification and prediction with new data. Some methods I have come stumbled across are manually updating new inputs into the code, manually updating new inputs into a .CSV file and for bigger datasets updating new data into .H5 file that the model recognises. This would help take the enormous learnings you offer to a level where the models become an ongoing tool for work or research….definitely something I have not yet mastered!
Great suggestion, thanks.
See this post on final models:
https://machinelearningmastery.com/train-final-machine-learning-model/
And this post on models in production:
https://machinelearningmastery.com/deploy-machine-learning-model-to-production/
I hope that helps as a start.
Sir, your post is very informative, and it gives me novel intuitions into this area.
Thank you very much for sharing your knowledge.
I’m completely new in this field.
Actually, I used to translate research papers and articles as my freelance job. So, I know nothing academic in the computer science field.
However, I have been interested in machine learning since 2 years ago. I worked with python and attended in some online courses. The whole field is full of joy, and challenges, of course.
I’m not an English native speaker, as it can be inferred from my english writing skills; sorry for that.
My first language is Persian (Farsi) and Persian has no ASCII representation. We use unicode charset, just like Arabic.
I was wondering if the aforementioned issue (lack of ASCII support), and the special properties of Persian language (e.g., its syntax which is way different from that of English, Spanish, French, or even Arabic) would affect NLP techniques and algorithms used in translation services like Google Translate?
I think google service translates English-Arabic pair so much better than English-Persian pair, and I feel like it has nothing to do with the volume of data (Persian texts, particularly) provided for the engine.
Also, I really like to develope a minimal machine translation project (for my research purposes), but I have no idea in terms of best algorithms, platforms, or techniques.
It would be useful if you share your opinion with us on this particular matter, and I would really appreciate that.
Again, thank you for the intuitive information you post here.
Best Wishes,
Ali from Persia
It is an interesting question and not something I know much about. Off the cuff, I would try to model the problem using unicode instead of chars, but I’d encourage you to read up in the literature how it is addressed generally.
Dear Jason
Do you have any thoughts on the usefulness of NNT to the task of Bible translation? I consult for a Bible translation agency and am eager to show the application of NNT to the production of first draft translations in small and threatened languages of the world. FYI: the Hebrew Bible has only about 6,000+ discrete words, the Christian New Testament about the same amount. Many of the small and endangered languages have about the same number of discrete words.
I don’t know. Perhaps prototype some models and see how well it performs.
Hi Jason, would NMT a good method to do code translation from one language to another: let’s say from R to Python? Thanks
Maybe.
Hello Jason,
Thanks for the post its very constructive and interesting, and it gives me good understanding but I got some questions on Neural Machine Translation
1 As I understand, In NMT we don’t need a separate language model, so how does a Decoder learns the grammar of the target language during predicting the next word, Or does a Seq2seq model do not need to learn grammar of a language ?
It learns a conditional probabilistic model, e.g. output the next word conditioned on the input and on the words generated so far.
Hello Jason:
As luck would have it, I’m glad I came across your informative post. It is a good introduction–thanks to your good analysis and gentle approach (your headline got me here).
I have been translating from Japanese to English for about 40 years now, and since the beginning of MT, I do see surprising progress, but it still seems the “attention” or equivalent level of improvement in the Western languages is greater than for the Asian languages, as nuanced in some of the earlier posts to you in this blog. Goofy Google translations (Google Maps) made headlines recently in Japan, in addition to the continued cry for help with Chinese to English translations.
I perceive this is still simply a “cultural issue” and in time this too will improve; sorry for being in the wrong forum. It seems to me NMT providers should at least use qualified human checks before publishing (sometimes perverse) translations. “Well, this too will get better sooner or later.”
Great comment Ben, thanks for sharing.
Regarding Chinese translation, I would expect that systems by Baidu may be more effective thatn those by google.
Probably one of the issues is that historically most of the investment came from the west (IBM, Google), and it’s easier to collaborate in a language that many understand (English).
However, I think you might want to have a look at DeepL, I have clients in China and Japan that are really happy with it (https://www.deepl.com/en/blog/20200319)
Thank you so much for the comprehensive explanation of how neural machine translation works, I have a question regarding probabilities learning; for commonly used words, pronouns, helping verbs, etc. Are they treated differently than domain-specific terms?
Thanks!
You can handle them differently if you want, or remove them completely if needed.
A valuable and well-structured overview of this fascinating field, for which many thanks.
I have great respect for the quantum leaps which neural nets have brought to Speech and Language Technology in general – my own specific interest has been real-time transcription. Nevertheless, I still believe that another very significant quantum leap is still required. That will involve bridging the huge capability gap between the neural net approach and the approach taken by a human being: the human approach is explicitly informed by “meaning”.
Whilst neural nets encode the “meaning-driven” human skill which has created example target texts from given source texts, they have no explicit concept of that meaning. Hence they may still “lose the thread”. An even if working at a sentence level rather than by word or by phrase, even a sentence is not normally an independent entity: sentences are usually part of a self-consistent text which has been created for a purpose – to convey meaning from one human to another.
I’d be interested in your comments on this, and on how the next quantum leap to address the challenge of respecting context and meaning in SLT in general might be taken?
Thanks.
I think things have come a long way even since I wrote this article. No leaps required I think, just incremental improvement.
Newer methods don’t seem to lose the thread anymore even after long input sequences.
Thank you. I am asked to write a paper and I will mainly be discussing modern translation tools and their impact on the process and product of translation. Also the challenges of machine translation. So, for those two ideas which translation tools fit the ideas to be examined?
I recommend performing a literature review.
Thank you.
You’re welcome.
Dear Jason,
In your book “Deep Learning for Natural Language Processing” chapter 15, the predictions seemed not be influenced by the number of epochs. Increasing the number of epochs to 40 still gave me a wrong prediction:
However increasing the level of detail of the movie review examples gave me a good prediction:
This is a confirmation of your remark “this may be the two contrived reviews are very short and the model is expecting sequences of 1,000 or more words.”
Kind regards,
Dominique
Great finding, thank you!
Hello,
I am looking for a way to set up machine translation of older versions of languages to their modern equivalents. For example, I would like to be able to translate an Old French text into Modern French. Any help or suggestions would be appreciated.
Thanks!
Try seq2seq model first. That should be the easiest.
your post was really very informative.
where can i get the actual code for
sequence to sequence models
encoder decoder model
encoder decoder model with attention
Also any suggestion for hands on neural machine translation course.
i will be highly grateful to you for your help.
Thanks a lot !
A simple model is here. Do you think this helps? https://machinelearningmastery.com/develop-neural-machine-translation-system-keras/