How to Develop a Neural Machine Translation System from Scratch

Last Updated on

Develop a Deep Learning Model to Automatically
Translate from German to English in Python with Keras, Step-by-Step.

Machine translation is a challenging task that traditionally involves large statistical models developed using highly sophisticated linguistic knowledge.

Neural machine translation is the use of deep neural networks for the problem of machine translation.

In this tutorial, you will discover how to develop a neural machine translation system for translating German phrases to English.

After completing this tutorial, you will know:

  • How to clean and prepare data ready to train a neural machine translation system.
  • How to develop an encoder-decoder model for machine translation.
  • How to use a trained model for inference on new input phrases and evaluate the model skill.

Discover how to develop deep learning models for text classification, translation, photo captioning and more in my new book, with 30 step-by-step tutorials and full source code.

Let’s get started.

  • Update Apr/2019: Fixed bug in the calculation of BLEU score (Zhongpu Chen).
How to Develop a Neural Machine Translation System in Keras

How to Develop a Neural Machine Translation System in Keras
Photo by Björn Groß, some rights reserved.

Tutorial Overview

This tutorial is divided into 4 parts; they are:

  1. German to English Translation Dataset
  2. Preparing the Text Data
  3. Train Neural Translation Model
  4. Evaluate Neural Translation Model

Python Environment

This tutorial assumes you have a Python 3 SciPy environment installed.

You must have Keras (2.0 or higher) installed with either the TensorFlow or Theano backend.

The tutorial also assumes you have NumPy and Matplotlib installed.

If you need help with your environment, see this post:

A GPU is not require for thus tutorial, nevertheless, you can access GPUs cheaply on Amazon Web Services. Learn how in this tutorial:

Let’s dive in.

Need help with Deep Learning for Text Data?

Take my free 7-day email crash course now (with code).

Click to sign-up and also get a free PDF Ebook version of the course.

Start Your FREE Crash-Course Now

German to English Translation Dataset

In this tutorial, we will use a dataset of German to English terms used as the basis for flashcards for language learning.

The dataset is available from the ManyThings.org website, with examples drawn from the Tatoeba Project. The dataset is comprised of German phrases and their English counterparts and is intended to be used with the Anki flashcard software.

The page provides a list of many language pairs, and I encourage you to explore other languages:

The dataset we will use in this tutorial is available for download here:

Download the dataset to your current working directory and decompress; for example:

You will have a file called deu.txt that contains 152,820 pairs of English to German phases, one pair per line with a tab separating the language.

For example, the first 5 lines of the file look as follows:

We will frame the prediction problem as given a sequence of words in German as input, translate or predict the sequence of words in English.

The model we will develop will be suitable for some beginner German phrases.

Preparing the Text Data

The next step is to prepare the text data ready for modeling.

If you are new to cleaning text data, see this post:

Take a look at the raw data and note what you see that we might need to handle in a data cleaning operation.

For example, here are some observations I note from reviewing the raw data:

  • There is punctuation.
  • The text contains uppercase and lowercase.
  • There are special characters in the German.
  • There are duplicate phrases in English with different translations in German.
  • The file is ordered by sentence length with very long sentences toward the end of the file.

Did you note anything else that could be important?
Let me know in the comments below.

A good text cleaning procedure may handle some or all of these observations.

Data preparation is divided into two subsections:

  1. Clean Text
  2. Split Text

1. Clean Text

First, we must load the data in a way that preserves the Unicode German characters. The function below called load_doc() will load the file as a blob of text.

Each line contains a single pair of phrases, first English and then German, separated by a tab character.

We must split the loaded text by line and then by phrase. The function to_pairs() below will split the loaded text.

We are now ready to clean each sentence. The specific cleaning operations we will perform are as follows:

  • Remove all non-printable characters.
  • Remove all punctuation characters.
  • Normalize all Unicode characters to ASCII (e.g. Latin characters).
  • Normalize the case to lowercase.
  • Remove any remaining tokens that are not alphabetic.

We will perform these operations on each phrase for each pair in the loaded dataset.

The clean_pairs() function below implements these operations.

Finally, now that the data has been cleaned, we can save the list of phrase pairs to a file ready for use.

The function save_clean_data() uses the pickle API to save the list of clean text to file.

Pulling all of this together, the complete example is listed below.

Running the example creates a new file in the current working directory with the cleaned text called english-german.pkl.

Some examples of the clean text are printed for us to evaluate at the end of the run to confirm that the clean operations were performed as expected.

2. Split Text

The clean data contains a little over 150,000 phrase pairs and some of the pairs toward the end of the file are very long.

This is a good number of examples for developing a small translation model. The complexity of the model increases with the number of examples, length of phrases, and size of the vocabulary.

Although we have a good dataset for modeling translation, we will simplify the problem slightly to dramatically reduce the size of the model required, and in turn the training time required to fit the model.

You can explore developing a model on the fuller dataset as an extension; I would love to hear how you do.

We will simplify the problem by reducing the dataset to the first 10,000 examples in the file; these will be the shortest phrases in the dataset.

Further, we will then stake the first 9,000 of those as examples for training and the remaining 1,000 examples to test the fit model.

Below is the complete example of loading the clean data, splitting it, and saving the split portions of data to new files.

Running the example creates three new files: the english-german-both.pkl that contains all of the train and test examples that we can use to define the parameters of the problem, such as max phrase lengths and the vocabulary, and the english-german-train.pkl and english-german-test.pkl files for the train and test dataset.

We are now ready to start developing our translation model.

Train Neural Translation Model

In this section, we will develop the neural translation model.

If you are new to neural translation models, see the post:

This involves both loading and preparing the clean text data ready for modeling and defining and training the model on the prepared data.

Let’s start off by loading the datasets so that we can prepare the data. The function below named load_clean_sentences() can be used to load the train, test, and both datasets in turn.

We will use the “both” or combination of the train and test datasets to define the maximum length and vocabulary of the problem.

This is for simplicity. Alternately, we could define these properties from the training dataset alone and truncate examples in the test set that are too long or have words that are out of the vocabulary.

We can use the Keras Tokenize class to map words to integers, as needed for modeling. We will use separate tokenizer for the English sequences and the German sequences. The function below-named create_tokenizer() will train a tokenizer on a list of phrases.

Similarly, the function named max_length() below will find the length of the longest sequence in a list of phrases.

We can call these functions with the combined dataset to prepare tokenizers, vocabulary sizes, and maximum lengths for both the English and German phrases.

We are now ready to prepare the training dataset.

Each input and output sequence must be encoded to integers and padded to the maximum phrase length. This is because we will use a word embedding for the input sequences and one hot encode the output sequences The function below named encode_sequences() will perform these operations and return the result.

The output sequence needs to be one-hot encoded. This is because the model will predict the probability of each word in the vocabulary as output.

The function encode_output() below will one-hot encode English output sequences.

We can make use of these two functions and prepare both the train and test dataset ready for training the model.

We are now ready to define the model.

We will use an encoder-decoder LSTM model on this problem. In this architecture, the input sequence is encoded by a front-end model called the encoder then decoded word by word by a backend model called the decoder.

The function define_model() below defines the model and takes a number of arguments used to configure the model, such as the size of the input and output vocabularies, the maximum length of input and output phrases, and the number of memory units used to configure the model.

The model is trained using the efficient Adam approach to stochastic gradient descent and minimizes the categorical loss function because we have framed the prediction problem as multi-class classification.

The model configuration was not optimized for this problem, meaning that there is plenty of opportunity for you to tune it and lift the skill of the translations. I would love to see what you can come up with.

For more advice on configuring neural machine translation models, see the post:

Finally, we can train the model.

We train the model for 30 epochs and a batch size of 64 examples.

We use checkpointing to ensure that each time the model skill on the test set improves, the model is saved to file.

We can tie all of this together and fit the neural translation model.

The complete working example is listed below.

Running the example first prints a summary of the parameters of the dataset such as vocabulary size and maximum phrase lengths.

Next, a summary of the defined model is printed, allowing us to confirm the model configuration.

A plot of the model is also created providing another perspective on the model configuration.

Plot of Model Graph for NMT

Plot of Model Graph for NMT

Next, the model is trained.

Each epoch takes about 30 seconds on modern CPU hardware; no GPU is required.

During the run, the model will be saved to the file model.h5, ready for inference in the next step.

Evaluate Neural Translation Model

We will evaluate the model on the train and the test dataset.

The model should perform very well on the train dataset and ideally have been generalized to perform well on the test dataset.

Ideally, we would use a separate validation dataset to help with model selection during training instead of the test set. You can try this as an extension.

The clean datasets must be loaded and prepared as before.

Next, the best model saved during training must be loaded.

Evaluation involves two steps: first generating a translated output sequence, and then repeating this process for many input examples and summarizing the skill of the model across multiple cases.

Starting with inference, the model can predict the entire output sequence in a one-shot manner.

This will be a sequence of integers that we can enumerate and lookup in the tokenizer to map back to words.

The function below, named word_for_id(), will perform this reverse mapping.

We can perform this mapping for each integer in the translation and return the result as a string of words.

The function predict_sequence() below performs this operation for a single encoded source phrase.

Next, we can repeat this for each source phrase in a dataset and compare the predicted result to the expected target phrase in English.

We can print some of these comparisons to screen to get an idea of how the model performs in practice.

We will also calculate the BLEU scores to get a quantitative idea of how well the model has performed.

You can learn more about the BLEU score here:

The evaluate_model() function below implements this, calling the above predict_sequence() function for each phrase in a provided dataset.

We can tie all of this together and evaluate the loaded model on both the training and test datasets.

The complete code listing is provided below.

Running the example first prints examples of source text, expected and predicted translations, as well as scores for the training dataset, followed by the test dataset.

Your specific results will differ given the random shuffling of the dataset and the stochastic nature of neural networks.

Looking at the results for the test dataset first, we can see that the translations are readable and mostly correct.

For example: “ich bin brillentrager” was correctly translated to “i wear glasses“.

We can also see that the translations were not perfect, with “hab ich nicht recht” translated to “am i fat” instead of the expected “am i wrong“.

We can also see the BLEU-4 score of about 0.45, which provides an upper bound on what we might expect from this model.

Looking at the results on the test set, do see readable translations, which is not an easy task.

For example, we see “tom erblasste” correctly translated to “tom turned pale“.

We also see some poor translations and a good case that the model could suffer from further tuning, such as “ich brauche erste hilfe” translated as “i need them you” instead of the expected “i need first aid“.

A BLEU-4 score of about 0.153 was achieved, providing a baseline skill to improve upon with further improvements to the model.

Extensions

This section lists some ideas for extending the tutorial that you may wish to explore.

  • Data Cleaning. Different data cleaning operations could be performed on the data, such as not removing punctuation or normalizing case, or perhaps removing duplicate English phrases.
  • Vocabulary. The vocabulary could be refined, perhaps removing words used less than 5 or 10 times in the dataset and replaced with “unk“.
  • More Data. The dataset used to fit the model could be expanded to 50,000, 100,000 phrases, or more.
  • Input Order. The order of input phrases could be reversed, which has been reported to lift skill, or a Bidirectional input layer could be used.
  • Layers. The encoder and/or the decoder models could be expanded with additional layers and trained for more epochs, providing more representational capacity for the model.
  • Units. The number of memory units in the encoder and decoder could be increased, providing more representational capacity for the model.
  • Regularization. The model could use regularization, such as weight or activation regularization, or the use of dropout on the LSTM layers.
  • Pre-Trained Word Vectors. Pre-trained word vectors could be used in the model.
  • Recursive Model. A recursive formulation of the model could be used where the next word in the output sequence could be conditional on the input sequence and the output sequence generated so far.

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Summary

In this tutorial, you discovered how to develop a neural machine translation system for translating German phrases to English.

Specifically, you learned:

  • How to clean and prepare data ready to train a neural machine translation system.
  • How to develop an encoder-decoder model for machine translation.
  • How to use a trained model for inference on new input phrases and evaluate the model skill.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Note: This post is an excerpt chapter from: “Deep Learning for Natural Language Processing“. Take a look, if you want more step-by-step tutorials on getting the most out of deep learning methods when working with text data.

Develop Deep Learning models for Text Data Today!

Deep Learning for Natural Language Processing

Develop Your Own Text models in Minutes

...with just a few lines of python code

Discover how in my new Ebook:
Deep Learning for Natural Language Processing

It provides self-study tutorials on topics like:
Bag-of-Words, Word Embedding, Language Models, Caption Generation, Text Translation and much more...

Finally Bring Deep Learning to your Natural Language Processing Projects

Skip the Academics. Just Results.

See What's Inside

379 Responses to How to Develop a Neural Machine Translation System from Scratch

  1. Klaas January 10, 2018 at 7:53 am #

    amazing work again. One Question. Do you have a seperate tutorial where you explain the LSTM layers (Timedistributed, Repeatvector,…)?

  2. Mohamed January 10, 2018 at 1:51 pm #

    Your tutorials are amazing indeed. Thank you!
    Hope you will have the time to work on the Extensions lists above. This will complete this amazing tutorial.

    Thanks again!

  3. Richard January 12, 2018 at 5:52 am #

    Brilliant, thanks Jason. I’m looking forward to giving this a try.

  4. Parul January 14, 2018 at 7:47 am #

    hey i want to know one thing that if we are giving english to german translations to the model for training 9000 and for testing 1000.. then what is the encoder decoder model is actually doing ..as we are giving everything to the model at the time of testing.

    • Jason Brownlee January 15, 2018 at 6:54 am #

      The model is not given the answer, it must translate new examples.

      Perhaps I don’t follow your question?

      • Barnabas March 13, 2019 at 10:18 pm #

        Then how do i enter the example? on which line are you picking it

  5. abkul orto January 15, 2018 at 5:38 pm #

    Hi Jason,

    I am regular reader of your articles and purchased books.i want to work on translation of a local language to english.kindly advice on the steps.

    thanks you

  6. kannu January 20, 2018 at 4:50 am #

    # prepare regex for char filtering
    re_print = re.compile(‘[^%s]’ % re.escape(string.printable))

    can u please explain me the meaning of this code for ex what is string.printable actually doing and what is the meaning of (‘[^%s]’

    • Jason Brownlee January 20, 2018 at 8:24 am #

      I am selecting “not the printable characters”.

      You can learn more about regex from a good book on Python.

  7. Harish Yadav January 20, 2018 at 9:22 pm #

    Excellent explanation i would say!!!! damn good !!!looking to develop text-phonemes with your model !!!

  8. Drishty January 23, 2018 at 8:28 pm #

    Hi , Jason your wok is amazing and while i was doing this code i found this and i want to know i it’s required ti reshape the sequence ? and what sequence.shape[0],sequence.shape[1] is doing.
    and why we need the vocab size ?
    y = y.reshape(sequences.shape[0], sequences.shape[1], vocab_size)

  9. Drishty January 23, 2018 at 8:29 pm #

    *want to know why it’s required to reshape the sequence ? and what

    • Jason Brownlee January 24, 2018 at 9:55 am #

      We must ensure that the data is the correct shape that is expected by the model, e.g. 2d for MLPs, 3D for LSTMs, etc.

  10. firoz January 24, 2018 at 4:41 am #

    hi ,

    i wanted to ask tyou why we have not done one-hot encoding for text in german.?

    • Jason Brownlee January 24, 2018 at 9:58 am #

      The input data is integer encoded and passed through a word embedding. No need to one hot encode in this case.

  11. ravi January 25, 2018 at 4:59 am #

    hello sir,

    over here the load_model is not defined .

    thank you .

    • Jason Brownlee January 25, 2018 at 5:58 am #

      from keras.models import load_model

    • ravi January 25, 2018 at 6:17 am #

      can please tell me where the

      translation = model.predict(source, verbose=0)

      error: source is not deifined

      • Jason Brownlee January 25, 2018 at 9:07 am #

        Sorry, I have not seen that error. Perhaps try copying the entire example at the end of the post?

  12. asheesh January 25, 2018 at 6:36 am #

    while running above code i am facing memory error in to_categorical function. I am doing translation for english to hindi. Pls give any suggestion.

    • Jason Brownlee January 25, 2018 at 9:09 am #

      Perhaps try updating Keras?
      Perhaps try modifying the code to use progressive loading?
      Perhaps try running on AWS with an instance that has more RAM?

  13. Harish Yadav January 25, 2018 at 11:20 pm #

    please do a model on attention with gru and beam search

  14. Harish Yadav January 30, 2018 at 4:13 pm #

    i have used bidirectional lstm,got a better result…i want to improve more …but i dont know how to implement attention layer in keras…could you please help me out…

  15. hayet January 31, 2018 at 9:48 pm #

    Hi, I want know why you use model.add(RepeatVector(tar_timesteps))?

  16. hayet February 2, 2018 at 12:11 am #

    is it possible to calculate the NMT model score with this method

    model.compile(optimizer=’adam’, loss=’categorical_crossentropy’, metrics=[‘accuracy’])

    scores = model.evaluate(testX,testY)

    • Jason Brownlee February 2, 2018 at 8:20 am #

      It will estimate accuracy and loss, but not bot give you any insight into the skill of the NMT on text data.

  17. Darren February 20, 2018 at 5:03 am #

    Hi Jason, brilliant article!

    Just a quick question, when you configure the encoder-decoder model, there seems no inference model as you mentioned in your previous articles? If this model has achieved what inference model did, in which layer? If not, how does it compare to the suite of train model, inference-encoder model and inference-decoder model? Thank you so much!

  18. Jakobe February 25, 2018 at 4:45 am #

    Does text_to_sequences encode data ?
    according to the documentation it just transform texts to a list of sequences

    • Jason Brownlee February 25, 2018 at 7:45 am #

      Yes, it encodes words in text to integers.

      • Jakobe March 6, 2018 at 9:38 am #

        Could you verify This documentation. It is mentionned that text_to_sequences return STR.
        I am confusing right now.
        https://keras.io/preprocessing/text/

        • Jason Brownlee March 6, 2018 at 2:55 pm #

          For “texts_to_sequences” on Tokenizer it says:

          “Return: list of sequences (one per text input).”

  19. Emil March 6, 2018 at 10:41 am #

    ImportError: cannot import name ‘corpus_bleu’
    Did anyone have an idea about this error.

  20. Dirck March 10, 2018 at 8:54 pm #

    By following your tutorial, I was able to find BLEU scores on test dataset as follow :
    BLEU-1: 0.069345
    BLEU-2: 0.255634
    BLEU-3: 0.430785
    BLEU-4: 0.490818

    So we can notice that they are very close to the scores on train dataset.
    Is it about overfitting or it is a normal behavior ?

    • Jason Brownlee March 11, 2018 at 6:25 am #

      Nice work!

      Similar scores on train and test is a sign of a stable model. If the skill is poor, it might be a stable but underfit model.

  21. vikas dixit March 10, 2018 at 11:12 pm #

    Hello sir, you are using test data as validation data. This means model has seen test data during training phase only. I think test data is kept separated. Am I right?? If yes please explain logic behind it.

  22. sindhu reddy March 20, 2018 at 2:32 am #

    Hello sir, great explanation. everything works well with the given corpus.when i am using the own corpus it says .pkl file is not encoded in utf-8.

    can you please share the the encoding of the text files used for the above project?

    It is giving following error
    —————————————————————————
    IndexError Traceback (most recent call last)
    in ()
    65 # spot check
    66 for i in range(100):
    —> 67 print(‘[%s] => [%s]’ % (clean_pairs[i,0], clean_pairs[i,1]))

    IndexError: too many indices for array

    Kindly help

    • Jason Brownlee March 20, 2018 at 6:26 am #

      Perhaps double check you are using Python 3?

      • sindhu reddy March 20, 2018 at 6:30 pm #

        yes i am using python 3.5

        • Jason Brownlee March 21, 2018 at 6:31 am #

          Are you able to confirm that all other libs are up to date and that you copied all of the code from the example?

  23. sindhu reddy March 21, 2018 at 5:06 pm #

    yes jason i have updated all the libraries. it is working completely fine for the deu,txt file .
    when ever i use my own text file it is giving the following error.

    can you kindly tell what formatting is used in text file.

    Thanks

    • Jason Brownlee March 22, 2018 at 6:19 am #

      As stated in the post, the format is “Tab-delimited Bilingual Sentence Pairs”.

  24. Jigyasa Sakhuja March 24, 2018 at 3:47 am #

    hi Jason i am a fan of yours and i have implemented this machine translation and it was awesome i got all the results perfectly .. now i wanted to generate code using natural language by using RNN.. and when i am reading my file which is of declartaion and docstrings it is not showing as it is the ouput .. like it should show the declarations but it is showing something like x00/x00/x00/x00/x00/x00/x00/x00/x00/x00/x00/x00/x00/x00/x00/x00/x00/x00/x00/x00/

    but it should show
    if cint(frappe.db.get_single_value(u’System DCSP Settings’, u’setup_complete’)):

  25. sasi March 28, 2018 at 5:59 pm #

    In your data x is English and y is german… but in the code x is German, and y is english… why that difference????????????

    • Jason Brownlee March 29, 2018 at 6:31 am #

      We are translating from German (X) to English (Y).

      You can learn the reverse if you prefer. I chose not to because my english is better than my german.

  26. Kam March 29, 2018 at 8:48 pm #

    Hi,
    I am trying to use pre trained word embeddings to make translation.
    But, after making some researrch I found that pre-trained word embeddings are just only user for initialize encoder and decoder and also we nedd only the src embeddings.
    So, for the moment I am confused.
    Normally, must we provide source and target embeddings to the algorithme ?
    Please if they are some documentation or links about this topic.

    • Jason Brownlee March 30, 2018 at 6:37 am #

      Not sure I follow, what do you mean exactly?

      You can use a pre-trained embedding. This is separate from needing to have input and output data pairs to train the model.

  27. Sindhura April 4, 2018 at 3:57 am #

    Regarding recursive model in extensions, isn’t it already implemented in the current code? Because the decoder part is lstm and is lstm output of one unit is fed to the next unit.

  28. Max b April 17, 2018 at 3:55 am #

    “be stolen returned” is my systems translation of “vielen dank jason”, which ist supposed to mean: Thank you so much Jason!

    This post helped me a lot and I’ll now continue to tune it. Keep up the awesome work!

  29. suraj April 17, 2018 at 7:38 pm #

    In machine translation why we need vocabulary with the english text and german text …?

    • Jason Brownlee April 18, 2018 at 8:02 am #

      We need to limit the number of words that we model, it cannot be unbounded, at least in the way I’m choosing to model the problem.

      • michael April 20, 2018 at 12:24 am #

        That suggests that it can be unbounded if you model it in a different way.

  30. AlgoP April 24, 2018 at 11:42 pm #

    Hi Jason,
    I have just tested the clean_pairs method against ENG-PL set provided on the same website.One of the characters does not print on the screen( ‘all the other non ASCII chars are converted correctly), it is ignored as per this line I guess:

    I did an experiment with replacing the above with line = normalize(‘NFD’, line).encode(‘utf-8’, ‘ignore’), but there is no difference between these two in results.I am not sure why this is happening as it is only one letter.Also,( I assume your chose was ascii as you built a German to English translator am I correct?).Could you plase share your thoughts, if possible?

    • Jason Brownlee April 25, 2018 at 6:33 am #

      Perhaps you’re able to inspect the text or search the text for non-ascii chars to see what the offending characters are?

      This might give you insight into what is going on.

    • AlgoP April 25, 2018 at 6:44 am #

      I am working on it -it looks like it may be the issue with re.escape method rather than with encoding itself.

  31. Johny May 1, 2018 at 9:49 pm #

    Does removing punctuation not preventing the model to be used to predict a paragraph? How can you evaluate it with one sentence or paragraph not in the test set?

    • Jason Brownlee May 2, 2018 at 5:39 am #

      You can provide data to the model and make a prediction.

      call the predict_sequence() function we wrote above.

  32. Umesh May 1, 2018 at 10:53 pm #

    From Keras. Proprocessing. Text import Tokenizer
    ..
    Does not woking after installing keras..
    ..
    It’s says that no module named tensorflow
    ..
    I have windows 32 it machine.
    ..
    Your article very good…!
    .
    But I can’t process ahead due to this problem!

  33. Jundong May 4, 2018 at 9:53 am #

    Thank you for your article, Jason!

    I have one question about the difference between your implementation and the Keras Tutorial “https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html”. It seems to me that, there is a ‘teaching forcing’ element in the “Keras Tutorial” using “target” (offset by one step) as “decoder input data”. This element is not presented in your model. My question is: is it necessary? or you just use “RepeatedVector” and “TimeDistributed” to implement the similar function?

    Thank you!

  34. Beay May 5, 2018 at 9:08 pm #

    Great help Jason, thank you one more time, i want to ask you:

    How can i implement bidirectional lstm code for further improvements? at below what i did on codes please fix it with your knowledge.

    def define_model(src_vocab, tar_vocab, src_timesteps, tar_timesteps, n_units):
    model = Sequential()
    model.add(Embedding(src_vocab, n_units, input_length=src_timesteps, mask_zero=True))
    model.add(Bidirectional(LSTM(n_units)))
    model.add(RepeatVector(tar_timesteps))
    model.add(Bidirectional(LSTM(n_units, return_sequences=True)))
    model.add(TimeDistributed(Dense(tar_vocab, activation=’softmax’)))
    return model

  35. Beay May 6, 2018 at 1:05 am #

    In this below code

    # remove non-printable chars form each token
    line = [re_print.sub(”, w) for w in line]

    in Turkish words i got this sample errors for example

    “kaç” -> “kac” , “koş”->”kos”

    how can i fix it ?

    thank you

    • Jason Brownlee May 6, 2018 at 6:31 am #

      I don’t follow sorry. What is the problem exactly?

  36. Beay May 6, 2018 at 7:25 am #

    i have used these codes on a Turkish-English corpus file and some Turkish characters are

    missing (ç,ğ,ü,ğ,Ö,Ğ,Ü,İ,ı)

    thank you.

    • Jason Brownlee May 7, 2018 at 6:45 am #

      Missing after the conversion?

      Perhaps normalizing to Latin characters is not the best approach for your specific problem?

  37. Sai May 18, 2018 at 4:55 am #

    Thank you very much. Could you please help where can I get good dataset for Thai to English. The dataset for Thai language is available from the ManyThings.org website is with lesser data.I am trying to use this approach to build similar for Thai.

    • Jason Brownlee May 18, 2018 at 6:27 am #

      Sorry, I don’t know off hand.

    • Sai May 18, 2018 at 10:39 pm #

      Please ignore my query, i have searched and got the dataset. Thank you for these articles

  38. pep May 18, 2018 at 7:35 pm #

    Once the model is trained, could be used the model to predict in both directions, I mean: english-german, german-english.

  39. Meghna May 23, 2018 at 9:10 pm #

    Hi Jason, thank you for the amazing tutorial. It really helped me. I implemented the above code and understood each function. Further, I want to implement Neural conversation model as given in https://arxiv.org/pdf/1506.05869.pdf on dialogue data. So, I have 2 questions, first is how to make pairing in dialogue data and second is how to feed previous conversations as input to the decoder model.

    • Jason Brownlee May 24, 2018 at 8:11 am #

      Sorry, I don’t have an example of a dialog system. I hope to cover it in the future.

  40. Ahmad Ahmad May 24, 2018 at 6:30 pm #

    G.M Mr Jason …

    In my model , I find BLEU scores on train dataset as follow :

    BLEU-1: 0.736022
    BLEU-2: 0.717377
    BLEU-3: 0.710192
    BLEU-4: 0.692681

    So we can notice that they are higher from the scores on train dataset.
    Is it normal behavior or is it bad ?

  41. maitha May 28, 2018 at 1:07 pm #

    Hi Jason,
    Great and helpful work, I am trying the code to translate Arabic to English but in first step (Clean Text) and it give me an empty [ ]?! how can I solve this one.
    [hi] => []
    [run] => []
    [help] => []

  42. Sastry May 28, 2018 at 11:24 pm #

    Hi Jason,

    Thanks for sharing a easy and simple approach for translations.

    I tried your code to work with Indian languages and found Hindi data set in the same location from where you shared the German dataset.

    The following normalize code for Hindi removes the character from line. I have tried with NFC, still facing the same problem. If I skip this line then, the non-printable character line is skipping the hindi text.

    print(‘Before: ‘, line)
    # normalize unicode characters
    line = normalize(‘NFD’, line).encode(‘ascii’, ‘ignore’)
    print(‘After: ‘,line)

    Before: Go.
    After: b’Go.’
    Before: जा.
    After: b’.’

    Does skipping these two lines of code affect the training in any way?

    Thanks,
    Sastry

    • Jason Brownlee May 29, 2018 at 6:26 am #

      Yes, the code example expects to work with Latin characters.

    • kamal deep garg October 1, 2018 at 12:49 pm #

      Hi Sastry sir

      Does your problem with hindi data resolve?

  43. kamal deep garg May 29, 2018 at 3:43 pm #

    Hello sir

    what is minimum Hardware requirement to train nmt using keras?

  44. Srijan Verma May 31, 2018 at 6:31 pm #

    Hi Jason,

    This post is really helpful. Thanks for this.

    I am working on building a translator which translates from English to Hindi (or any other Indian language). But I am facing a problem while cleaning the data.
    The normalize code does not work for Indian languages, and if I skip that line of code then I am not getting any output after training my data.

    Is there a way to use the same code on your post and some other way to clean the data for Indian languages to get the desired output..? Like are there any python modules/Libraries that i should install so as to use them for Indian Languages.?

    Thanks!

    • Jason Brownlee June 1, 2018 at 8:17 am #

      You may have to research how to prepare hindi data for NLP.

      Perhaps converting to latin chars in not the best approach.

  45. lakshm June 1, 2018 at 3:02 pm #

    Hello,

    Aren’t we supposed to pass the English data along with the encoded data to decoder.As per my understanding only the encoded German data has been passed to the decoder right??

  46. Sai June 5, 2018 at 6:57 pm #

    Hi Jason,

    I have now progressed upto Training the model. Cleaning & tokenizing the data set took time as i used a different language, but was a good learning.

    Wanted to know whats the significance of “30 epochs and a batch size of 64 examples” in your example. Are these anyways related to Total vocabulary (or) total trainable parameters ?

    Also, could you please guide me to any article of yours where i can learn more around what is epochs, what is BLEU score , what is loss etc.

    Thank you

  47. Sai June 7, 2018 at 9:43 pm #

    Hi Jason,

    I have a silly question, but wanted to seek clarification.

    In step “Train Neural Translation Model” :- have used 10,000 rows from the dataset, and established the model in file model.h5 for xxx number of vocabularies.
    If I extract next 10,000 rows from data and continue to train the model using the same lines of code above, would it use the previously established model from model.h5 or would it be overwritten and start as fresh data being used to train ?

    Thank you,

    • Jason Brownlee June 8, 2018 at 6:11 am #

      Yes, the model will be trained using the existing model as a starting point.

  48. Sai June 8, 2018 at 3:02 pm #

    Hi Jason,

    ok, understood.

    Referred to your article https://machinelearningmastery.com/check-point-deep-learning-models-keras/ and understood that, before compiling the model using model.compile(), i have to load the model from file, to use existing model as starting point in training.

    Thank you very much.

    • Jason Brownlee June 9, 2018 at 6:45 am #

      Glad it helped.

    • Deeksha May 8, 2019 at 5:19 am #

      DId you try using model.fit_generator?

  49. Paul June 8, 2018 at 3:19 pm #

    Hi Jason,
    Can Word2Vec be used as the input embedding to boost the LSTM model ? Or say that pre-trained word vector by Word2Vec as input of the model can get better?

    Thanks!

  50. Raghavendra June 12, 2018 at 11:06 am #

    Hello Jason,
    Excellently written article with intricate concepts explained in such a simple manner.However it would be great if you can add a attention layer for handling larger sentences.

    I tried to add a attention layer to the code above by referring the below one.
    https://github.com/keras-team/keras/issues/4962

    I am unable to add the attention layer..I have read your previous blog on adding attention

    https://machinelearningmastery.com/encoder-decoder-attention-sequence-to-sequence-prediction-keras/

    But the vocabulary at the output end is too large to be processed and this is not solving the problem

    It would be great if you add attention ( bahdanu’s or luong’s ) to your above code and solve the problem of larger sentences

    Thanking you !

    • Jason Brownlee June 12, 2018 at 2:27 pm #

      Thanks, I hope to develop some attention tutorials once it is officially supported by Keras.

      • Raghavendra June 12, 2018 at 3:23 pm #

        How about including the attention snippet as you did in the later case.this code is working fine for me except that attention can handle longer sentences and this is where I am facing issues.I was actually asking for adding attention to the above code as you did in the later case.

        • Jason Brownlee June 13, 2018 at 6:15 am #

          Sorry, I cannot create a custom example for you.

          I hope to give more examples of attention when Keras officially supports attention.

  51. Aparajita June 21, 2018 at 9:55 pm #

    Hi, I want to convert from english to german, Please help me what kind of changes required? I did few changes but it didn’t work. Please help me how can I reverse it?

    • Jason Brownlee June 22, 2018 at 6:08 am #

      It should be straight forward. Sorry, I don’t have the capacity to prepare an example for you.

  52. ricky June 22, 2018 at 5:48 pm #

    halo sir, how to modification this project to use existing model (.h5) for next project running without training again, so i just use the model ?

  53. Basil June 23, 2018 at 5:21 am #

    Jason – What’s your next tutorial, would be waiting for the next one eagerly, how would i get notified about your next one?

  54. Alex J July 3, 2018 at 4:47 pm #

    Hi Jason! Thanks for your amazing tutorial! Very clear and easy to understand. One question comes up during my reproducing of your code: the console warns that “The hypothesis contains 0 counts of 2-gram, 3-gram and 4-gram overlaps”, which leads to BLEU-2 to 4 are 0. I can’t find the reason, coz I just completely copied your code and it still doesn’t work. Can you help me with that? Thank you!

  55. Hani July 4, 2018 at 3:12 am #

    Hi,

    Could you please help me to convert a German word to a sequence of numbers?

  56. sree harsha July 5, 2018 at 2:04 am #

    Hi,
    amazing article! Here we encode the sequences(into one hot vector) and then give them input to encoder lstm and this is passed onto the decoder lstm. Is my understanding correct? how can I give an input to hidden states of an lstm?

    • Jason Brownlee July 5, 2018 at 8:00 am #

      No, we do not one hot encode the input, we provide sequences of integers to the word embedding.

  57. Hani July 5, 2018 at 7:33 am #

    Hi,

    thank you for answering. I have another question. How can I use one hot encoding for the sequences in which it returns a 2D array not a 3D?

  58. Jack July 6, 2018 at 6:19 am #

    Really amazing post! Was surprised by the accuracy and limited training time. I have tried the model with a different dataset (two columns of sentences), but get a problem in the code for loading the clean data, splitting it, and saving the split portions of data to new files. line 20:
    dataset = raw_dataset[:n_sentences, :]

    IndexError: too many indices for array

    For print(raw_dataset) with your deu.txt, I get:
    [[‘Sentence A’ ‘Sentence a’] [‘Sentence B’ ‘Sentence b’] etc. ]

    But for print(raw_dataset) with my file, I get:
    [ list([‘sentence A’, ‘sentence a’]) list([‘sentence B’, ‘sentence b’]) etc.]

    Any tips what I could do about this?

  59. Josh Reid July 8, 2018 at 12:17 am #

    Hey Jason, amazing article, this helped immensely improve my understanding of how NMT works in the background!

    I experienced the same issue as Alex J where the evaluation portion of the code where BLEU-2, 3 and 4 scores are all 0 and throw warnings like:
    “The hypothesis contains 0 counts of 2-gram overlaps.
    Therefore the BLEU score evaluates to 0, independently of
    how many N-gram overlaps of lower order it contains.
    Consider using lower n-gram order or use SmoothingFunction()”

    I’m not sure if something within nltk.bleu_score.corpus_bleu changed since you created this script but it looks like you need an additional list around each entry in actual. This is fixed by changing line 60 in that script from:
    actual.append(raw_target.split())
    to:
    actual.append([raw_target.split()])

    • Jason Brownlee July 8, 2018 at 6:23 am #

      Thanks Josh.

      • Karim November 27, 2018 at 4:02 am #

        Yes, indeed it works with:
        actual.append([raw_target.split()])
        The reference for each sentence should be a list of different correct sentences.

  60. Jack July 8, 2018 at 10:21 pm #

    Dear Jason, would it also be possible to use this model to do ‘translations’ within one language? For example, to use duplicate sentences as pairs such as:

    [‘The distance from the earth to the moon is 384.400 km’ ‘The moon is located 384.400 km away from the earth’]

    Given enough good examples, do you think this would work? I have tried it but get lousy results. Perhaps doing something wrong.

    • Jason Brownlee July 9, 2018 at 6:35 am #

      With enough training data, yes, you could do this.

      • Jack July 19, 2018 at 12:56 am #

        Dear Jason, I have just replaced the deu.txt dataset with a dataset containing two columns of English sentences and get the following (strange) predictions. Any suggestions what might cause this?

        src=[the best apps for increasing vocabulary are], target=[what are the best apps for increasing vocabulary], predicted=[and and and and and and and and and and and does does el el el el el]
        BLEU-1: 0.027778
        BLEU-2: 0.166667
        BLEU-3: 0.341279
        BLEU-4: 0.408248

        • Jason Brownlee July 19, 2018 at 7:54 am #

          Perhaps confirm that you are loading the dataset as you expect.

          You may then have to tune the model to this new dataset.

        • Remi June 11, 2019 at 11:59 pm #

          Hi,
          I’m currently doing something similar as I am trying to translate grammatically wrong french to correct french. Thing is, I also get some strange results like yours
          I’m not sure you will see this message but have you solved your problem? 🙂

          • Jason Brownlee June 12, 2019 at 8:04 am #

            Perhaps try tuning the model?

            Perhaps try more data?

            Perhaps try a different model architecture?

          • Raghav Sood June 17, 2019 at 7:39 pm #

            “There are duplicate phrases in English with different translations in German”. What problems does having duplicate phrases cause? What if I want a model to learn sentences similar in meaning to the input sentence( i.e. multiple possible outputs for the same input)? Which model would you recommend for such a situation?

          • Jason Brownlee June 18, 2019 at 6:37 am #

            It can be confusing to the model and result in lower skill.

            Simplify the problem for the model whenever possible.

  61. Sayantika Dey July 12, 2018 at 8:44 am #

    how much time does it take to print the Bleu score?
    Actually that part of the code is not working for me and its not printing the Bleu score and again again when i try to plot the model, it shows install Graphviz but i already have that.

    • Jason Brownlee July 12, 2018 at 3:28 pm #

      It depends on your hardware, but it should not take excessively long.

      If you are getting strange results, ensure you have the latest versions of all of the libraries and that you have copied all of the code required.

  62. C M Khaled Saifullah July 18, 2018 at 5:38 am #

    First of all thanks for the tutorial, it helps me a lot.

    If i like to incorporate attention mechanism and beam search in the decoder, which part of the code need to be changed?

    From my basic understanding i received from the your following tutorial:

    https://machinelearningmastery.com/encoder-decoder-attention-sequence-to-sequence-prediction-keras/

    I need to replace the following code:

    def define_model(src_vocab, tar_vocab, src_timesteps, tar_timesteps, n_units):
    model = Sequential()
    model.add(Embedding(src_vocab, n_units, input_length=src_timesteps, mask_zero=True))
    model.add(LSTM(n_units))
    model.add(RepeatVector(tar_timesteps))
    model.add(LSTM(n_units, return_sequences=True))
    model.add(TimeDistributed(Dense(tar_vocab, activation=’softmax’)))
    return model

    into

    def define_model(src_vocab, tar_vocab, src_timesteps, tar_timesteps, n_units):
    model = Sequential()
    model.add(Embedding(src_vocab, n_units, input_length=src_timesteps, mask_zero=True))
    model.add(LSTM(n_units))
    model.add(AttentionDecoder(n_units, n_features))
    return model

    After writing the custom attention layer code given in that post.

    I am not sure about the parameter n_features for this problem. Can you clarify it? Beside, can you help me to find the implementation of beam search?

    Thanks for your time.

  63. Parul Singla July 18, 2018 at 3:58 pm #

    Sir, i’m using english-hindi translation dataset. while printing the saved file code is showing the output like…

    [has tom left] => []
    [he is french] => []
    [i am at home] => []
    [i cant move] => []
    [i dont know] => []

    Why i’m not able to see Hindi text. Is there any requirement of encoding decoding again?

    • Jason Brownlee July 19, 2018 at 7:46 am #

      Sorry, I don’t know. I don’t have any examples working with Hindi text.

  64. Souraj July 22, 2018 at 11:24 pm #

    Hello Jason,

    Would it be possible to include a diagram or visualization to show how the dimensions match up in layers used? I am having a hard time figuring out how does the network exactly look like. Thanks in advance. For example, why repeat vector is necessary.

    • Jason Brownlee July 23, 2018 at 6:11 am #

      Yes, you can summarize what the model expects:

      And you can review your data:

  65. Nitin July 30, 2018 at 12:57 am #

    After save the model and load the model then i want to translate only one line randomly then how can i do that?

  66. kamalika August 3, 2018 at 11:01 pm #

    H Jason,
    Thanks for this tutorial.
    I was trying to translate from Chinese to English and looking at clean_pairs function, I think for Chinese characters, this can’t be applied.
    Can you give me some pointers on how to generate the clean text for translation model.
    I am using the dataset from many.org.

    • Jason Brownlee August 4, 2018 at 6:10 am #

      You may have to update the example to work with unicode instead of chars.

  67. Rohit August 29, 2018 at 4:18 pm #

    Hello Jason, It was a great article. I tried to implement it for ger – eng and it worked fine. But when I am implementing it for Korean to English junk output is coming

    src=[경고 고마워], target=[thanks for the warning], predicted=[i i the]
    src=[입조심해라], target=[watch your language], predicted=[i i you]
    src=[없다], target=[there arent any], predicted=[i i you]
    src=[톰은 외롭고 불행해], target=[tom is lonely and unhappy], predicted=[i i the]
    src=[그녀의 신앙심은 굳건하다], target=[her faith in god is unshaken], predicted=[i i the to]
    src=[세계는 너를 중심으로 돌아가지 않는다], target=[the world doesnt revolve around you], predicted=[i i i to to]
    src=[못 믿겠는데], target=[i can hardly believe it], predicted=[i i the]
    src=[그 약은 효과가 있었다], target=[that medicine worked], predicted=[i i]
    src=[모두 그녀를 사랑한다], target=[everybody loves her], predicted=[i i]

    I have used training data from manythings.org having 773 lines(600 lines for training ,173 lines for testing).

    Can you please guide me what can be the issue.

    • Jason Brownlee August 30, 2018 at 6:26 am #

      Perhaps the Korean characters need special handling?

      Perhaps the model needs further tuning?

  68. Ajita September 10, 2018 at 9:33 pm #

    Hey Jason,thanks for such an awesome content.I have a doubt regarding why it is necessary to convert unicode to ascii for preparing the dataset.And why NFD format is exclusively used?

    • Jason Brownlee September 11, 2018 at 6:29 am #

      It is not required, it just made my example simpler.

  69. Bhimasen September 26, 2018 at 3:34 pm #

    HI, Very Nice works in this blog. This LSTM also i applied for native Indian languages and got good results and scores. Great tutorial.!!!

    My question is, i want to make kind of federated learning here. The model created by this dataset will be kept as general model. Suppose I have a another dataset (similar, but small), and I train a model using same code and a new model is generated. Now i want to merge the weights of this new model with the one previously generated.

    How can I work around to achieve this. ? Any suggestions would be greatly appreciated.

    • Jason Brownlee September 27, 2018 at 5:55 am #

      Nice work!

      You could keep both models and use them in an ensemble.

    • kamal deep garg October 1, 2018 at 12:52 pm #

      Hi Bhimasen

      i am also doing work on Indian languages.

      getting stuck in preprocessing of Punjabi

  70. Michał September 26, 2018 at 5:38 pm #

    Hi Jason
    great tutorial – works fine with german -> english, but when I am using my own dictoniary then the predicted output is empty (“[]”).
    My dictionary is quite specific, it is sentence to sentence, like:
    “when raining then use umbrella6” -> “trigger raining check umbrella6”
    I have like 1000 lines (maybe too little) of simillar sentences and they contain this strange “umbrella6” strings (so string+ID).
    I was expecting that the results may not make any sense, but empty predict is something strange – there should be something?

  71. Ash September 28, 2018 at 7:27 am #

    May be I missed that but what happens if there is a new/unseen word in the input text? Rather what is expected in the output?

    • Jason Brownlee September 28, 2018 at 2:58 pm #

      Unseen words are marked as 0 by the Tokenizer.

  72. Cathal October 6, 2018 at 7:34 am #

    Hi Jason,

    Great tutorial, love your blog! I was just wondering how I can pass in my own input to be translated. How do I just pass in one sentence. Everything I have tried is not working!

    • Jason Brownlee October 6, 2018 at 11:42 am #

      If you have text to be translated, you can use google translate.

      If you want to use the model to make a prediction, you must encode new text using the same scheme used to prepare the training data then call model.predict().

  73. Tom Chan October 12, 2018 at 2:32 am #

    Hi Jason,

    Thanks for your detailed step by step process in walking everyone through. I have one help needed.

    What needs to be changed above for Chinese Portuguese machine translator?

    I target to do a (bi-directional) LSTM but cannot find existing word data file as the source.

    Hope you can point me the direction and thanks.

    B.Rgds,
    Tom

    • Jason Brownlee October 12, 2018 at 6:42 am #

      The model may need to be tuned for your new dataset.

  74. Ali October 15, 2018 at 3:38 am #

    When I run the evaluation I get the following result:
    UserWarning:
    The hypothesis contains 0 counts of 4-gram overlaps.
    Therefore the BLEU score evaluates to 0, independently of
    how many N-gram overlaps of lower order it contains.
    Consider using lower n-gram order or use SmoothingFunction()
    warnings.warn(_msg)
    BLEU-1: 0.077830
    BLEU-2: 0.000000
    BLEU-3: 0.000000
    BLEU-4: 0.000000

    How can I fix this?

    • Jason Brownlee October 15, 2018 at 7:33 am #

      Perhaps check the types of text generated by your model, your model may not have converged to a useful solution.

      • Bond October 23, 2018 at 12:18 am #

        How do we fix the issue? I tried re-running the model from the start again. It is showing the same result.

        /usr/local/lib/python3.5/dist-packages/nltk/translate/bleu_score.py:503: UserWarning:
        The hypothesis contains 0 counts of 4-gram overlaps.
        Therefore the BLEU score evaluates to 0, independently of
        how many N-gram overlaps of lower order it contains.
        Consider using lower n-gram order or use SmoothingFunction()
        warnings.warn(_msg)
        BLEU-1: 0.077346
        BLEU-2: 0.000000
        BLEU-3: 0.000000
        BLEU-4: 0.000000

        The same warning is there for 2-gram and 3-gram.

        • Jason Brownlee October 23, 2018 at 6:27 am #

          Perhaps try changing the configuration of the model?

  75. Bond October 22, 2018 at 5:14 pm #

    Hi, thanks for your contribution.

    Could you please clarify some of the doubts:

    1. In the CLEAN TEXT step, inside clean_pairs() function, line number 7 talks about making a translation table for removing punctuation.

    In the code, str.maketrans(”, ”, string.punctuation)
    gives error with str as an undefined attribute.

    And also what is “maketrans” function?

    2. Regarding the function “to_pairs”, this function converts the dataset in the following format:

    Original:
    Hi. Hallo!
    Hi. Grüß Gott!
    Run! Lauf!

    After:
    Hi.
    Hallo!
    Hi.
    Grüß Gott!
    Run!
    Lauf!

    i.e. put the corresponding translation in the next line by splitting the phrase pairs.

    Thanks.

    • Jason Brownlee October 23, 2018 at 6:21 am #

      You may be trying to use Python 2.7, I recommend using Python 3.5 or higher.

  76. satya October 25, 2018 at 5:41 pm #

    how this implementation differs from keras implemenation ?

    https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html

    which one to prefer ?

    • Jason Brownlee October 26, 2018 at 5:32 am #

      Here we use an auto-encoder approach, in the keras blog post an encoder-decoder using only internal state is used instead.

      Use an approach that results in the best performance for your problem.

  77. Akshat Jain October 28, 2018 at 12:26 am #

    Hiii Jason,
    Thanks for this wonderful article. I have been trying to implement this and I got a doubt in

    prediction = model.predict(testX, verbose=1)[0]

    Why we only take single encoded source?

    • Jason Brownlee October 28, 2018 at 6:13 am #

      There is only one prediction/row, so we take it from the 2D array.

      • Akshat Jain October 29, 2018 at 8:56 pm #

        Sorry I don’t understand, the shape of prediction would be (1000, 5, 2309) but we only take the zeroth element from it. Why?

        • Jason Brownlee October 30, 2018 at 6:00 am #

          No, we are only translating one sentence of words at a time.

          To confirm, print the shape of the input and output of the predict function prior to only selecting the zero’th element.

  78. Daniel Fernandez Boada November 20, 2018 at 6:40 pm #

    Hi Jason,

    Thank you for sharing this great article. Because of my null progress in learning German, after four years living in a German speaking country, I decided to create an application that I think could help me with it, and maybe to others too.

    As a first step I think your approach may fit well with my requirements. My question is, are all the codes shown here free to reproduce or is there and copyright?

    Thanks again,
    Dani.

  79. saas November 27, 2018 at 3:49 am #

    hello
    could you please help me
    i m doing same work neural translation from English to arabic !!
    how I follow the steps which is provided but I got an error

    • Jason Brownlee November 27, 2018 at 6:38 am #

      Perhaps post your error to stackoverflow?

      • ssaa December 11, 2018 at 4:25 am #

        hello sir
        I got this result while running but does not apear probably

        train
        src=[], target=[continue digging], predicted=[i is to]
        src=[], target=[tom laid the gun down on the floor], predicted=[i is to]
        src=[], target=[i have to find it], predicted=[i is to]
        src=[], target=[i believe in god], predicted=[i is to]
        src=[], target=[im a free man], predicted=[i is to]
        src=[], target=[can i use my credit card], predicted=[i is to]
        src=[], target=[she is about to leave], predicted=[i is to]
        src=[], target=[she raised her hands], predicted=[i is to]
        src=[], target=[my uncle died a year ago], predicted=[i is to]
        src=[], target=[im sitting alone in my house], predicted=[i is to]
        /anaconda3/lib/python3.6/site-packages/nltk/translate/bleu_score.py:490: UserWarning:
        Corpus/Sentence contains 0 counts of 2-gram overlaps.
        BLEU scores might be undesirable; use SmoothingFunction().
        warnings.warn(_msg)
        BLEU-1: 0.266528
        BLEU-2: 0.516264
        BLEU-3: 0.672548
        BLEU-4: 0.718515
        test
        src=[], target=[im working in a town near rome], predicted=[i is to]
        src=[], target=[she despised him], predicted=[i is to]
        src=[], target=[the clock is ticking], predicted=[i is to]
        src=[], target=[this river is one mile across], predicted=[i is to]
        src=[], target=[birds of a feather flock together], predicted=[i is to]
        src=[], target=[why did you turn down his offer], predicted=[i is to]
        src=[], target=[shes as clever as they make em], predicted=[i is to]
        src=[], target=[how can i help], predicted=[i is to]
        src=[], target=[our living room is sunny], predicted=[i is to]
        src=[], target=[can you speak french], predicted=[i is to]
        BLEU-1: 0.260667
        BLEU-2: 0.510555
        BLEU-3: 0.668076
        BLEU-4: 0.714531

        • Jason Brownlee December 11, 2018 at 7:51 am #

          Perhaps try fitting the model again?

          • ssaa December 12, 2018 at 1:02 am #

            my dataset English-arabic
            when load it and clean the data I got this
            [hi] => []
            [run] => []
            [help] => []
            [jump] => []
            [stop] => []
            [go on] => []
            [go on] => []
            [hello] => []
            [hurry] => []
            [hurry] => []
            [i see] => []
            [i won] => []
            [relax] => []
            [smile] => []
            [cheers] => []
            [got it] => []
            [he ran] => []
            [i know] => []
            [i know] => []
            [i know] => []
            [im] => []
            [im ok] => []
            [listen] => []
            [no way] => []
            [really] => []
            [thanks] => []
            [why me] => []
            [awesome] => []
            [call me] => []
            [call me] => []
            [come in] => []
            [come in] => []
            [come on] => []
            [come on] => []
            [come on] => []
            [get out] => []
            [get out] => []
            [get out] => []
            [go away] => []
            [go away] => []
            [go away] => []
            [goodbye] => []
            [he came] => []
            [he runs] => []
            [help me] => []
            [help me] => []
            [im sad] => []
            [me too] => []
            [shut up] => []
            [shut up] => []
            [shut up] => []
            [shut up] => []
            [stop it] => []
            [take it] => []
            [tom won] => []
            [tom won] => []
            [wake up] => []
            [welcome] => []
            [welcome] => []
            [welcome] => []
            [welcome] => []
            [who won] => []
            [who won] => []
            [why not] => []
            [why not] => []
            [have fun] => []
            [hurry up] => []
            [i forgot] => []
            [i got it] => []
            [i got it] => []
            [i got it] => []
            [i use it] => []
            [ill pay] => []
            [im busy] => []
            [im busy] => []
            [im cold] => []
            [im free] => []
            [im here] => []
            [im home] => []
            [im poor] => []
            [im rich] => []
            [it hurts] => []
            [its hot] => []
            [its new] => []
            [lets go] => []
            [lets go] => []
            [lets go] => []
            [lets go] => []
            [lets go] => []
            [look out] => []
            [look out] => []
            [look out] => []
            [speak up] => []
            [stand up] => []
            [terrific] => []
            [terrific] => []
            [tom died] => []
            [tom died] => []
            [tom left] => []
            [tom lied] => []

          • Jason Brownlee December 12, 2018 at 5:55 am #

            Perhaps your model requires further tuning?

          • henok meskele December 27, 2018 at 12:53 am #

            l need Universal networking language based algorithms how it works and l want to integrate other algorithms with UNL framwork enco and deco functions

          • Jason Brownlee December 27, 2018 at 5:45 am #

            I don’t have material in that topic, sorry.

  80. Nikos November 29, 2018 at 9:07 am #

    Excellent work! Thank you, Jason!

  81. Naresh December 3, 2018 at 3:11 am #

    Can i use this model to train chinese to english translation, as chinese is something different then other language what precaution i need to take care.

  82. Sourabh December 5, 2018 at 12:15 am #

    Hello Sir, Thank you very much for this wonderful guide!!!
    I just have one doubt….Can we build a model which could translate both-ways…i.e. Language1 to Language2 and also Language2 to Language1?

  83. sree harsha December 11, 2018 at 12:39 am #

    Hi Jason, can you please clarify: in this model, are we giving the word embeddings as hidden state input to the encoder- lstm?

    Thanks in advance!

    • Jason Brownlee December 11, 2018 at 7:45 am #

      The embedding is provided as input to the LSTM, not hidden state.

      • sree harsha December 12, 2018 at 7:11 am #

        Thankyou for your reply 🙂 Is any direct input given to the second LSTM? or it receives only hidden input from the first one?

        • Jason Brownlee December 12, 2018 at 2:11 pm #

          You can use the output and hidden state or just the output. I prefer the latter – like an autoencoder. It is simpler and performs very well.

        • Gaurav July 8, 2019 at 2:59 am #

          Can I develop a multilingual machine translation using any pretrained model? How to do that?

          • Jason Brownlee July 8, 2019 at 8:44 am #

            Perhaps.

            I don’t have a tutorial on this topic, sorry.

  84. Kushal Davendra December 19, 2018 at 9:02 am #

    Hi Jason,

    Thanks for a wonderful post. I am trying to run the code on a different translation (English to Hindi), the training runs fine but while evaluating I am getting the following error:
    Using TensorFlow backend.
    train
    Traceback (most recent call last):
    File “c:\program files (x86)\microsoft visual studio\2017\enterprise\common7\i
    de\extensions\microsoft\python\core\ptvsd_launcher.py”, line 119, in
    vspd.debug(filename, port_num, debug_id, debug_options, run_as)
    File “c:\program files (x86)\microsoft visual studio\2017\enterprise\common7\i
    de\extensions\microsoft\python\core\Packages\ptvsd\debugger.py”, line 37, in deb
    ug
    run(address, filename, *args, **kwargs)
    File “c:\program files (x86)\microsoft visual studio\2017\enterprise\common7\i
    de\extensions\microsoft\python\core\Packages\ptvsd\_local.py”, line 64, in run_f
    ile
    run(argv, addr, **kwargs)
    File “c:\program files (x86)\microsoft visual studio\2017\enterprise\common7\i
    de\extensions\microsoft\python\core\Packages\ptvsd\_local.py”, line 125, in _run

    _pydevd.main()
    File “c:\program files (x86)\microsoft visual studio\2017\enterprise\common7\i
    de\extensions\microsoft\python\core\Packages\ptvsd\_vendored\pydevd\pydevd.py”,
    line 1752, in main
    globals = debugger.run(setup[‘file’], None, None, is_module)
    File “c:\program files (x86)\microsoft visual studio\2017\enterprise\common7\i
    de\extensions\microsoft\python\core\Packages\ptvsd\_vendored\pydevd\pydevd.py”,
    line 1099, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, loc
    als)
    File “c:\program files (x86)\microsoft visual studio\2017\enterprise\common7\i
    de\extensions\microsoft\python\core\Packages\ptvsd\_vendored\pydevd\pydevd.py”,
    line 1106, in _exec
    pydev_imports.execfile(file, globals, locals) # execute the script
    File “c:\program files (x86)\microsoft visual studio\2017\enterprise\common7\i
    de\extensions\microsoft\python\core\Packages\ptvsd\_vendored\pydevd\_pydev_imps\
    _pydev_execfile.py”, line 25, in execfile
    exec(compile(contents+”\n”, file, ‘exec’), glob, loc)
    File “D:\kudave\EEM\Code\EEMDNN\EEMDNN\infer.py”, line 88, in
    evaluate_model(model, eng_tokenizer, trainX, train)
    File “D:\kudave\EEM\Code\EEMDNN\EEMDNN\infer.py”, line 56, in evaluate_model
    translation = predict_sequence(model, eng_tokenizer, source)
    File “D:\kudave\EEM\Code\EEMDNN\EEMDNN\infer.py”, line 40, in predict_sequence

    prediction = model.predict(source, verbose=0)[0]
    File “C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\si
    te-packages\keras\engine\training.py”, line 1149, in predict
    x, _, _ = self._standardize_user_data(x)
    File “C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\si
    te-packages\keras\engine\training.py”, line 751, in _standardize_user_data
    exception_prefix=’input’)
    File “C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\si
    te-packages\keras\engine\training_utils.py”, line 138, in standardize_input_data

    str(data_shape))
    ValueError: Error when checking input: expected embedding_1_input to have shape
    (8,) but got array with shape (11,)

    ————————————————————–
    I get the same error when I evaluate on test data. Do you know what is wrong. To start with I have not changed much except the data in the code.

    • Jason Brownlee December 19, 2018 at 2:28 pm #

      Sorry, I don’t have the capacity to debug your changes, perhaps post your code and error to stackoverflow?

  85. Hareem December 26, 2018 at 8:41 pm #

    Hi Jason!
    I am using your NMT code for converting sentences from present perfect to past perfect tense. I trained it for 50 epochs

    Epoch 50/50
    2500/2500 [==============================] – 37s 15ms/step – loss: 1.0273 – acc: 0.8178 – val_loss: 1.1926 – val_acc: 0.8187

    But its giving me out put like this

    train
    src=[i have no idea what i need to do now], target=[i had no idea what i need to do then], predicted=[i had not to had had had had had]
    src=[i will get by if i have a place to sleep], target=[i will get by if i had a place to sleep], predicted=[i had not had had had had had had had]
    src=[this is the worst book i have ever read], target=[this was the worst book i had ever read], predicted=[i had not had had had had had had]
    src=[does anybody have any good news], target=[does anybody had any good news], predicted=[i had a a a]
    src=[i have everything here that i need], target=[i had everything here that i need], predicted=[i had had to to had]
    src=[can i have my gun back], target=[could i have my gun back], predicted=[i had have a a]
    src=[i want to go and have a drink], target=[i want to go and had a drink], predicted=[i had to to to to the me]
    src=[i have an orange and an apple], target=[i had an orange and an apple], predicted=[i had had to to my]
    src=[i have a dog that can run fast], target=[i had a dog that could run fast], predicted=[i had had to had had had]
    src=[i have a sweet tooth], target=[i had a sweet tooth], predicted=[i had a a]
    src=[i have already told tom i will not help him], target=[i had already told tom i will not help him], predicted=[i had not had had had had had had]

    and on test data

    src=[i have no regrets for what i have done], target=[i had no regrets for what i had done], predicted=[i had had to to had had the]
    src=[tom must have heard about what happened], target=[tom must had heard about what happened], predicted=[i had had had had to you]
    src=[i have left my umbrella in a bus], target=[i had left my umbrella in a bus], predicted=[i had had to to for for]
    src=[could i have money for my piano lesson], target=[could i had money for my piano lesson], predicted=[i had had to to to for me]
    src=[i have said i am sorry], target=[i had said i was sorry], predicted=[i had had to to you]
    src=[i have been to the u], target=[i had been to the u], predicted=[i had had to to you]
    src=[may i have a glass of milk please], target=[may i had a glass of milk please], predicted=[i had had had to to to me]
    src=[recently i have had no appetite], target=[recently i had had no appetite], predicted=[i had had had been]
    src=[my friend has been here this week], target=[my friend had been here this week], predicted=[i had to to to the]
    src=[i have waited two whole hours], target=[i had waited two whole hours], predicted=[i had had a my]

    can you tell what i am doing wrong. The source an target both languages are english.

    • Jason Brownlee December 27, 2018 at 5:42 am #

      Perhaps try re-fitting the model a few times and compare results?

  86. Rajan December 27, 2018 at 5:02 pm #

    TypeError : int() argument must be a string, a bytes-like object or a number, not ‘TensorShapeProto’
    i got this error

  87. kunu January 4, 2019 at 8:04 pm #

    I have a trained model and it will successfully evaluate in the model.h5 and i want a code for translate single line sentence . like i passed hallo! the it will say hello

    • Jason Brownlee January 5, 2019 at 6:54 am #

      I show how to use the model in inference mode in the above tutorial.

  88. sourav January 8, 2019 at 1:25 am #

    from tensorflow.python import pywrap_tensorflow

    ImportError: cannot import name ‘pywrap_tensorflow’ from ‘tensorflow.python’ (unknown location)

    how can I solve this bug

    • Jason Brownlee January 8, 2019 at 6:51 am #

      Seems unrelated to this post, try posting to stackoverflow.

  89. swathi January 8, 2019 at 4:00 pm #

    Hi Jason,
    can u please let me know how to give only one input sentence in german language and find its translation using the model constructed??

    • Jason Brownlee January 9, 2019 at 8:38 am #

      Use the final code example and call:

      Where “source” is your integer encoded sentence of text with the shape [1, n]

      • Muho March 23, 2019 at 6:05 pm #

        Is this meaning that we have to convert our text using a tokenizer before?

        • Jason Brownlee March 24, 2019 at 7:04 am #

          Yes.

        • Mahmudul October 1, 2019 at 11:26 am #

          Hey did you translate it……if you please help me

  90. Mohamed ashraf January 22, 2019 at 7:57 pm #

    great work Jason , i tried to built the same model but for English to Arabic langauge , but a got an error when trying to load and validate the model

    Using TensorFlow backend.
    2019-01-22 10:00:35.388677: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
    train
    Traceback (most recent call last):
    File “validation.py”, line 88, in
    evaluate_model(model, eng_tokenizer, trainX, train)
    File “validation.py”, line 56, in evaluate_model
    translation = predict_sequence(model, eng_tokenizer, source)
    File “validation.py”, line 40, in predict_sequence
    prediction = model.predict(source, verbose=0)[0]
    ValueError: Error when checking input: expected embedding_1_input to have shape (36,) but got array with shape (14,)

    • Jason Brownlee January 23, 2019 at 8:46 am #

      Looks like there is a mismatch between the data provided and the model’s expectations.

      Change the data or the model.

      • Mohamed Hossam January 29, 2019 at 10:33 pm #

        hi , Jason

        thank you for this great article,

        i tried this code and it works very well , but when am trying much bigger data set i have an error

        English Vocabulary Size: 20428
        English Max Length: 48
        arabic Vocabulary Size: 33623
        arabic Max Length: 59
        Traceback (most recent call last):
        File “tokniezer.py”, line 79, in
        trainY = encode_output(trainY, eng_vocab_size)
        File “tokniezer.py”, line 43, in encode_output
        y = array(ylist)
        MemoryError

        Can i solve this error without increasing my RAM memory , i am using now 16GB memory and a data set about 100000 Line

        • Jason Brownlee January 30, 2019 at 8:12 am #

          Perhaps try on a machine with more RAM?
          Perhaps try working on a subset of the dataset?

        • Abdul Basit May 8, 2019 at 5:32 am #

          I tried to use this for English Urdu but after cleaning i am getting blank from urdu side
          [we won] => []
          [beat it] => []
          [beat it] => []
          [we lost] => []
          [good job] => []
          [lets go] => []
          [toms up] => []
          [i am sick] => []
          [let me in] => []
          [lets try] => []
          [stay thin] => []
          [stay thin] => []
          [stay thin] => []
          [stay thin] => []
          [toms fat] => []
          [toms mad] => []
          [toms sad] => []
          [toms shy] => []
          [we talked] => []
          [well try] => []
          [well win] => []
          [whats up] => []
          [are you ok] => []
          [i like tea] => []
          [i love her] => []
          [i love you] => []
          [i love you] => []
          [i need you] => []
          [i need you] => []
          [im sleepy] => []
          [im sleepy] => []
          [toms dead] => []
          [toms deaf] => []
          [toms died] => []
          [toms fast] => []
          [toms free] => []
          [toms gone] => []
          [toms here] => []
          [toms home] => []
          [toms hurt] => []
          [toms safe] => []
          [toms sick] => []
          [toms weak] => []
          [toms well] => []
          [well help] => []
          [well wait] => []
          [how are you] => []
          [how are you] => []
          [i live here] => []
          [i live here] => []
          [i love rock] => []
          [i need help] => []
          [i trust you] => []
          [im at home] => []
          [is it white] => []
          [it may rain] => []
          [it may snow] => []
          [lets do it] => []
          [toms bored] => []
          [toms drunk] => []
          [toms right] => []

          • Jason Brownlee May 8, 2019 at 6:46 am #

            Perhaps your model requires tuning?

  91. Ivan williams January 25, 2019 at 12:19 pm #

    I’m sorry if this is the wrong place to ask, but I’m having trouble with cleaning and saving the Data, it keeps saying IndexError: index 1 is out of bounds for axis 1 with size 1, does this mean i’m creating clean_pairs wrong?

    • Jason Brownlee January 26, 2019 at 6:04 am #

      Perhaps try posting your code and your error to stackoverflow?

  92. Nikhil Ramesh January 31, 2019 at 3:11 pm #

    If i worked with languages such as hindi how would I go about the cleaning of data since the scripts and symbols are entirely different

    • Jason Brownlee February 1, 2019 at 5:32 am #

      You may have to update the examples to support unicode characters.

  93. GAUTAM February 2, 2019 at 3:50 am #

    Hey jason,
    I saw your post nice work. Well i am also working on a SMT project using python. I would like to know if can you provide a running project code about it or help me in some how to get it.
    Thanks.

    • Jason Brownlee February 2, 2019 at 6:25 am #

      Sorry, I don’t understand, can you please restate or elaborate your question?

  94. Lars Ericson February 10, 2019 at 11:57 am #

    I am trying to convert this to run on TPUs. Following other notebooks for TPUs, I use the Keras layer in Tensorflow rather than the other way around. Before getting to the TPU part, I am doing this conversion to see if it still runs on GPU. This means mainly changing this function:

    # define NMT model
    def define_model(src_vocab, tar_vocab, src_timesteps, tar_timesteps, n_units):
    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.Embedding(src_vocab, n_units, input_length=src_timesteps, mask_zero=True))
    model.add(tf.keras.layers.LSTM(n_units))
    model.add(tf.keras.layers.RepeatVector(tar_timesteps))
    model.add(tf.keras.layers.LSTM(n_units, return_sequences=True))
    model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(tar_vocab, activation=’softmax’)))
    return model

    However this results in a complaint followed by a runtime error when I run fit:

    lib\site-packages\tensorflow\python\ops\gradients_impl.py:112: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
    “Converting sparse IndexedSlices to a dense Tensor of unknown shape. ”

    Then during the fit inside the GPU it fails on a BLAS load as follows:

    InternalError: Blas GEMM launch failed : a.shape=(64, 256), b.shape=(256, 256), m=64, n=256, k=256
    [[{{node lstm/while/MatMul}} = MatMul[T=DT_FLOAT, _class=[“loc:@training/Adam/gradients/lstm/while/strided_slice_grad/StridedSliceGrad”], transpose_a=false, transpose_b=false, _device=”/job:localhost/replica:0/task:0/device:GPU:0″](lstm/while/TensorArrayReadV3, lstm/while/strided_slice)]]
    [[{{node loss/time_distributed_loss/broadcast_weights/assert_broadcastable/AssertGuard/Assert/Switch/_175}} = _Recv[client_terminated=false, recv_device=”/job:localhost/replica:0/task:0/device:CPU:0″, send_device=”/job:localhost/replica:0/task:0/device:GPU:0″, send_device_incarnation=1, tensor_name=”edge_2728_…ert/Switch”, tensor_type=DT_BOOL, _device=”/job:localhost/replica:0/task:0/device:CPU:0″]()]]

    Any thoughts?

    • Jason Brownlee February 11, 2019 at 7:54 am #

      Sorry, I have not used kf.keras and I don’t use notebooks or TPUs, I don’t have any good advice for you.

      Perhaps try posting on a tensorflow user group or stackoverflow?

  95. A Hannan February 11, 2019 at 8:47 pm #

    Hi Jason,
    Great job.. Thank you for the tutorial.
    I gave it a try. I used your dataset only but facing some issue. Here it is like—

    train
    src=[er lief], target=[he ran], predicted=[he he]
    src=[er rannte], target=[he ran], predicted=[ran he]
    src=[donnerwetter], target=[wow], predicted=[]
    src=[keine bewegung], target=[freeze], predicted=[]
    src=[ich verstehe], target=[i see], predicted=[i fell]
    src=[feuer], target=[fire], predicted=[]
    src=[im ernst], target=[really], predicted=[]
    src=[mach mit], target=[hop in], predicted=[he he]
    src=[ich bin jahre alt], target=[im], predicted=[i fell]
    src=[ausgeschlossen], target=[no way], predicted=[]
    BLEU-1: 0.066384
    BLEU-2: 0.128366
    BLEU-3: 0.167111
    BLEU-4: 0.178502
    test
    /usr/local/lib/python3.6/dist-packages/nltk/translate/bleu_score.py:490: UserWarning:
    Corpus/Sentence contains 0 counts of 2-gram overlaps.
    BLEU scores might be undesirable; use SmoothingFunction().
    warnings.warn(_msg)
    —————————————————————————
    ZeroDivisionError Traceback (most recent call last)
    in ()
    89 # test on some test sequences
    90 print(‘test’)
    —> 91 evaluate_model(model, eng_tokenizer, testX, test)

    in evaluate_model(model, tokenizer, sources, raw_dataset)
    61 predicted.append(translation.split())
    62 # calculate BLEU score
    —> 63 print(‘BLEU-1: %f’ % corpus_bleu(actual, predicted, weights=(1.0, 0, 0, 0)))
    64 print(‘BLEU-2: %f’ % corpus_bleu(actual, predicted, weights=(0.5, 0.5, 0, 0)))
    65 print(‘BLEU-3: %f’ % corpus_bleu(actual, predicted, weights=(0.3, 0.3, 0.3, 0)))

    /usr/local/lib/python3.6/dist-packages/nltk/translate/bleu_score.py in corpus_bleu(list_of_references, hypotheses, weights, smoothing_function, auto_reweigh, emulate_multibleu)
    181 # Collects the various precision values for the different ngram orders.
    182 p_n = [Fraction(p_numerators[i], p_denominators[i], _normalize=False)
    –> 183 for i, _ in enumerate(weights, start=1)]
    184
    185 # Returns 0 if there’s no matching n-grams

    /usr/local/lib/python3.6/dist-packages/nltk/translate/bleu_score.py in (.0)
    181 # Collects the various precision values for the different ngram orders.
    182 p_n = [Fraction(p_numerators[i], p_denominators[i], _normalize=False)
    –> 183 for i, _ in enumerate(weights, start=1)]
    184
    185 # Returns 0 if there’s no matching n-grams

    /usr/lib/python3.6/fractions.py in __new__(cls, numerator, denominator, _normalize)
    176
    177 if denominator == 0:
    –> 178 raise ZeroDivisionError(‘Fraction(%s, 0)’ % numerator)
    179 if _normalize:
    180 if type(numerator) is int is type(denominator):

    ZeroDivisionError: Fraction(0, 0)

    • Jason Brownlee February 12, 2019 at 7:58 am #

      Not sure why you got an error.

      Perhaps try fitting the model again and see if it gives different results.

  96. Arshi February 11, 2019 at 10:14 pm #

    Hello Sir.

    I am facing a problem while running the code using English and another Indian language.
    while printing the sentence pair it shows blank in the right hand side. please suggest something to fix it.

    [jaipur popularly known as the pink city is the capital of rajasthan state india] => []
    [the city is famous for its majestic forts palaces and beautiful lakes which attract tourists from all over the world] => []
    [the city palace was built by maharaja jai singh ii and is a synthesis of mughal and rajasthani architecture] => []
    [the hawa mahal was built by the maharaja sawai pratap singh in ad and lal chand usta was the architect] => []
    [the amber fort complex has several apartments with palaces halls stairways pillared pavilions gardens and temples] => []
    [the amber palace is a classic example of mughal and hindu architecture] => []
    [the government central museum was constructed in when the prince of wales had visited india and was opened to public in] => []
    [the government central museum has a rich collection of ivory work textiles jewellery carved wooden objects miniature paintings marble statues arms and weapons] => []
    [sisodiya ranikabagh was built by sawai jai singh ii for his sisodiya queen] => []
    [the jal mahal is a picturesque palace which was built for royal duck shooting parties] => []
    [kanak vrindavan is a popular picnic spot in jaipur] => []
    [jaipur bazaars are vibrant and the shops are full with colorful items which include handicraft items precious stones textiles minakari items jewellery rajasthani paintings etc] => []
    [jaipur is also famous for marble statues blue pottery and the rajasthani shoes] => []

    • Jason Brownlee February 12, 2019 at 8:02 am #

      I have a few ideas:

      Perhaps there’s a bug in your code?
      Perhaps the model has not fit the problem?
      Perhaps a different model configuration is required?

  97. Jitendra February 17, 2019 at 1:13 pm #

    Hi I am having issue with my code and unable to find error. For every prediction it gives almost same result. I tried the your tutorial with English-Hindi example. Please find code and tell me where I am making mistake.

    def define_model(src_vocab, tar_vocab, src_timesteps, tar_timesteps, n_units):
    model = Sequential()
    model.add(Embedding(src_vocab, n_units, input_length=src_timesteps, mask_zero=True))
    model.add(LSTM(n_units))
    model.add(RepeatVector(tar_timesteps))
    model.add(LSTM(n_units, return_sequences=True))
    model.add(TimeDistributed(Dense(tar_vocab, activation=’softmax’)))
    return model

    # define model
    model = define_model(english_vocab_size, hindi_vocab_size, english_max_sentence_size, hindi_max_sentence_size, 32)
    print(english_vocab_size, hindi_vocab_size, english_max_sentence_size, hindi_max_sentence_size)
    model.compile(optimizer=’adam’, loss=’categorical_crossentropy’)
    # summarize defined model
    print(model.summary())
    # plot_model(model, to_file=’model.png’, show_shapes=True)

    checkpoint = ModelCheckpoint(“model.h6″, monitor=’val_loss’, verbose=1, save_best_only=True, mode=’min’)
    # print(hindi_preproc.shape, english_preproc.shape)
    model.fit(english_train_preproc, hindi_train_preproc, epochs=30, validation_data=(english_test_preproc, hindi_test_preproc), batch_size=16, callbacks=[checkpoint], verbose=2)

    model = load_model(‘model.h6’)

    print(english_test_preproc.shape)
    hindi_index_to_words = {id:word for word, id in hindi_tokenize.word_index.items()}
    hindi_index_to_words[0] = ”
    english_index_to_words = {id:word for word, id in english_tokenize.word_index.items()}
    english_index_to_words[0] = ”

    english_test_preproc_temp = english_train_preproc[600:620,:]
    for i, source in enumerate(english_test_preproc_temp):
    print(‘ ‘.join([english_index_to_words[p] for p in source.tolist()]))
    source = source.reshape((1, source.shape[0]))
    prediction = model.predict(source, verbose=0)
    print(ids_to_text(prediction[0], hindi_tokenize))

    All the variables have values as their names meaning.

  98. Shreya February 20, 2019 at 10:02 pm #

    In the code, str.maketrans(””, ””, string.punctuation)
    gives error as str as an undefined attribute maketrans.

    We even tried it as using string.maketrans then it gives error as –
    maketrans() takes excatly 2 arguments

    And if we pass only 2 arguments then it gives error as – arguments must have same length.

    We are using Python 3.5 and Platform as Eclipse still its giving us an error.

    What could be the possible solution ?

    Thank You.

    • Jason Brownlee February 21, 2019 at 8:03 am #

      I believe maketrans() is a Python 2 function, try Python 2.7.

      Alternately, try this in Python 3:

      • Shreya February 25, 2019 at 7:47 pm #

        Thank you Sir. This helped.

        Sir but we are now facing an issue with keras.
        Please help us in moving forward as we have tried all ways of installing keras.

        This is the copy pasted error shown on Spyder even after we have installed keras with conda

        runfile(‘/home/ccoewitlab1-99/.spyder2-py3/temp.py’, wdir=’/home/ccoewitlab1-99/.spyder2-py3′)
        Traceback (most recent call last):

        File “”, line 1, in
        runfile(‘/home/ccoewitlab1-99/.spyder2-py3/temp.py’, wdir=’/home/ccoewitlab1-99/.spyder2-py3′)

        File “/usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py”, line 699, in runfile
        execfile(filename, namespace)

        File “/usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py”, line 88, in execfile
        exec(compile(open(filename, ‘rb’).read(), filename, ‘exec’), namespace)

        File “/home/ccoewitlab1-99/.spyder2-py3/temp.py”, line 10, in
        from keras.preprocessing.sequence import pad_sequences

        ImportError: No module named ‘keras’

        • Jason Brownlee February 26, 2019 at 6:17 am #

          It looks like Keras is not installed, perhaps try this tutorial to setup your environment:
          https://machinelearningmastery.com/setup-python-environment-machine-learning-deep-learning-anaconda/

          • Shreya February 27, 2019 at 4:50 am #

            Thank You very much once again Sir.
            After following the above steps, all the errors were solved.

            Our code is now running for small dataset, but for the dataset which this tutorial contains it is gives error as Memory Load and not giving us the Final output.

            Also is there some way where instead of our output occuring as src[], target[] and prdicted[], can we ask the user to enter the input in german whether it may be a word or paragraph and on running the code gives us the output in English Language. In short giving the Test Data as input from user.

            Thanks and Regards.

          • Jason Brownlee February 27, 2019 at 7:36 am #

            Perhaps try running the code on a machine with more memory, e.g. on an EC2 instance:
            https://machinelearningmastery.com/develop-evaluate-large-deep-learning-models-keras-amazon-web-services/

            You can use the model within normal software, but that is a software engineering question, not a machine learning question.

  99. arun February 27, 2019 at 7:43 am #

    Dear Jason
    please add attention layer in this example. it will help us a lot.
    thanks a lot.

  100. Staimer Florian February 28, 2019 at 2:12 pm #

    Great article, thank you very much.

    When you talk about “A BLEU-4 score of 0.076238 was achieved, providing a baseline skill to improve upon with further improvements to the model.” Is it necessary to train the model whenever new lines are added to improve the translation?

    Regards!

    • Jason Brownlee February 28, 2019 at 2:34 pm #

      Perhaps. This is something that can be tested and considered.

  101. Riya March 7, 2019 at 4:22 am #

    Hello,
    When I run the code along with the whole dataset I get an error saying IndexError: too many indices for array
    And it says that error is at eng_tokenizer = create_tokenizer(dataset[:,0]).
    Can you please help.

  102. Riya March 8, 2019 at 9:06 pm #

    Hello,
    I went through the instructions given in the link and everything seems in accordance with the points.
    I am still getting this error when I run the code

    runfile(‘/home/ccoewitlab1-99/.config/spyder-py3/temp.py’, wdir=’/home/ccoewitlab1-99/.config/spyder-py3′)
    Saved: english-german.pkl
    [hi] => [hallo]
    [hi] => [gru gott]
    [run] => [lauf]
    [wow] => [potzdonner]
    [wow] => [donnerwetter]
    [fire] => [feuer]
    [help] => [hilfe]
    [help] => [zu hulf]
    [stop] => [stopp]
    [wait] => [warte]
    [go on] => [mach weiter]
    [hello] => [hallo]
    [i ran] => [ich rannte]
    [i see] => [ich verstehe]
    [i see] => [aha]
    [i try] => [ich probiere es]
    [i won] => [ich hab gewonnen]
    [i won] => [ich habe gewonnen]
    [smile] => [lacheln]
    [cheers] => [zum wohl]
    [freeze] => [keine bewegung]
    [freeze] => [stehenbleiben]
    [got it] => [kapiert]
    [got it] => [verstanden]
    [got it] => [einverstanden]
    [he ran] => [er rannte]
    [he ran] => [er lief]
    [hop in] => [mach mit]
    [hug me] => [druck mich]
    [hug me] => [nimm mich in den arm]
    Saved: english-german-both.pkl
    Saved: english-german-train.pkl
    Saved: english-german-test.pkl
    Traceback (most recent call last):

    File “”, line 1, in
    runfile(‘/home/ccoewitlab1-99/.config/spyder-py3/temp.py’, wdir=’/home/ccoewitlab1-99/.config/spyder-py3′)

    File “/home/ccoewitlab1-99/anaconda3/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py”, line 866, in runfile
    execfile(filename, namespace)

    File “/home/ccoewitlab1-99/anaconda3/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py”, line 102, in execfile
    exec(compile(f.read(), filename, ‘exec’), namespace)

    File “/home/ccoewitlab1-99/.config/spyder-py3/temp.py”, line 203, in
    eng_tokenizer = create_tokenizer(dataset[:, 0])

    IndexError: too many indices for array

    Please help I am stuck over this error since two weeks. Your help will be of great value sir.

    • Jason Brownlee March 9, 2019 at 6:26 am #

      Sorry to hear that, what version of Keras are you using?

  103. Satwik March 20, 2019 at 6:29 am #

    TypeError: data type not understood

    This the error i am getting at model.fit()
    Help me solve this pls

  104. Priyanka March 25, 2019 at 4:13 am #

    Hello,
    keras fit_on_texts assign a unique integer to each word. So can you please tell if this considers
    similar words and if yes, how? Thanks in advance

  105. Fadil March 26, 2019 at 9:43 pm #

    Hello, Does this NMT support training long sentences? I am trying to train 3000 sentences of my own and and test data of 900 sentences but after evaluating, all the predicted results are the same characters for all test data.

    • Jason Brownlee March 27, 2019 at 8:59 am #

      Yes, some models can do quite well on long sentences. It really depends on the model and choice of training data.

  106. Poojana March 27, 2019 at 4:18 am #

    What should be the input shape if I am using Bidirectional layer?

  107. Poojana March 27, 2019 at 4:20 am #

    Hello Sir. I am a novice in Deep Learning. Can you suggest me how to use Bidirectional as input layer?

  108. Fadil March 27, 2019 at 9:31 pm #

    Sir, I just started the training using 23000 sentence pair and I am getting this memory error:
    trainY = encode_output(trainY, eng_vocab_size)
    File “nmt.py”, line 43, in encode_output
    y = array(ylist)
    MemoryError

    I have 16gb of ram BTW.

    • Jason Brownlee March 28, 2019 at 8:12 am #

      Perhaps try progressive loading (e.g. a data generator)?
      Perhaps try AWS EC2?
      Perhaps try less data?

  109. Vishesh Srivastava April 1, 2019 at 5:21 am #

    Sir,
    I am a student and i am trying to translate from Hindi to English using your code.The code works fine for training but when it predicts sequences it is giving null.The predicted output is coming as empty.I changed only one thing in the code which is that i transliterated the Hindi devanagiri script to Latin script so that normalization of source language data can be done.Can you give your views on the issue?

  110. Poojana April 4, 2019 at 10:01 pm #

    Hi Sir. I tried doing Hindi to English translator making few changes to your code. I get a bleu of 0.74 for 4 gram but the prediction is very bad. It almost gives the same prediction for all the sentences. Any suggestions on that?

  111. SYED ABDUL BASIT April 13, 2019 at 10:19 pm #

    I got Error in creating Tokenizer

    # fit a tokenizer
    2 def create_tokenizer(lines):
    —-> 3 tokenizer = Tokenizer()
    4 tokenizer.fit_on_texts(lines)
    5 return tokenizer

    NameError: name ‘Tokenizer’ is not defined

    • Jason Brownlee April 14, 2019 at 5:48 am #

      Sorry to hear that, perhaps ensure that you have copied all of the lines of code for the full code example.

    • shantanu May 21, 2019 at 2:13 pm #

      hi, how did you resolve this error @Syed Abdul Basit

    • madhura June 17, 2019 at 8:20 pm #

      ur problem solved?

  112. Jane April 15, 2019 at 4:46 am #

    Hi
    I tried doing English to Indonesian but i have problem with prediction. It predicts only the english words I had, I do, I you repeatedly. But the BLEU value is ok at 0.5 – 9.0.

    I’ve tried fixing the weights on the BLEU code but it remains the same.

    And I you know you’ve said the model needs fine tuning, but can you perhaps suggest what is problem? Is it the tokenizer or not too much training, or verbose?

    I’ve changed every parameter in the Model following your book, but I still get the same result on prediction

    • Jason Brownlee April 15, 2019 at 7:56 am #

      Perhaps confirm that the inputs and outputs in the data are as you expect?

  113. Thirawat April 24, 2019 at 4:35 am #

    Hi , I had a problem that I got the different result when I used the model after it was trained instantly and when I saved and then loaded to use it with other dataset. Maybe it is because the change of tokenizer that is the new data have different tokenizer to encode which is different from tokenizer we create before training . How can I fix this problem , or I have to train the new model every time before use it.

    • Jason Brownlee April 24, 2019 at 8:09 am #

      You must use the same tokenizer. Perhaps save it along with the model or develop a consistent way of creating it?

      • Thirawat April 24, 2019 at 10:31 pm #

        How can I do that ? .Do you have any suggestion ? , Thank you in advance.

        • Jason Brownlee April 25, 2019 at 8:15 am #

          You can use pickle, I have many examples on the blog, including an example of using pickle in the above tutorial.

  114. Zhongpu Chen April 25, 2019 at 1:31 pm #

    There is some errors in the code. To be specific, you have to use to actual.append([raw_target.split()]) according to the definition of the references in corpus-level BLEU score.

  115. Abdullah April 25, 2019 at 6:58 pm #

    Hi Jason ,
    You post one of the best deep learning content on internet. A must appreciated effort.
    Can you please post a project on building an automatic speech recognition in tensorflow using LSTMs and teach how to process the audio data and label sentences. usually ASR with preprocess data is given which is not my requirement. Not only me many people will benefit from it.
    Thank you very much

  116. Priyanka April 28, 2019 at 6:07 am #

    Can you suggest me how to use model.fit_generator here? Thanks in advance

  117. Ash May 3, 2019 at 6:57 am #

    Hi
    The tutorial is really helpful, i would just like to know if you can provide the translation for the whole data set. here I can see only few translations.

    • Jason Brownlee May 3, 2019 at 2:39 pm #

      The dataset used in the post contains all english phrases and their translations.

  118. Roger May 13, 2019 at 1:13 am #

    This started ok but pydot.py was called but not there.

    (base) C:\Users\Roger\Documents\Python Scripts>python model.py
    Using Theano backend.
    English Vocabulary Size: 2233
    English Max Length: 5
    German Vocabulary Size: 3566
    German Max Length: 10
    _________________________________________________________________
    Layer (type) Output Shape Param #
    =================================================================
    embedding_1 (Embedding) (None, 10, 256) 912896
    _________________________________________________________________
    lstm_1 (LSTM) (None, 256) 525312
    _________________________________________________________________
    repeat_vector_1 (RepeatVecto (None, 5, 256) 0
    _________________________________________________________________
    lstm_2 (LSTM) (None, 5, 256) 525312
    _________________________________________________________________
    time_distributed_1 (TimeDist (None, 5, 2233) 573881
    =================================================================
    Total params: 2,537,401
    Trainable params: 2,537,401
    Non-trainable params: 0
    _________________________________________________________________
    None
    Traceback (most recent call last):
    File “model.py”, line 79, in
    plot_model(model, to_file=’model.png’, show_shapes=True)
    File “C:\Users\Roger\Anaconda3\lib\site-packages\keras\utils\vis_utils.py”, line 132, in plot_model
    dot = model_to_dot(model, show_shapes, show_layer_names, rankdir)
    File “C:\Users\Roger\Anaconda3\lib\site-packages\keras\utils\vis_utils.py”, line 55, in model_to_dot
    _check_pydot()
    File “C:\Users\Roger\Anaconda3\lib\site-packages\keras\utils\vis_utils.py”, line 20, in _check_pydot
    ‘Failed to import pydot. ‘
    ImportError: Failed to import pydot. Please install pydot. For example with pip install pydot.

    • Jason Brownlee May 13, 2019 at 6:47 am #

      You can comment out the plot_model line if you like.

  119. an luu May 18, 2019 at 7:57 pm #

    Hi Jason!
    im working on NMT for English to Vietnamese, i went throught all step in the article but my BLEU score is really bad, my custom data set contain 50000 sentences:

    train
    BLEU-1: 0.015223
    BLEU-2: 0.004198
    BLEU-3: 0.003481
    BLEU-4: 0.001052
    test
    BLEU-1: 0.013051
    BLEU-2: 0.001648
    BLEU-3: 0.001538
    BLEU-4: 0.000486

  120. Sravan Malla May 21, 2019 at 11:36 pm #

    Hi Jason,

    may be a very basic thing,but I am not getting the reason behind adding + 1 to voab size
    eng_vocab_size = len(eng_tokenizer.word_index) + 1

    • Jason Brownlee May 22, 2019 at 8:08 am #

      Good question, so that we leave room for 0==no word or “unknown”, therefore the first word in the vocab will be mapped to 1 and we can use 0 for all words we don’t have in our vocab.

      • Sravan Malla May 22, 2019 at 1:05 pm #

        Okay, so when we have that code to replace OOV with UNK in place, we should make sure to have that use ‘0’…am i right

  121. Aniruddha May 21, 2019 at 11:53 pm #

    I get empty string ‘ ‘ as output for predict_sequence() function. Why is it so?

    • Jason Brownlee May 22, 2019 at 8:09 am #

      Perhaps the model did not converge for you, try fitting the model again?

  122. Sravan Malla May 22, 2019 at 1:07 pm #

    Data Cleaning. Different data cleaning operations could be performed on the data, such as not removing punctuation or normalizing case, or perhaps removing duplicate English phrases

    Jason, why do you think removing punctuations or not normalizing cases would help? becasue converting all into one lower case seems to be better idea than leaving as-is. Please share your thoughts

    • Jason Brownlee May 22, 2019 at 2:34 pm #

      It would give a larger vocab and more nuance to the words. It would also require more training data, larger models and long training times.

      • Sravan Malla May 23, 2019 at 4:43 am #

        Jason, I was running the model with larger corpus data i.e. about 160000 records, model stopped after 5th epoch as there isnt any improvement in loss…

        So I though to consider the points listed in your extension and start training, I have modified the model as below i.e. adding Bidirectional (to input/encoder), including more units (256 to 512)and adding dropout aftre encoder and decorder LSTM layers.

        I an not sure how to add more additional layers and where to add them for more represntational capacity…appreciate if you can help me in that.

        # define the model
        model = Sequential()
        model.add(Embedding(ger_vocab_size, 512, input_length=ger_length, mask_zero=True))
        model.add(Bidirectional(LSTM(512), merge_mode=’concat’))
        model.add(Dropout(0.2))
        model.add(RepeatVector(eng_length))
        model.add(LSTM(512, return_sequences=True))
        model.add(Dropout(0.2))
        model.add(TimeDistributed(Dense(eng_vocab_size, activation=’softmax’)))

        • Jason Brownlee May 23, 2019 at 6:10 am #

          I have some suggestions here that might help:
          https://machinelearningmastery.com/start-here/#better

          • Sravan Malla May 23, 2019 at 2:22 pm #

            Jason, Need your help…

            I have gone through one of your post where a baseline model configuration was described.

            Embedding: 512-dimensions
            RNN Cell: Gated Recurrent Unit or GRU
            Encoder: Bidirectional
            Encoder Depth: 2-layers (1 layer in each direction)
            Decoder Depth: 2-layers
            Attention: Bahdanau-style
            Optimizer: Adam
            Dropout: 20% on input

            I am confused on below thing…
            Encoder: Bidirectional
            Encoder Depth: 2-layers (1 layer in each direction)

            Is it how we frame the above Encoder Bidirectional with 2 layers depth (1-layer in each direction)

            model = Sequential()
            model.add(Embedding(ger_vocab_size, 256, input_length=ger_length, mask_zero=True))
            model.add(Bidirectional(LSTM(256, return_sequences=True), merge_mode=’concat’))
            model.add(Bidirectional(LSTM(256, go_backwards=True),merge_mode=’concat’))
            model.add(Dropout(0.2))

          • Jason Brownlee May 23, 2019 at 2:34 pm #

            That looks reasonable, perhaps test it to confirm.

          • Sravan Malla May 24, 2019 at 8:40 pm #

            Sure, I’ll try with it and update you.
            Meanwhile, I have tried with below model architecture passing
            1. Huge data around 160000 (progressive loading through generator),
            2. Limited vocabulary converting words which are occuring less than 5 times to ‘unk’
            2. Increased units i.e. 256 -> 512
            3. Bidirections Input/Encoder Layer
            4. Dropout/regularization for Encoder and Decoder (20%)
            and surprisingly found that loss isn’t decreasing further after 1 or 2 epochs. any ideas? Please help

            model = Sequential()
            model.add(Embedding(ger_vocab_size, 512, input_length=ger_length, mask_zero=True))
            model.add(Bidirectional(LSTM(512), merge_mode=’concat’))
            model.add(Dropout(0.2))
            model.add(RepeatVector(eng_length))
            model.add(LSTM(512, return_sequences=True))
            model.add(Dropout(0.2))
            model.add(TimeDistributed(Dense(eng_vocab_size, activation=’softmax’)))

          • Jason Brownlee May 25, 2019 at 7:48 am #

            Perhaps you need a deeper encoder, decoder or both?
            Perhaps try relu?

            I have a ton of suggestions here:
            https://machinelearningmastery.com/start-here/#better

          • Sravan Malla May 26, 2019 at 4:19 pm #

            Jason, I would try to build a deeper encoder/decoder. meanwhile could you please confirm if we can implement Attention and Beam Search things in Keras, like is Keras supporting ? else how can I try to implement using the base model I pasted above.

          • Jason Brownlee May 27, 2019 at 6:44 am #

            Sorry, I cannot confirm your code/models.

            I recommend testing a suite of approaches in order to discover what works best for your specific dataset.

          • Sravan Malla May 27, 2019 at 5:03 pm #

            Hey Json, I am not asking to confirm my model/code.
            I just want to know, if we have support from Keras in implementing Attention Layers and Beam Search? if so do we have any reference material for that?

          • Jason Brownlee May 28, 2019 at 8:10 am #

            Keras does not support attention or beam search, you must implement them yourself.

            This may help for attention:
            https://machinelearningmastery.com/?s=attention&post_type=post&submit=Search

            This may help for beam search:
            https://machinelearningmastery.com/?s=beam+search&post_type=post&submit=Search

  123. Nandita May 24, 2019 at 6:02 pm #

    i am getting memoryError as:

    Traceback (most recent call last):
    File “cleantext.py”, line 70, in
    save_clean_data(clean_pairs, ‘english-german.pkl’)
    File “cleantext.py”, line 56, in save_clean_data
    dump(sentences, open(filename, ‘wb’))
    MemoryError

    please tell me how to resolve it.

    • Jason Brownlee May 25, 2019 at 7:45 am #

      Sorry to hear that.

      Perhaps try working with less data?
      Perhaps try running on a machine with more memory?

  124. Novie May 24, 2019 at 8:40 pm #

    Error in training neural model:

    Using TensorFlow backend.
    Traceback (most recent call last):
    File “D:\anaconda\envs\myenv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py”, line 58, in
    from tensorflow.python.pywrap_tensorflow_internal import *
    File “D:\anaconda\envs\myenv\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py”, line 28, in
    _pywrap_tensorflow_internal = swig_import_helper()
    File “D:\anaconda\envs\myenv\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py”, line 24, in swig_import_helper
    _mod = imp.load_module(‘_pywrap_tensorflow_internal’, fp, pathname, description)
    File “D:\anaconda\envs\myenv\lib\imp.py”, line 243, in load_module
    return load_dynamic(name, filename, file)
    File “D:\anaconda\envs\myenv\lib\imp.py”, line 343, in load_dynamic
    return _load(spec)
    ImportError: DLL load failed: %1 is not a valid Win32 application.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File “train.py”, line 3, in
    from keras.preprocessing.text import Tokenizer
    File “D:\anaconda\envs\myenv\lib\site-packages\keras\__init__.py”, line 3, in
    from . import utils
    File “D:\anaconda\envs\myenv\lib\site-packages\keras\utils\__init__.py”, line 6, in
    from . import conv_utils
    File “D:\anaconda\envs\myenv\lib\site-packages\keras\utils\conv_utils.py”, line 9, in
    from .. import backend as K
    File “D:\anaconda\envs\myenv\lib\site-packages\keras\backend\__init__.py”, line 89, in
    from .tensorflow_backend import *
    File “D:\anaconda\envs\myenv\lib\site-packages\keras\backend\tensorflow_backend.py”, line 5, in
    import tensorflow as tf
    File “D:\anaconda\envs\myenv\lib\site-packages\tensorflow\__init__.py”, line 22, in
    from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
    File “D:\anaconda\envs\myenv\lib\site-packages\tensorflow\python\__init__.py”, line 49, in
    from tensorflow.python import pywrap_tensorflow
    File “D:\anaconda\envs\myenv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py”, line 74, in
    raise ImportError(msg)
    ImportError: Traceback (most recent call last):
    File “D:\anaconda\envs\myenv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py”, line 58, in
    from tensorflow.python.pywrap_tensorflow_internal import *
    File “D:\anaconda\envs\myenv\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py”, line 28, in
    _pywrap_tensorflow_internal = swig_import_helper()
    File “D:\anaconda\envs\myenv\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py”, line 24, in swig_import_helper
    _mod = imp.load_module(‘_pywrap_tensorflow_internal’, fp, pathname, description)
    File “D:\anaconda\envs\myenv\lib\imp.py”, line 243, in load_module
    return load_dynamic(name, filename, file)
    File “D:\anaconda\envs\myenv\lib\imp.py”, line 343, in load_dynamic
    return _load(spec)
    ImportError: DLL load failed: %1 is not a valid Win32 application.

    PLease someone help me…
    Thankyou…

  125. nandita May 28, 2019 at 11:20 pm #

    how how to find number of sentences in the complete dataset?

    • Jason Brownlee May 29, 2019 at 8:43 am #

      The number of lines in the original files is the number of sentences.

  126. madhura June 17, 2019 at 8:17 pm #

    GETTING ERROR

    NameError: name ‘Tokenizer’ is not defined

  127. Mahmudul Hassan June 23, 2019 at 3:39 am #

    Hello Sir Nice Tutorial .Please Help me how can i translate other sentence like i just want to translate “I Love You” just this sentence using the saved model please help me please?

    • Jason Brownlee June 23, 2019 at 5:41 am #

      You can load the model and make translations directly. What problem are you having exactly?

  128. Antonio June 23, 2019 at 10:28 am #

    Nice tutorial, Jason. When I was reading the section where you clean the text, I was wondering how the commercial translators deal with numbers and other special symbols (e.g., currency). On Google Translator, for instance, If we input a text that contains numbers, uppercase words, and so on; they output the final text with the same symbols. Do you have any idea on how they implement that? I’d like to have an idea on how they recover the information that is “missed” on the preprocessing steps.

    • Jason Brownlee June 24, 2019 at 6:20 am #

      Great question.

      I would expect that system that have access to so much text can actually handle almost all symbols directly. E.g. they’re all used so many times.

  129. Zhongpu Chen July 8, 2019 at 7:27 pm #

    A stupid question. How do you determin a good value for output size of LSTM? model.add(LSTM(n_units)) must I use n_units?

  130. Roger July 12, 2019 at 11:58 pm #

    Hello

    I translate from French into English and have used the French – English corpus. I added some steps to understand better what is happening. I want to get a performance at least as good as Moses and tried to train a model using much more data using your code. I made the dataset 100000 instead of 10000 but got the following error categorical = np.zeros((n, num_classes))
    MemoryError. It seems that my RAM (8GB) is not big enough. I do not use BLUE but evaluate the output manually by post editing. With Moses I used the whole Europarl Corpus and expect to do so with NMT. I need to find a way to train a lot of data on a machine with 8GB memory for it to be any good for use by individuals.

    • Jason Brownlee July 13, 2019 at 6:56 am #

      Perhaps try using less data?
      Perhaps try running on a larger machine, e.g. AWS EC2?
      Perhaps try using a data generator to prepare one batch of data at a time?

  131. Roger July 20, 2019 at 2:49 am #

    Anymore than 10000 sentences and the one hot coding takes up all my memory. I can train moses with 8G of ram but NMT training using your code seems useless to build a serious system. Don’t want to use AWS and how does a data generator work?

    • Roger August 18, 2019 at 6:02 pm #

      I have managed to train a model using 100000 sentences on Floydhub using a machine with 59 GB cpu and 11GB GPU memory. The trouble for me is that this is really a tiny amount of data in MT terms. I used 1million segments to train Moses and Koehn et al show that in tests with Nematus and Moses, Nematus did not equal Moses until this amount of data was used. The problem is that using your method is likely to use resources that are mind blowing, mainly because of all the one hot vectors that have to be stored, wheras I can train my Moses engines on my laptop.

      Going from French to English with 10000 sentences the only thing that can be said is that the output is in English. It has to be, because the words are represented by tokens that are translated into English.
      My next step is to try 150000 sentences because this is all the data.

      In your tutorial you fix the number of epochs to 30. How can I arrange to keep going until the cost is not changing?

  132. Rajat July 21, 2019 at 6:31 pm #

    I love your blog really It is very amazing.
    But I am not getting how to make a prediction with the model. Could you please remove my doubt

  133. Christos Mantas July 29, 2019 at 4:25 am #

    Hello Jason.

    Very nice work here.
    In your code for predict_sequence I think you could use the predict_classes method .

  134. LeonJ August 11, 2019 at 4:27 pm #

    How can I give my own input into the model??????

    • LeonJ August 11, 2019 at 9:08 pm #

      And I have also discovered that this code can able to predict only first two lines of my given input. How can we code to predict and translate the sentences of two or more lines????

    • Jason Brownlee August 12, 2019 at 6:33 am #

      See the evaluate_model() function, for example this line:

      • LeonJ August 12, 2019 at 5:48 pm #

        How can i create my own datasets????

        • Jason Brownlee August 13, 2019 at 6:07 am #

          You can translate yourself?

          It may be too time consuming, perhaps start with public datasets?

  135. Jass August 13, 2019 at 12:33 am #

    I can translate only two words correctly in a sentence. I want to translate completely. How It can be done??? Help me??

    • Jass August 13, 2019 at 1:04 am #

      if i give an example as “Als Jugendlicher war Tom sehr beliebt.” that means “As a teenager, Tom was very popular.” in English but i get output as “you you you to to to you”. Its somewhat translating the sentence partially but how to get the exact output?????

      • Jason Brownlee August 13, 2019 at 6:12 am #

        Perhaps you model has overfit?

        Perhaps try running the example again and fit a new model?

    • Jason Brownlee August 13, 2019 at 6:11 am #

      Perhaps develop a larger model on a different dataset?

      Perhaps use a pre-trained model?

      • Roger August 20, 2019 at 6:11 pm #

        Jass

        This model is only a toy and will not in reality translate anything. This only uses 10K sentences (or segments) and I think you will need to train with getting on for 1 million sentences to get a reasonable performance. Remember that Google have put huge resources into developing neural translation. I found this tutorial very good for learning using python, keras and tensorflow etc but its a bit like learning to weld. In my case the instructor demonstrated how to do it in five minutes, but I never managed to master it.

        • Jason Brownlee August 21, 2019 at 6:37 am #

          Yes, this tutorial (like all of them on my site) is an example for educational purposes.

          It’s a starting point to kick-start your own project.

  136. Roger August 22, 2019 at 5:39 pm #

    The trouble is it does not look like kick-starting my own project. Yesterday I tried to train a model with 149,000 sentences. It had an English vocabulary of 12004 words and max length of 12 but 59 GB memory was not enough. This turns out to be a known problem (Khan 201, p61 https://arxiv.org/abs/1709.07809) and vocabularies are restricted to between 20000 and 80000 words.

    The main lesson here, for me, is that NMT is not all that it is cracked up to be.

  137. Naga August 26, 2019 at 4:10 pm #

    Hi Browniee,
    Thank you very much for nice representation of simple neural machine translation system,
    Could you please provide the link for trained model. I could’nt find over there and i am not been able to train even with the use of gtx-1070 8gb graphics , 12 core processor 😐

    • Jason Brownlee August 27, 2019 at 6:34 am #

      Sorry, I cannot share a link to a trained model. I don’t want to get into the business of hosting models.

  138. Neha Hada August 29, 2019 at 4:26 pm #

    Hi Jason,
    Can you please tell me what code should I use for hindi-english transliteration instead of

    line = normalize(‘NFD’, line).encode(‘ascii’, ‘ignore’)

    line = line.decode(‘UTF-8’) ?

    • Jason Brownlee August 30, 2019 at 6:14 am #

      I am not familiar with that translation, perhaps try experimenting?

  139. Neha Hada August 29, 2019 at 8:12 pm #

    Here’s my output
    train
    src=[फर्ग्यूसन], target=[ferguson], predicted=[raymond]
    src=[काम्प्लैक्स], target=[complex], predicted=[raymond]
    src=[लूकीज], target=[lookeys], predicted=[raymond]
    src=[च्यवनप्राश], target=[chyavanprash], predicted=[raymond]
    src=[कौशल], target=[koshala], predicted=[raymond]
    src=[माइम], target=[mime], predicted=[raymond]
    src=[अनअनपेन्टियम], target=[ununpentium], predicted=[raymond]
    src=[केडीईलिब्स], target=[kdelibs], predicted=[raymond]
    src=[जैस], target=[jais], predicted=[raymond]
    src=[दक्षिणा], target=[dakshina], predicted=[raymond]
    /usr/local/lib/python3.6/dist-packages/nltk/translate/bleu_score.py:490: UserWarning:
    Corpus/Sentence contains 0 counts of 2-gram overlaps.
    BLEU scores might be undesirable; use SmoothingFunction().
    warnings.warn(_msg)
    BLEU-1: 0.000054
    BLEU-2: 0.007376
    BLEU-3: 0.052566
    BLEU-4: 0.085885
    test
    src=[ड्रेगन], target=[dragons], predicted=[raymond]
    src=[कंवर्जेंस], target=[convergence], predicted=[raymond]
    src=[हेदुआ], target=[hedua], predicted=[raymond]
    src=[शुएब], target=[shoaib], predicted=[raymond]
    src=[ब्रेंट], target=[brent], predicted=[raymond]
    src=[करने], target=[kane], predicted=[raymond]
    src=[हेस्टिंग्स], target=[hastings], predicted=[raymond]
    src=[कैप्टिव], target=[captive], predicted=[raymond]
    src=[नाटिका], target=[natika], predicted=[raymond]
    src=[शंभु], target=[sambhu], predicted=[raymond]
    BLEU-1: 0.000000
    BLEU-2: 0.000000
    BLEU-3: 0.000000
    BLEU-4: 0.000000
    Can you please help me to figure out why only one output is coming ?

    • Jason Brownlee August 30, 2019 at 6:17 am #

      Nice progress, hang in there.