How to Develop a Word Embedding Model for Predicting Movie Review Sentiment

Word embeddings are a technique for representing text where different words with similar meaning have a similar real-valued vector representation.

They are a key breakthrough that has led to great performance of neural network models on a suite of challenging natural language processing problems.

In this tutorial, you will discover how to develop word embedding models for neural networks to classify movie reviews.

After completing this tutorial, you will know:

  • How to prepare movie review text data for classification with deep learning methods.
  • How to learn a word embedding as part of fitting a deep learning model.
  • How to learn a standalone word embedding and how to use a pre-trained embedding in a neural network model.

Let’s get started.

How to Develop a Word Embedding Model for Predicting Movie Review Sentiment

How to Develop a Word Embedding Model for Predicting Movie Review Sentiment
Photo by Katrina Br*?#*!@nd, some rights reserved.

Tutorial Overview

This tutorial is divided into 5 parts; they are:

  1. Movie Review Dataset
  2. Data Preparation
  3. Train Embedding Layer
  4. Train word2vec Embedding
  5. Use Pre-trained Embedding

Need help with Deep Learning for Text Data?

Take my free 7-day email crash course now (with code).

Click to sign-up and also get a free PDF Ebook version of the course.

Start Your FREE Crash-Course Now

1. Movie Review Dataset

The Movie Review Data is a collection of movie reviews retrieved from the imdb.com website in the early 2000s by Bo Pang and Lillian Lee. The reviews were collected and made available as part of their research on natural language processing.

The reviews were originally released in 2002, but an updated and cleaned up version were released in 2004, referred to as “v2.0”.

The dataset is comprised of 1,000 positive and 1,000 negative movie reviews drawn from an archive of the rec.arts.movies.reviews newsgroup hosted at imdb.com. The authors refer to this dataset as the “polarity dataset.”

Our data contains 1000 positive and 1000 negative reviews all written before 2002, with a cap of 20 reviews per author (312 authors total) per category. We refer to this corpus as the polarity dataset.

A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts, 2004.

The data has been cleaned up somewhat, for example:

  • The dataset is comprised of only English reviews.
  • All text has been converted to lowercase.
  • There is white space around punctuation like periods, commas, and brackets.
  • Text has been split into one sentence per line.

The data has been used for a few related natural language processing tasks. For classification, the performance of machine learning models (such as Support Vector Machines) on the data is in the range of high 70% to low 80% (e.g. 78%-82%).

More sophisticated data preparation may see results as high as 86% with 10-fold cross validation. This gives us a ballpark of low-to-mid 80s if we were looking to use this dataset in experiments of modern methods.

… depending on choice of downstream polarity classifier, we can achieve highly statistically significant improvement (from 82.8% to 86.4%)

A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts, 2004.

You can download the dataset from here:

After unzipping the file, you will have a directory called “txt_sentoken” with two sub-directories containing the text “neg” and “pos” for negative and positive reviews. Reviews are stored one per file with a naming convention cv000 to cv999 for each neg and pos.

Next, let’s look at loading and preparing the text data.

2. Data Preparation

In this section, we will look at 3 things:

  1. Separation of data into training and test sets.
  2. Loading and cleaning the data to remove punctuation and numbers.
  3. Defining a vocabulary of preferred words.

Split into Train and Test Sets

We are pretending that we are developing a system that can predict the sentiment of a textual movie review as either positive or negative.

This means that after the model is developed, we will need to make predictions on new textual reviews. This will require all of the same data preparation to be performed on those new reviews as is performed on the training data for the model.

We will ensure that this constraint is built into the evaluation of our models by splitting the training and test datasets prior to any data preparation. This means that any knowledge in the data in the test set that could help us better prepare the data (e.g. the words used) are unavailable in the preparation of data used for training the model.

That being said, we will use the last 100 positive reviews and the last 100 negative reviews as a test set (100 reviews) and the remaining 1,800 reviews as the training dataset.

This is a 90% train, 10% split of the data.

The split can be imposed easily by using the filenames of the reviews where reviews named 000 to 899 are for training data and reviews named 900 onwards are for test.

Loading and Cleaning Reviews

The text data is already pretty clean; not much preparation is required.

Without getting bogged down too much in the details, we will prepare the data using the following way:

  • Split tokens on white space.
  • Remove all punctuation from words.
  • Remove all words that are not purely comprised of alphabetical characters.
  • Remove all words that are known stop words.
  • Remove all words that have a length <= 1 character.

We can put all of these steps into a function called clean_doc() that takes as an argument the raw text loaded from a file and returns a list of cleaned tokens. We can also define a function load_doc() that loads a document from file ready for use with the clean_doc() function.

An example of cleaning the first positive review is listed below.

Running the example prints a long list of clean tokens.

There are many more cleaning steps we may want to explore and I leave them as further exercises.

I’d love to see what you can come up with.
Post your approaches and findings in the comments at the end.

Define a Vocabulary

It is important to define a vocabulary of known words when using a bag-of-words or embedding model.

The more words, the larger the representation of documents, therefore it is important to constrain the words to only those believed to be predictive. This is difficult to know beforehand and often it is important to test different hypotheses about how to construct a useful vocabulary.

We have already seen how we can remove punctuation and numbers from the vocabulary in the previous section. We can repeat this for all documents and build a set of all known words.

We can develop a vocabulary as a Counter, which is a dictionary mapping of words and their counts that allow us to easily update and query.

Each document can be added to the counter (a new function called add_doc_to_vocab()) and we can step over all of the reviews in the negative directory and then the positive directory (a new function called process_docs()).

The complete example is listed below.

Running the example shows that we have a vocabulary of 44,276 words.

We also can see a sample of the top 50 most used words in the movie reviews.

Note, that this vocabulary was constructed based on only those reviews in the training dataset.

We can step through the vocabulary and remove all words that have a low occurrence, such as only being used once or twice in all reviews.

For example, the following snippet will retrieve only the tokens that of appears 2 or more times in all reviews.

Running the above example with this addition shows that the vocabulary size drops by a little more than half its size from 44,276 to 25,767 words.

Finally, the vocabulary can be saved to a new file called vocab.txt that we can later load and use to filter movie reviews prior to encoding them for modeling. We define a new function called save_list() that saves the vocabulary to file, with one word per file.

For example:

Running the min occurrence filter on the vocabulary and saving it to file, you should now have a new file called vocab.txt with only the words we are interested in.

The order of words in your file will differ, but should look something like the following:

We are now ready to look at learning features from the reviews.

3. Train Embedding Layer

In this section, we will learn a word embedding while training a neural network on the classification problem.

A word embedding is a way of representing text where each word in the vocabulary is represented by a real valued vector in a high-dimensional space. The vectors are learned in such a way that words that have similar meanings will have similar representation in the vector space (close in the vector space). This is a more expressive representation for text than more classical methods like bag-of-words, where relationships between words or tokens are ignored, or forced in bigram and trigram approaches.

The real valued vector representation for words can be learned while training the neural network. We can do this in the Keras deep learning library using the Embedding layer.

The first step is to load the vocabulary. We will use it to filter out words from movie reviews that we are not interested in.

If you have worked through the previous section, you should have a local file called ‘vocab.txt‘ with one word per line. We can load that file and build a vocabulary as a set for checking the validity of tokens.

Next, we need to load all of the training data movie reviews. For that we can adapt the process_docs() from the previous section to load the documents, clean them, and return them as a list of strings, with one document per string. We want each document to be a string for easy encoding as a sequence of integers later.

Cleaning the document involves splitting each review based on white space, removing punctuation, and then filtering out all tokens not in the vocabulary.

The updated clean_doc() function is listed below.

The updated process_docs() can then call the clean_doc() for each document on the ‘pos‘ and ‘neg‘ directories that are in our training dataset.

The next step is to encode each document as a sequence of integers.

The Keras Embedding layer requires integer inputs where each integer maps to a single token that has a specific real-valued vector representation within the embedding. These vectors are random at the beginning of training, but during training become meaningful to the network.

We can encode the training documents as sequences of integers using the Tokenizer class in the Keras API.

First, we must construct an instance of the class then train it on all documents in the training dataset. In this case, it develops a vocabulary of all tokens in the training dataset and develops a consistent mapping from words in the vocabulary to unique integers. We could just as easily develop this mapping ourselves using our vocabulary file.

Now that the mapping of words to integers has been prepared, we can use it to encode the reviews in the training dataset. We can do that by calling the texts_to_sequences() function on the Tokenizer.

We also need to ensure that all documents have the same length.

This is a requirement of Keras for efficient computation. We could truncate reviews to the smallest size or zero-pad (pad with the value ‘0’) reviews to the maximum length, or some hybrid. In this case, we will pad all reviews to the length of the longest review in the training dataset.

First, we can find the longest review using the max() function on the training dataset and take its length. We can then call the Keras function pad_sequences() to pad the sequences to the maximum length by adding 0 values on the end.

Finally, we can define the class labels for the training dataset, needed to fit the supervised neural network model to predict the sentiment of reviews.

We can then encode and pad the test dataset, needed later to evaluate the model after we train it.

We are now ready to define our neural network model.

The model will use an Embedding layer as the first hidden layer. The Embedding requires the specification of the vocabulary size, the size of the real-valued vector space, and the maximum length of input documents.

The vocabulary size is the total number of words in our vocabulary, plus one for unknown words. This could be the vocab set length or the size of the vocab within the tokenizer used to integer encode the documents, for example:

We will use a 100-dimensional vector space, but you could try other values, such as 50 or 150. Finally, the maximum document length was calculated above in the max_length variable used during padding.

The complete model definition is listed below including the Embedding layer.

We use a Convolutional Neural Network (CNN) as they have proven to be successful at document classification problems. A conservative CNN configuration is used with 32 filters (parallel fields for processing words) and a kernel size of 8 with a rectified linear (‘relu’) activation function. This is followed by a pooling layer that reduces the output of the convolutional layer by half.

Next, the 2D output from the CNN part of the model is flattened to one long 2D vector to represent the ‘features’ extracted by the CNN. The back-end of the model is a standard Multilayer Perceptron layers to interpret the CNN features. The output layer uses a sigmoid activation function to output a value between 0 and 1 for the negative and positive sentiment in the review.

Running just this piece provides a summary of the defined network.

We can see that the Embedding layer expects documents with a length of 442 words as input and encodes each word in the document as a 100 element vector.

Next, we fit the network on the training data.

We use a binary cross entropy loss function because the problem we are learning is a binary classification problem. The efficient Adam implementation of stochastic gradient descent is used and we keep track of accuracy in addition to loss during training. The model is trained for 10 epochs, or 10 passes through the training data.

The network configuration and training schedule were found with a little trial and error, but are by no means optimal for this problem. If you can get better results with a different configuration, let me know.

After the model is fit, it is evaluated on the test dataset. This dataset contains words that we have not seen before and reviews not seen during training.

We can tie all of this together.

The complete code listing is provided below.

Running the example prints the loss and accuracy at the end of each training epoch. We can see that the model very quickly achieves 100% accuracy on the training dataset.

At the end of the run, the model achieves an accuracy of 84.5% on the test dataset, which is a great score.

Given the stochastic nature of neural networks, your specific results will vary. Consider running the example a few times and taking the average score as the skill of the model.

We have just seen an example of how we can learn a word embedding as part of fitting a neural network model.

Next, let’s look at how we can efficiently learn a standalone embedding that we could later use in our neural network.

4. Train word2vec Embedding

In this section, we will discover how to learn a standalone word embedding using an efficient algorithm called word2vec.

A downside of learning a word embedding as part of the network is that it can be very slow, especially for very large text datasets.

The word2vec algorithm is an approach to learning a word embedding from a text corpus in a standalone way. The benefit of the method is that it can produce high-quality word embeddings very efficiently, in terms of space and time complexity.

The first step is to prepare the documents ready for learning the embedding.

This involves the same data cleaning steps from the previous section, namely splitting documents by their white space, removing punctuation, and filtering out tokens not in the vocabulary.

The word2vec algorithm processes documents sentence by sentence. This means we will preserve the sentence-based structure during cleaning.

We start by loading the vocabulary, as before.

Next, we define a function named doc_to_clean_lines() to clean a loaded document line by line and return a list of the cleaned lines.

Next, we adapt the process_docs() function to load and clean all of the documents in a folder and return a list of all document lines.

The results from this function will be the training data for the word2vec model.

We can then load all of the training data and convert it into a long list of ‘sentences’ (lists of tokens) ready for fitting the word2vec model.

We will use the word2vec implementation provided in the Gensim Python library. Specifically the Word2Vec class.

The model is fit when constructing the class. We pass in the list of clean sentences from the training data, then specify the size of the embedding vector space (we use 100 again), the number of neighboring words to look at when learning how to embed each word in the training sentences (we use 5 neighbors), the number of threads to use when fitting the model (we use 8, but change this if you have more or less CPU cores), and the minimum occurrence count for words to consider in the vocabulary (we set this to 1 as we have already prepared the vocabulary).

After the model is fit, we print the size of the learned vocabulary, which should match the size of our vocabulary in vocab.txt of 25,767 tokens.

Finally, we save the learned embedding vectors to file using the save_word2vec_format() on the model’s ‘wv‘ (word vector) attribute. The embedding is saved in ASCII format with one word and vector per line.

The complete example is listed below.

Running the example loads 58,109 sentences from the training data and creates an embedding for a vocabulary of 25,767 words.

You should now have a file ’embedding_word2vec.txt’ with the learned vectors in your current working directory.

Next, let’s look at using these learned vectors in our model.

5. Use Pre-trained Embedding

In this section, we will use a pre-trained word embedding prepared on a very large text corpus.

We can use the pre-trained word embedding developed in the previous section and the CNN model developed in the section before that.

The first step is to load the word embedding as a directory of words to vectors. The word embedding was saved in so-called ‘word2vec‘ format that contains a header line. We will skip this header line when loading the embedding.

The function below named load_embedding() loads the embedding and returns a directory of words mapped to the vectors in NumPy format.

Now that we have all of the vectors in memory, we can order them in such a way as to match the integer encoding prepared by the Keras Tokenizer.

Recall that we integer encode the review documents prior to passing them to the Embedding layer. The integer maps to the index of a specific vector in the embedding layer. Therefore, it is important that we lay the vectors out in the Embedding layer such that the encoded words map to the correct vector.

Below defines a function get_weight_matrix() that takes the loaded embedding and the tokenizer.word_index vocabulary as arguments and returns a matrix with the word vectors in the correct locations.

Now we can use these functions to create our new Embedding layer for our model.

Note that the prepared weight matrix embedding_vectors is passed to the new Embedding layer as an argument and that we set the ‘trainable‘ argument to ‘False‘ to ensure that the network does not try to adapt the pre-learned vectors as part of training the network.

We can now add this layer to our model. We also have a slightly different model configuration with a lot more filters (128) in the CNN model and a kernel that matches the 5 words used as neighbors when developing the word2vec embedding. Finally, the back-end of the model was simplified.

These changes were found with a little trial and error.

The complete code listing is provided below.

Running the example shows that performance was not improved.

In fact, performance was a lot worse. The results show that the training dataset was learned successfully, but evaluation on the test dataset was very poor, at just above 50% accuracy.

The cause of the poor test performance may be because of the chosen word2vec configuration or the chosen neural network configuration.

The weights in the embedding layer can be used as a starting point for the network, and adapted during the training of the network. We can do this by setting ‘trainable=True‘ (the default) in the creation of the embedding layer.

Repeating the experiment with this change shows slightly better results, but still poor.

I would encourage you to explore alternate configurations of the embedding and network to see if you can do better. Let me know how you do.

It is possible to use pre-trained word vectors prepared on very large corpora of text data.

For example, both Google and Stanford provide pre-trained word vectors that you can download, trained with the efficient word2vec and GloVe methods respectively.

Let’s try to use pre-trained vectors in our model.

You can download pre-trained GloVe vectors from the Stanford webpage. Specifically, vectors trained on Wikipedia data:

Unzipping the file, you will find pre-trained embeddings for various different dimensions. We will load the 100 dimension version in the file ‘glove.6B.100d.txt

The Glove file does not contain a header file, so we do not need to skip the first line when loading the embedding into memory. The updated load_embedding() function is listed below.

It is possible that the loaded embedding does not contain all of the words in our chosen vocabulary. As such, when creating the Embedding weight matrix, we need to skip words that do not have a corresponding vector in the loaded GloVe data. Below is the updated, more defensive version of the get_weight_matrix() function.

We can now load the GloVe embedding and create the Embedding layer as before.

We will use the same model as before.

The complete example is listed below.

Running the example shows better performance.

Again, the training dataset is easily learned and the model achieves 76% accuracy on the test dataset. This is good, but not as good as using a learned Embedding layer.

This may be cause of the higher quality vectors trained on more data and/or using a slightly different training process.

Your specific results may vary given the stochastic nature of neural networks. Try running the example a few times.

In this case, it seems that learning the embedding as part of the learning task may be a better direction than using a specifically trained embedding or a more general pre-trained embedding.

Further Reading

This section provides more resources on the topic if you are looking go deeper.

Dataset

APIs

Embedding Methods

Related Posts

Summary

In this tutorial, you discovered how to develop word embeddings for the classification of movie reviews.

Specifically, you learned:

  • How to prepare movie review text data for classification with deep learning methods.
  • How to learn a word embedding as part of fitting a deep learning model.
  • How to learn a standalone word embedding and how to use a pre-trained embedding in a neural network model.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.


Develop Deep Learning models for Text Data Today!

Deep Learning for Natural Language Processing

Develop Your Own Text models in Minutes

…with just a few lines of python code

Discover how in my new Ebook:
Deep Learning for Natural Language Processing

It provides self-study tutorials on topics like:
Bag-of-Words, Word Embedding, Language Models, Caption Generation, Text Translation and much more…

Finally Bring Deep Learning to your Natural Language Processing Projects

Skip the Academics. Just Results.

Click to learn more.


58 Responses to How to Develop a Word Embedding Model for Predicting Movie Review Sentiment

  1. Alexander October 30, 2017 at 7:11 pm #

    Thank you for interest work.
    Jason, help me please.
    Imagine that we have an dataset which contains reviews with very different lengths (from just two words “good film” to long description “I remember the first work of this director….”.
    Which length should we choose when we talk about “pad_sequence”?

    • Jason Brownlee October 31, 2017 at 5:32 am #

      All reviews must be padded to the same length.

      You can use the longest review, you can use the average length, etc. Try a few and see what works best.

  2. JingChunzhen October 31, 2017 at 2:37 am #

    Hi Jason
    I think it’s difficult to distinguish ”good” and “bad” using word2vec, but it’s important in sentiment analysis anyway .
    Is there a good method to solve this ? Thank you and sorry for my poor English .

    • Jason Brownlee October 31, 2017 at 5:35 am #

      Generally, the model will minimize a loss function. We want the loss to be low.

      I agree, it feels like clustering to some degree.

  3. Alexander October 31, 2017 at 10:38 pm #

    Thanks for feedback.
    Jason, sorry i don’t understand one thing. Help me please.
    I read you works about technique “Word Embeddings”. And I saw that when you use Embedding layer with Convolutional layer as feature extractor you always use 1D version.
    Since the machine sees the sentence in form of matrix why we don’t base on Conv2D?

    • Jason Brownlee November 1, 2017 at 5:47 am #

      Good question.

      A sequence of words is a one dimensions vector of integers. The embedding maps each integer onto a real vector, so now we have a sequence of vectors. It is still a one-dimensional sequence, it just so happens each item in the sequence has many features.

      It is not like an image that has spatial relationships in two dimensions.

      Does that help?

  4. Alexander November 1, 2017 at 6:19 pm #

    Thank for clear explanation, Jason.
    I try to find an opportunity to use knowledges we have in Embedding layer the best way.

  5. Ramesh Subrahmanyam November 5, 2017 at 9:12 am #

    Excellent post. Thanks.

  6. Amit Adesara November 15, 2017 at 9:54 pm #

    Thank you for amazing posts. Eagerly waiting for your new book launch. I had one query on test,train split. Rather than defining a X and Y test/train, can’t we just use split model from sklearn. Doesn’t is automatically split the entire dataset into train and test? Thank you.

    • Jason Brownlee November 16, 2017 at 10:30 am #

      Thanks Amit!

      Yes, you can use sklearn if you wish.

  7. Abdur Rehman Nadeem December 15, 2017 at 8:50 am #

    Hi jason,

    Sorry once again I am asking the same question which I mentioned in some other blog that what if I am using my own dataset. How can I make my dataset compatible to the above pre-built dataset ? I have a dataset of tweets so how can i make my dataset of positive and negative tweets and then use them as a test and train set ?

  8. Vladimir January 23, 2018 at 1:14 am #

    Thank you Jason for a valuable article! It is an exploration on it’s own.
    I also get better results if Embedding Layer is learned as part of the model, rather than using predifined embeddings.

    First I was very much excited about using GloVe embeddings, yet it gives quite poor results. I think it is because real-world texts contain different word forms (for. example: cat, cats), while GloVe has embeddings only for a single form of each word (for. example: cat). And even a small deviation is considered as a different word, and thus ignored. Probably if this “word-forms” problem is solved, GloVe could show tremendous results…

    • Jason Brownlee January 23, 2018 at 8:04 am #

      You may be right in the general case. It is also critical that the vocab/word usage/etc is a good fit for the problem being solved.

  9. Michelle February 27, 2018 at 2:05 pm #

    Hi,

    I am following this example to classify malicious urls. I am able to filter out words, encode and pad my training and testing data. However, when I apply the training data to the model, I get errors.

    Do you have the code up on github? I’d like to run through the code to debug.

    Thanks!

  10. A.S March 13, 2018 at 7:56 pm #

    Hi, This is very helpful thanks! Is there a java version of this please?
    Thanks

    • Jason Brownlee March 14, 2018 at 6:17 am #

      Sorry, I do not have Java versions at this stage.

  11. pANTHER March 24, 2018 at 9:48 pm #

    how this fucntion

    “encoded_docs = tokenizer.texts_to_sequences(test_docs)”

    index test vocabs when we didn’t call

    “tokenizer.fit_on_texts(train_docs)”

    on test_docs.

    • Jason Brownlee March 25, 2018 at 6:29 am #

      It uses the vocab learned from the training data to index words in test. If there are new works, they are marked as zero.

      • pANTHER April 1, 2018 at 1:31 am #

        Thanks, really appreciate your help.

  12. pANTHER March 24, 2018 at 11:35 pm #

    can we use CNN with multiclass problem?

  13. Jin April 23, 2018 at 3:32 am #

    Hi Jason, thanks for your sharing, For word embedding layer in keras, what is the window size? For example, in word2vec, we can set the window size to 5, but in keras embedding layer, there is no parameter for that.

    • Jason Brownlee April 23, 2018 at 6:23 am #

      The window size is the number of words around each word to consider, e.g. the context, when building the model.

      Keras does not need this, as the embedding is built via back-propagating error during training, ideally using BPTT which builds up error over input time steps.

      From my experiments, I find the Keras approach results in better skill than using a prebuilt model. It should be the other way around according to intuition, but not in my experience.

      • Jin Zhou April 23, 2018 at 10:45 am #

        I read your blog about building language model with LSTM, and I tried work with pre-trained embedding weights, actually results in lower perplexity, but a little lower BLEU. So, it is hard for me to say which one is better.

        Thanks for your blogs! I learned a lot from them.

  14. Anam May 8, 2018 at 4:15 pm #

    Dear Jason,
    In above post, you have used a CNN model after data preparation and embedding layer training tasks. Kindly, can you help that how I can perform
    (i) .Data Preparation
    (ii).Train Embedding Layer
    to use in LSTM Recurrent Neural Networks instead of CNN OR How can I use LSTM Recurrent Neural Networks instead of CNN after these two tasks?
    Thanks for your time!

  15. Anam May 9, 2018 at 2:59 pm #

    Thanks Sir for your guidance.

  16. Maryam June 13, 2018 at 8:04 am #

    Hi Jason,
    I am so grateful for the practical tutorial. I am a beginner at keras and deep learning. I applied a data set which is about 2 diseases. I wanna to train RNN via GloVe in order to classification but the output result is as follow :
    precision 0.0
    FPR 0.9999999999807433
    TPR 0.0
    FNR 0.999999999980609
    specificity 0.0
    accuracy 0.0
    F_score 0.0

    I think just because GloVe did not pre-train for disease dataset???
    am I right ?? or it needs to train in many eopche?
    thank you for guiding me.
    Best
    Maram

  17. Sarah June 21, 2018 at 9:51 am #

    Hi Jason,
    that was awesome like the others tutorials. I do not have a separate data set for train dataset and test data set as I want to select test dataset randomly each time. so I have already done it by this statment:
    ‘from sklearn.model_selection import train_test_split
    X_train, X_test, y_train, y_test = train_test_split(x_datasetpad,y_datasetpad,stratify=y_datasetpad,test_size=0.25)’
    according to your tutorial, you set ytest in this way :
    ytest = array([0 for _ in range(100)] + [1 for _ in range(100)])

    I have already created x_dataset but I do not know how i can create ydataset??
    please guide me Jason, I realy need it.

    Sarah

    • Jason Brownlee June 21, 2018 at 4:50 pm #

      The output will be the sentiment, or an integer representing the sentiment classes for each input review.

  18. Hegg June 21, 2018 at 11:59 pm #

    Hello Jason,

    I was following your awesome tutorial but in the end I was asking myself: How can I predict a new sentence represented as a string for its sentiment? I think there will be some more people who would like to test their trained network for some example inputs. Could you maybe add such a section to this tutorial or give a short explanation on how to do this exactly? In such a way:

    exampleSentence = ‘The weather is very good today’
    prediction = trainedModel.predict(exampleSentence)

    print(sentimentForPrediction(prediction))

    Kind regards

    Hegg

    • Jason Brownlee June 22, 2018 at 6:10 am #

      Yes, you can use:;

      • Srijan Verma July 18, 2018 at 12:35 am #

        Hi Jason,
        I tried implementing one of your codes for predicting the sentiment for a sentence.
        First i trained the model the way you have. I am getting an accuracy of 86% on test data.
        This is the function that i used for predicting on a new sentence.

        # classify a review as negative (0) or positive (1)
        def predict_sentiment(review, vocab, tokenizer, model):
        # clean
        tokens = clean_doc(review)
        # filter by vocab
        tokens = [w for w in tokens if w in vocab]
        # convert to line
        line = ‘ ‘.join(tokens)
        #print(line)
        # encode
        tokenizer.fit_on_texts(line)
        encoded = tokenizer.texts_to_sequences(line)
        #print(encoded)
        # prediction
        max_length = max([len(s.split()) for s in line])
        pred = pad_sequences(encoded, maxlen=1317, padding=’post’)
        #print(pred)
        yhat = model.predict(pred, verbose=0)
        return round(yhat[0,0])

        # test positive text
        text = ‘this is a good movie’
        print(predict_sentiment(text, vocab, tokenizer, model))
        # test negative text
        text = ‘This is a bad movie.’
        print(predict_sentiment(text, vocab, tokenizer, model))

        somehow, I am getting a ‘0’ in both the cases. I tried using other test cases as well, but it is always giving a zero.

        I cant find where I am going wrong.

        Any insights would be really helpful.

        Thanks!

        • Jason Brownlee July 18, 2018 at 6:37 am #

          Sorry, I don’t have the capacity to debug your code. Perhaps try stackoverflow?

          • Srijan July 18, 2018 at 7:08 pm #

            Let us say I have a raw text which I want to predict using model.predict().
            First step would be to assign the same index value (on which the model was trained on) to each of the words in the raw text .
            Then we would have to convert it to sequences (using texts_to_sequences())
            Then will be the padding (max_length would be the same of that of the training set)
            And then this will be given to the model.predict()

            Is that correct ? ^

          • Jason Brownlee July 19, 2018 at 7:48 am #

            Yes. All preparation performed to training data must also be performed to any new data when you want to make a prediction with the trained model.

  19. Sara June 26, 2018 at 12:25 am #

    Hi Jason,

    Thank you so much for your great post. I always enjoy your work and they help me a lot.

    My question is that how can I handle other features and text feature at the same time in my classification. I mean if I have other features like the number of words, the number of punctuation, category, etc for each review how can I use them in my classification.

    Best

  20. Anjali Batra July 31, 2018 at 2:18 am #

    Hi Jason, awesome article. Can you please tell which python version can I use to run the code?
    As nltk wont work on pythong 2.7 and Keras is not available for python 3.6.

    Looking forward to your response.

    Thanks,
    Anjali Batra

    • Jason Brownlee July 31, 2018 at 6:10 am #

      Python 3.5 or 3.6. I believe Python 2.7 would work too, perhaps with minor tweaks.

  21. Shreyas Jeet July 31, 2018 at 7:28 pm #

    Hi Jason,
    Great tutorial, helped a lot.
    I am working on a multi-class classification , could you help me on how I would have to tweak the network to facilitate a 8 class classifier.

    Thank You

    • Jason Brownlee August 1, 2018 at 7:42 am #

      Change the number of nodes in the output layer to 8, the transfer function to softmax and the loss function to categorical cross entropy.

  22. Alberto August 27, 2018 at 1:15 am #

    Hi Jason,
    I’ve a question. To develop the vocab I’ve to use both training and testing reviews?
    Thanks a lot.

    • Jason Brownlee August 27, 2018 at 6:11 am #

      Ideally, you would just use training data to develop your vocab when evaluating the model.

      You would use all data when preparing the final model.

      It really comes down to your project goals.

  23. Arghyadeep Giri September 20, 2018 at 1:38 pm #

    Dear Jason,

    Absolutely loved the tutorial. Nevertheless, I had a question regarding the convolution. You already answered a question regarding that, but I still have a doubt. You mentioned that the output of the embedding layer is a one-dimensional sequence.
    But the output shape of the embedding layer is (None, 1317, 100). I know the “None” refers to the batch size of the number of training examples. But, how is convolution possible with a (1317,100) shape with a one-dimensional filter size of 8. I want to get into the depth of it. How are these values getting multiplied? Since the output shape of the convoluted layer is (1310,32), the only way I can make sense is that for every filter there are 100 values which are getting multiplied with every sequence for a certain word. Kindly correct me if I am wrong.

    Regards,
    Arghyadeep

    • Jason Brownlee September 20, 2018 at 2:28 pm #

      No, the input is a sequence of integers. The output is a sequence of vectors.

      • Arghyadeep Giri September 20, 2018 at 3:04 pm #

        Yes, absolutely! But I believe that is for the embedding layer.

        But my question is about the convolution layer. How is the input shape (None, 1317, 100) getting converted to (None, 1310, 32)?

        The 32 comes from the number of filters.
        1317 gets converted to 1310 for (1317-8+1) which is the kernel size.
        What about the number 100?

        • Jason Brownlee September 21, 2018 at 6:22 am #

          The 100 is the length of each word vector.

          • Arghyadeep Giri September 21, 2018 at 10:05 am #

            Thanks 🙂

Leave a Reply