Sentiment analysis is a natural language processing problem where text is understood and the underlying intent is predicted.

In this post, you will discover how you can predict the sentiment of movie reviews as either positive or negative in Python using the Keras deep learning library.

After reading this post you will know:

- About the IMDB sentiment analysis problem for natural language processing and how to load it in Keras.
- How to use word embedding in Keras for natural language problems.
- How to develop and evaluate a multi-layer perception model for the IMDB problem.
- How to develop a one-dimensional convolutional neural network model for the IMDB problem.

Let’s get started.

**Update Oct/2016**: Updated examples for Keras 1.1.0 and TensorFlow 0.10.0.**Update Mar/2017**: Updated example for Keras 2.0.2, TensorFlow 1.0.1 and Theano 0.9.0.

## IMDB Movie Review Sentiment Problem Description

The dataset is the Large Movie Review Dataset often referred to as the IMDB dataset.

The Large Movie Review Dataset (often referred to as the IMDB dataset) contains 25,000 highly polar moving reviews (good or bad) for training and the same amount again for testing. The problem is to determine whether a given moving review has a positive or negative sentiment.

The data was collected by Stanford researchers and was used in a 2011 paper [PDF] where a split of 50/50 of the data was used for training and test. An accuracy of 88.89% was achieved.

The data was also used as the basis for a Kaggle competition titled “Bag of Words Meets Bags of Popcorn” in late 2014 to early 2015. Accuracy was achieved above 97% with winners achieving 99%.

## Beat the Math/Theory Doldrums and Start using Deep Learning in your own projects Today, without getting lost in “documentation hell”

Get my free Deep Learning With Python mini course and develop your own deep nets by the time you’ve finished the first PDF with just a few lines of Python.

#### Daily lessons in your inbox for 14 days, and a DL-With-Python “Cheat Sheet” you can download right now.

## Load the IMDB Dataset With Keras

Keras provides access to the IMDB dataset built-in.

The keras.datasets.imdb.load_data() allows you to load the dataset in a format that is ready for use in neural network and deep learning models.

The words have been replaced by integers that indicate the absolute popularity of the word in the dataset. The sentences in each review are therefore comprised of a sequence of integers.

Calling imdb.load_data() the first time will download the IMDB dataset to your computer and store it in your home directory under ~/.keras/datasets/imdb.pkl as a 32 megabyte file.

Usefully, the imdb.load_data() provides additional arguments including the number of top words to load (where words with a lower integer are marked as zero in the returned data), the number of top words to skip (to avoid the “the”‘s) and the maximum length of reviews to support.

Let’s load the dataset and calculate some properties of it. We will start off by loading some libraries and loading the entire IMDB dataset as a training dataset.

1 2 3 4 5 6 7 |
import numpy from keras.datasets import imdb from matplotlib import pyplot # load the dataset (X_train, y_train), (X_test, y_test) = imdb.load_data() X = numpy.concatenate((X_train, X_test), axis=0) y = numpy.concatenate((y_train, y_test), axis=0) |

Next we can display the shape of the training dataset.

1 2 3 4 |
# summarize size print("Training data: ") print(X.shape) print(y.shape) |

Running this snippet, we can see that there are 50,000 records.

1 2 3 |
Training data: (50000,) (50000,) |

We can also print the unique class values.

1 2 3 |
# Summarize number of classes print("Classes: ") print(numpy.unique(y)) |

We can see that it is a binary classification problem for good and bad sentiment in the review.

1 2 |
Classes: [0 1] |

Next we can get an idea of the total number of unique words in the dataset.

1 2 3 |
# Summarize number of words print("Number of words: ") print(len(numpy.unique(numpy.hstack(X)))) |

Interestingly, we can see that there are just under 100,000 words across the entire dataset.

1 2 |
Number of words: 88585 |

Finally, we can get an idea of the average review length.

1 2 3 4 5 6 7 |
# Summarize review length print("Review length: ") result = [len(x) for x in X] print("Mean %.2f words (%f)" % (numpy.mean(result), numpy.std(result))) # plot review length pyplot.boxplot(result) pyplot.show() |

We can see that the average review has just under 300 words with a standard deviation of just over 200 words.

1 2 |
Review length: Mean 234.76 words (172.911495) |

Looking a box and whisker plot for the review lengths in words, we can probably see an exponential distribution that we can probably cover the mass of the distribution with a clipped length of 400 to 500 words.

## Word Embeddings

A recent breakthrough in the field of natural language processing is called word embedding.

This is a technique where words are encoded as real-valued vectors in a high-dimensional space, where the similarity between words in terms of meaning translates to closeness in the vector space.

Discrete words are mapped to vectors of continuous numbers. This is useful when working with natural language problems with neural networks and deep learning models are we require numbers as input.

Keras provides a convenient way to convert positive integer representations of words into a word embedding by an Embedding layer.

The layer takes arguments that define the mapping including the maximum number of expected words also called the vocabulary size (e.g. the largest integer value that will be seen as an integer). The layer also allows you to specify the dimensionality for each word vector, called the output dimension.

We would like to use a word embedding representation for the IMDB dataset.

Let’s say that we are only interested in the first 5,000 most used words in the dataset. Therefore our vocabulary size will be 5,000. We can choose to use a 32-dimension vector to represent each word. Finally, we may choose to cap the maximum review length at 500 words, truncating reviews longer than that and padding reviews shorter than that with 0 values.

We would load the IMDB dataset as follows:

1 |
imdb.load_data(nb_words=5000) |

We would then use the Keras utility to truncate or pad the dataset to a length of 500 for each observation using the sequence.pad_sequences() function.

1 2 |
X_train = sequence.pad_sequences(X_train, maxlen=500) X_test = sequence.pad_sequences(X_test, maxlen=500) |

Finally, later on, the first layer of our model would be an word embedding layer created using the Embedding class as follows:

1 |
Embedding(5000, 32, input_length=500) |

The output of this first layer would be a matrix with the size 32×500 for a given review training or test pattern in integer format.

Now that we know how to load the IMDB dataset in Keras and how to use a word embedding representation for it, let’s develop and evaluate some models.

## Simple Multi-Layer Perceptron Model for the IMDB Dataset

We can start off by developing a simple multi-layer perceptron model with a single hidden layer.

The word embedding representation is a true innovation and we will demonstrate what would have been considered world class results in 2011 with a relatively simple neural network.

Let’s start off by importing the classes and functions required for this model and initializing the random number generator to a constant value to ensure we can easily reproduce the results.

1 2 3 4 5 6 7 8 9 10 11 |
# MLP for the IMDB problem import numpy from keras.datasets import imdb from keras.models import Sequential from keras.layers import Dense from keras.layers import Flatten from keras.layers.embeddings import Embedding from keras.preprocessing import sequence # fix random seed for reproducibility seed = 7 numpy.random.seed(seed) |

Next we will load the IMDB dataset. We will simplify the dataset as discussed during the section on word embeddings. Only the top 5,000 words will be loaded.

We will also use a 50%/50% split of the dataset into training and test. This is a good standard split methodology.

1 2 3 |
# load the dataset but only keep the top n words, zero the rest top_words = 5000 (X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=top_words) |

We will bound reviews at 500 words, truncating longer reviews and zero-padding shorter reviews.

1 2 3 |
max_words = 500 X_train = sequence.pad_sequences(X_train, maxlen=max_words) X_test = sequence.pad_sequences(X_test, maxlen=max_words) |

Now we can create our model. We will use an Embedding layer as the input layer, setting the vocabulary to 5,000, the word vector size to 32 dimensions and the input_length to 500. The output of this first layer will be a 32×500 sized matrix as discussed in the previous section.

We will flatten the Embedded layers output to one dimension, then use one dense hidden layer of 250 units with a rectifier activation function. The output layer has one neuron and will use a sigmoid activation to output values of 0 and 1 as predictions.

The model uses logarithmic loss and is optimized using the efficient ADAM optimization procedure.

1 2 3 4 5 6 7 8 |
# create the model model = Sequential() model.add(Embedding(top_words, 32, input_length=max_words)) model.add(Flatten()) model.add(Dense(250, activation='relu')) model.add(Dense(1, activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) print(model.summary()) |

We can fit the model and use the test set as validation while training. This model overfits very quickly so we will use very few training epochs, in this case just 2.

There is a lot of data so we will use a batch size of 128. After the model is trained, we evaluate its accuracy on the test dataset.

1 2 3 4 5 |
# Fit the model model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=2, batch_size=128, verbose=2) # Final evaluation of the model scores = model.evaluate(X_test, y_test, verbose=0) print("Accuracy: %.2f%%" % (scores[1]*100)) |

Running this example fits the model and summarizes the estimated performance. We can see that this very simple model achieves a score of nearly 86.94% which is in the neighborhood of the original paper, with very little effort.

1 2 3 4 5 6 |
Train on 25000 samples, validate on 25000 samples Epoch 1/2 39s - loss: 0.5160 - acc: 0.7040 - val_loss: 0.2982 - val_acc: 0.8716 Epoch 2/2 37s - loss: 0.1897 - acc: 0.9266 - val_loss: 0.3143 - val_acc: 0.8694 Accuracy: 86.94% |

I’m sure we can do better if we trained this network, perhaps using a larger embedding and adding more hidden layers. Let’s try a different network type.

## One-Dimensional Convolutional Neural Network Model for the IMDB Dataset

Convolutional neural networks were designed to honor the spatial structure in image data whilst being robust to the position and orientation of learned objects in the scene.

This same principle can be used on sequences, such as the one-dimensional sequence of words in a movie review. The same properties that make the CNN model attractive for learning to recognize objects in images can help to learn structure in paragraphs of words, namely the techniques invariance to the specific position of features.

Keras supports one-dimensional convolutions and pooling by the Conv1D and MaxPooling1D classes respectively.

Again, let’s import the classes and functions needed for this example and initialize our random number generator to a constant value so that we can easily reproduce results.

1 2 3 4 5 6 7 8 9 10 11 12 13 |
# CNN for the IMDB problem import numpy from keras.datasets import imdb from keras.models import Sequential from keras.layers import Dense from keras.layers import Flatten from keras.layers.convolutional import Conv1D from keras.layers.convolutional import MaxPooling1D from keras.layers.embeddings import Embedding from keras.preprocessing import sequence # fix random seed for reproducibility seed = 7 numpy.random.seed(seed) |

We can also load and prepare our IMDB dataset as we did before.

1 2 3 4 5 6 7 |
# load the dataset but only keep the top n words, zero the rest top_words = 5000 (X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=top_words) # pad dataset to a maximum review length in words max_words = 500 X_train = sequence.pad_sequences(X_train, maxlen=max_words) X_test = sequence.pad_sequences(X_test, maxlen=max_words) |

We can now define our convolutional neural network model. This time, after the Embedding input layer, we insert a Conv1D layer. This convolutional layer has 32 feature maps and reads embedded word representations 3 vector elements of the word embedding at a time.

The convolutional layer is followed by a 1D max pooling layer with a length and stride of 2 that halves the size of the feature maps from the convolutional layer. The rest of the network is the same as the neural network above.

1 2 3 4 5 6 7 8 9 10 |
# create the model model = Sequential() model.add(Embedding(top_words, 32, input_length=max_words)) model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu')) model.add(MaxPooling1D(pool_size=2)) model.add(Flatten()) model.add(Dense(250, activation='relu')) model.add(Dense(1, activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) print(model.summary()) |

We also fit the network the same as before.

1 2 3 4 5 |
# Fit the model model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=2, batch_size=128, verbose=2) # Final evaluation of the model scores = model.evaluate(X_test, y_test, verbose=0) print("Accuracy: %.2f%%" % (scores[1]*100)) |

Running the example, we are first presented with a summary of the network structure. We can see our convolutional layer preserves the dimensionality of our Embedding input layer of 32-dimensional input with a maximum of 500 words. The pooling layer compresses this representation by halving it.

Running the example offers a small but welcome improvement over the neural network model above with an accuracy of nearly 87.79%.

1 2 3 4 5 6 |
Train on 25000 samples, validate on 25000 samples Epoch 1/2 38s - loss: 0.4451 - acc: 0.7640 - val_loss: 0.3107 - val_acc: 0.8660 Epoch 2/2 39s - loss: 0.2373 - acc: 0.9064 - val_loss: 0.2909 - val_acc: 0.8779 Accuracy: 87.79% |

Again, there is a lot of opportunity for further optimization, such as the use of deeper and/or larger convolutional layers. One interesting idea is to set the max pooling layer to use an input length of 500. This would compress each feature map to a single 32 length vector and may boost performance.

## Summary

In this post, you discovered the IMDB sentiment analysis dataset for natural language processing.

You learned how to develop deep learning models for sentiment analysis including:

- How to load and review the IMDB dataset within Keras.
- How to develop a large neural network model for sentiment analysis.
- How to develop a one-dimensional convolutional neural network model for sentiment analysis.

Do you have any questions about sentiment analysis or this post? Ask your questions in the comments and I will do my best to answer.

## Frustrated With Your Progress In Deep Learning?

#### What If You Could Develop Your Own Deep Nets in Minutes

...with just a few lines of Python

Discover how in my new Ebook: Deep Learning With Python

It covers **self-study tutorials** and **end-to-end projects** on topics like:*Multilayer Perceptrons*, *Convolutional Nets* and *Recurrent Neural Nets*, and more...

#### Finally Bring Deep Learning To

Your Own Projects

Skip the Academics. Just Results.

imdb.load_data(nb_words=5000, test_split=0.33)

TypeError: load_data() got an unexpected keyword argument ‘test_split’

The test_split argument doesn’t appear to exist in Keras 1.08, perhaps I’m doing something wrong?

The API has changed, sorry. I will update the example. You can remove the “test_split” argument.

I have updated the example to match the API changes for Keras 1.1.0 and TensorFlow 0.10.0.

Hi, Jason,

Thanks for the great tutorial! How could I modify this to perform sentiment analysis based on user input? Or from a Twitter stream?

Best wishes

You would have to encode the tweets using the same mapping from characters to integers.

I do not have code for you to do this at the moment.

1. embeddings are trainable, right? I mean, embeddings are dynamic, and they are changing during the training?

2. How to save the embeddings to a file? We have to load the embeddings to predict new data in the future.

3. I have a file named predict.py. I know how to load the model and graph architecture from here: http://machinelearningmastery.com/save-load-keras-deep-learning-models/

But how to load embeddings in predict.py in order to predict new data?

Hi Jie, great questions.

An embedding is a projection, they can be prepared from data. I would not say they are learned, but perhaps you could say that.

I believe they can be prepared deterministically so we would not have to save them to file. I could be wrong about that, but that is my intuition.

Jason, many thanks for your lessons! They’re amazing!!!!

Maybe I ask you a very stupid question, but I can’t understand one thing. What is the Embedding layer? Could you show an example of it. I mean this is word vector with dimentionality of 500X32. So how it looks like?

[0,0,1,….0,0,0] X 32

.

X500

.

[0,1,0,….0,0,0] X 32

What digits in it? And why if we lift it dimensionality up to 64, the accuracy will rise up?

Thank you!

test_split = 0.33

This is defined, but not used anywhere in the code. why is that?

It is a typo and should be removed, thanks Martin. I will update the example soon.

What params would you change typically to achieve what you have mentioned towards the end of the article to improve accuracy?

To quote you: “set the max pooling layer to use an input length of 500. This would compress each feature map to a single 32 length vector”

Could you please help what params (which lines) would need this change?

Hi Anand, use some trial and error and tune the method for the problem.

I am testing the predicted probabilities and values using model.predict(X_test) and model.predict_classes(X_test)

I noticed that the predicted probabilities for the 0 class are all 0.5 w/ median predicted probability of 0.9673.

Am I correct in assuming the model.predict returns probability of the 1 class always, and predicts a class of 0 when that probability is < 50%?

Hi Jim,

I would expect that model.predict() is performing an arg max on the probabilities (selecting the class with the highest probability).

This is knowable if you wade through the code:

https://github.com/fchollet/keras/tree/master/keras

When an element in the data is labeled as an integer, let’s say 4 for example, could that represent any word that has occurred 4 times in the data or is it a representation of a unique word?

Hi brent, each integer represents a unique word.

Hello Jason, recently i became acquainted with the basics in machine learning and deep learning, in part thanks to the information provided in this site, which I found most insightful.

However, lately I came upon a problem of generation simple and short questions automatically from a text. Due to my lack of knowledge and expertise i cant asses if it is possible to solve this problem with Deep Learning or any other method. Currently I have a database of several thousand questions based on around a hundred corpora that could be used as training data. Do you think I could get any successful results, and if so what approach will be the best? (Consider that even if it makes gibberish 50% of the time, it will still save a lot of work)

It does sound like a good deep learning problem Agustin.

I’d recommend reading some papers on the topic to spark some ideas on methods and ways to represent the problem.

Try searching on google scholar and on arvix.

Hi, thank you for the example! Do you know the NumPy and matplotlib versions you were using in this example? I am having trouble with several methods like mean, std and box plot.

Hi Chris,

This might be a Python 2 vs Python 3 issue, I used Python 2.

Actually, I am thinking that it is the call to map(). What version of Python are you using?

Sorry for post-spam. This works in Python 3:

# Summarize review length

print(“Review length: “)

result = list(map(len, X))

print(type(result))

print(result)

Python 3.5.2 :: Anaconda 4.1.1 (x86_64)

Keras (1.2.1)

tensorflow (0.12.1)

numpy (1.11.2)

matplotlib (1.5.3)

Hey,

Can you give me some idea on how to implement other Deep Learning techniqeus such as recursive autoencoders(RAE) , RBM deep learning algorithm for sentiment analysis

Any help would be appreciated :).

Regards

Hi Akshit,

I don’t have an example of RAE or RBM.

This post has an example of sentiment analysis that you can use as a starting point.

Hello, thanks for the example. I really appreciate you if you suggest me why I got this error.

File “C:\Anaconda\lib\site-packages\theano-0.9.0.dev4-py2.7.egg\theano\gof\cmodule.py”, line 2236, in compile_str

raise MissingGXX(“g++ not available! We can’t compile c code.”)

MissingGXX: (‘The following error happened while compiling the node’, Shape_i{1}(embedding_2_W), ‘\n’, “g++ not available! We can’t compile c code.”, ‘[Shape_i{1}(embedding_2_W)]’)

@Zhang, looks like you have a beta version of Theano. If you’re just looking to get started, maybe you want to try a stable channel instead? Looks like your error is because you’re installing from source and your environment isn’t set up quite right.

Thanks for the note Chri.

Hi Zhang,

It looks like g++ is not available. I’m not a windows user, I’m not sure how to interpret this message.

Consider searching or posting on stack overflow.

@Jason, thanks for your reply and thanks again for the post!

I am having trouble improving the results on this model. I have changed the pool_length (500,250,125,375,5,10,15,20), tried adding another dense layer at size 250 and 500, and changed the number of epochs (25,50).

Do you have any recommendations for tweaking the model? I tried the suggestions (deeper, larger, pool_length, and also number of epochs). Do you have any tips or reading suggestions for improving performance in general? This seems to be my last ‘wall’ to really being able to do ML.

Thanks!

Nice experiments Chris.

Perhaps changes to the structure of the problem itself are required.

This post might shake loose more ideas on how to improve performance:

http://machinelearningmastery.com/improve-deep-learning-performance/

Hi jason, I removed the denser layer with 250 neurons and it reduced the number of parameters to be trained drastically with an increased accuracy of about 1% over 5 epochs. Any idea why you added 2 dense layers after flatten layer?

Well done Kiran.

I came up with the configuration after some brief trial and error. It was not optimized.

Does it make sense to specify validation_data as X_test,y_test in the fit function if we evaluate our model in the scores function afterwards? Or could we skip specify validation_data in model.fit(…)?

No, you will get this for free when fitting your model. The validation data should be different from training data and is completely optional.

What is the accuracy of this approach? Which approach is better to get accuracy of at least 0.89?

Accuracy: 88.28%

I tried the sentiment analysis with convolutional NNs and LSTMs and find the CNNs give higher accuracy. Any insight into why?

Some ideas:

Perhaps the CNNs are better at capturing the spatial relationships.

Perhaps the LSTM needs to be larger and trained for longer to achieve the same skill.

Please add these to the imports

from keras.preprocessing import sequence

from keras.layers.embeddings import Embedding

These are listed in the imports for the section titled “One-Dimensional Convolutional Neural Network Model for the IMDB Dataset”