This is a good first step, but further pre-processing is required before you can work with the text.
Encoding with one_hot
It is popular to represent a document as a sequence of integer values, where each word in the document is represented as a unique integer.
Keras provides the one_hot() function that you can use to tokenize and integer encode a text document in one step. The name suggests that it will create a one-hot encoding of the document, which is not the case.
Instead, the function is a wrapper for the hashing_trick() function described in the next section. The function returns an integer encoded version of the document. The use of a hash function means that there may be collisions and not all words will be assigned unique integer values.
As with the text_to_word_sequence() function in the previous section, the one_hot() function will make the text lower case, filter out punctuation, and split words based on white space.
In addition to the text, the vocabulary size (total words) must be specified. This could be the total number of words in the document or more if you intend to encode additional documents that contains additional words. The size of the vocabulary defines the hashing space from which words are hashed. Ideally, this should be larger than the vocabulary by some percentage (perhaps 25%) to minimize the number of collisions. By default, the ‘hash’ function is used, although as we will see in the next section, alternate hash functions can be specified when calling the hashing_trick() function directly.
We can use the text_to_word_sequence() function from the previous section to split the document into words and then use a set to represent only the unique words in the document. The size of this set can be used to estimate the size of the vocabulary for one document.
For example:
1
2
3
4
5
6
7
from keras.preprocessing.text import text_to_word_sequence
# define the document
text='The quick brown fox jumped over the lazy dog.'
# estimate the size of the vocabulary
words=set(text_to_word_sequence(text))
vocab_size=len(words)
print(vocab_size)
We can put this together with the one_hot() function and one hot encode the words in the document. The complete example is listed below.
The vocabulary size is increased by one-third to minimize collisions when hashing words.
1
2
3
4
5
6
7
8
9
10
11
from keras.preprocessing.text import one_hot
from keras.preprocessing.text import text_to_word_sequence
# define the document
text='The quick brown fox jumped over the lazy dog.'
# estimate the size of the vocabulary
words=set(text_to_word_sequence(text))
vocab_size=len(words)
print(vocab_size)
# integer encode the document
result=one_hot(text,round(vocab_size*1.3))
print(result)
Running the example first prints the size of the vocabulary as 8. The encoded document is then printed as an array of integer encoded words.
1
2
8
[5, 9, 8, 7, 9, 1, 5, 3, 8]
Hash Encoding with hashing_trick
A limitation of integer and count base encodings is that they must maintain a vocabulary of words and their mapping to integers.
An alternative to this approach is to use a one-way hash function to convert words to integers. This avoids the need to keep track of a vocabulary, which is faster and requires less memory.
Keras provides the hashing_trick() function that tokenizes and then integer encodes the document, just like the one_hot() function. It provides more flexibility, allowing you to specify the hash function as either ‘hash’ (the default) or other hash functions such as the built in md5 function or your own function.
Below is an example of integer encoding a document using the md5 hash function.
1
2
3
4
5
6
7
8
9
10
11
from keras.preprocessing.text import hashing_trick
from keras.preprocessing.text import text_to_word_sequence
# define the document
text='The quick brown fox jumped over the lazy dog.'
Running the example prints the size of the vocabulary and the integer encoded document.
We can see that the use of a different hash function results in consistent, but different integers for words as the one_hot() function in the previous section.
1
2
8
[6, 4, 1, 2, 7, 5, 6, 2, 6]
Tokenizer API
So far we have looked at one-off convenience methods for preparing text with Keras.
Keras provides a more sophisticated API for preparing text that can be fit and reused to prepare multiple text documents. This may be the preferred approach for large projects.
Keras provides the Tokenizer class for preparing text documents for deep learning. The Tokenizer must be constructed and then fit on either raw text documents or integer encoded text documents.
For example:
1
2
3
4
5
6
7
8
9
10
11
from keras.preprocessing.text import Tokenizer
# define 5 documents
docs=['Well done!',
'Good work',
'Great effort',
'nice work',
'Excellent!']
# create the tokenizer
t=Tokenizer()
# fit the tokenizer on the documents
t.fit_on_texts(docs)
Once fit, the Tokenizer provides 4 attributes that you can use to query what has been learned about your documents:
word_counts: A dictionary of words and their counts.
word_docs:Â A dictionary of words and how many documents each appeared in.
word_index: A dictionary of words and their uniquely assigned integers.
document_count:An integer count of the total number of documents that were used to fit the Tokenizer.
For example:
1
2
3
4
5
# summarize what was learned
print(t.word_counts)
print(t.document_count)
print(t.word_index)
print(t.word_docs)
Once the Tokenizer has been fit on training data, it can be used to encode documents in the train or test datasets.
The texts_to_matrix() function on the Tokenizer can be used to create one vector per document provided per input. The length of the vectors is the total size of the vocabulary.
This function provides a suite of standard bag-of-words model text encoding schemes that can be provided via a mode argument to the function.
The modes available include:
‘binary‘: Whether or not each word is present in the document. This is the default.
‘count‘: The count of each word in the document.
‘tfidf‘: The Text Frequency-Inverse DocumentFrequency (TF-IDF) scoring for each word in the document.
‘freq‘: The frequency of each word as a ratio of words within each document.
We can put all of this together with a worked example.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from keras.preprocessing.text import Tokenizer
# define 5 documents
docs=['Well done!',
'Good work',
'Great effort',
'nice work',
'Excellent!']
# create the tokenizer
t=Tokenizer()
# fit the tokenizer on the documents
t.fit_on_texts(docs)
# summarize what was learned
print(t.word_counts)
print(t.document_count)
print(t.word_index)
print(t.word_docs)
# integer encode documents
encoded_docs=t.texts_to_matrix(docs,mode='count')
print(encoded_docs)
Running the example fits the Tokenizer with 5 small documents. The details of the fit Tokenizer are printed. Then the 5 documents are encoded using a word count.
Each document is encoded as a 9-element vector with one position for each word and the chosen encoding scheme value for each word position. In this case, a simple word count mode is used.
Hi Jason,
Kudos for all you efforts in these tutorials. I have a question though, in a documnent classification task, say a csv file where each row contains text from a document. How does one realistically determine the appropriate vocabulary size and max_length for word embedding? Thank you
You have a typo(s) in the above paragraph. You repeat sentences.
“Keras provides the one_hot() function that you can use to tokenize and integer encode a text document in one step. The name suggests that it will create a one-hot encoding of the document, which is not the case. Instead, the function is a wrapper for the hashing_trick() function described in the next section. The function returns an integer encoded version of the document. The use of a hash function means that there may be collisions and not all words will be assigned unique integer values.
Instead, the function is a wrapper for the hashing_trick() function described in the next section. The function returns an integer encoded version of the document. The use of a hash function means that there may be collisions and not all words will be assigned unique integer values.”
I am trying to compare how Hashing is done differently in Sklearn and in Keras. with HashingVectorizer (SciKit Learn) you would be bale to convert a document ( array of texts) into a matrix of n dimension. However when it comes to Keras we are more similar to CountVectorizer (SciKit Learn). Isnt this fundamentally different?
Hashing and counts are different. One uses a hash function to project the vocab into a smaller dimensional space, the other has one entry for each word and a count of the words next to it.
Also, keras and sklearn can do both approaches and more.
Try multiple methods on your dataset and use model skill to help choose the one that works best.
Thank you so much for your time and effort for preparing tutorials.
I just have a question regarding to this tutorial, I want to use http://archive.ics.uci.edu/ml/datasets/Adult as my dataset. As you can see, it has both Strings and Int as input.
My question is how to deal with this type of data and what is the best way to prepare my data for training?
If my test set include words which are not in train set, how can I handle it?
Does Keras embed OoV words to unknown vector implicitly?
If so, which function cover it?
Thank you!
Hi Jason,
Thank you for the nice tutorial. I have a question. why is (vocab_size*1.3) 1.3 and not an integer? you said vocab_size should be 25% larger than the vocabulary. So why is it not 8*125/100 = 10
Thank you for an excellent tutorial which has helped me enormously.
Just to let you know you have a mistake in the text under the Tokenizer Api…
This bit here:
word_counts: A dictionary of words and their counts.
word_docs: An integer count of the total number of documents that were used to fit the Tokenizer.
word_index: A dictionary of words and their uniquely assigned integers.
document_count: A dictionary of words and how many documents each appeared in.
Think you have the explanations for ‘word_docs’ and the ‘document_count’ mixed up…
I have this unusual case where I have short sentences (think about the size of a tweet) so there are, at most, say 24 tokens. These sentences have to be categorized into 64 categories.
With on the order of 10E+06 sentences, I can’t really use a counting technique, so I am thinking about using the hashing trick on each token going into a zero-padded vector that maps to a one-hot encoding of the label:
Thanks for the information. Just a quick question – suppose I am done training this on training data, and I am making predictions using test data.
What if test data has words in text that do not appear in training data? I am dealing with review text data so there are going to be many words that will only be in training data. And same question for making new predictions – what if it encounters words that were not there in original training dataset?
I want to build a text classification model. Where the network will predict Yes or No. based upon the input sentence. I can use the above approaches you mentioned to convert them to the real number but not sure how these will give me fixed-length vectors. For example. Okay and Let’s do it can relate to Yes. But if I use above approach then input vector won’t be fixed size. Can you suggest something?
Hi,
thank you for the informative tutorial!
in the text ‘The quick brown fox jumped over the lazy dog.’, the only repeating word is ‘the’, but in the one_hot and hashing_trick examples, the tokenized output seems to repeat several words:
one hot:
[5, 9, 8, 7, 9, 1, 5, 3, 8]
Here 5 is used twice, corresponding to ‘the’ appearing twice, but 9 appears twice as well (quick, jumped), as does 8 (brown, dog)
hashing trick:
[6, 4, 1, 2, 7, 5, 6, 2, 6]
here 6 seems to represent (the, the, dog), and 2 is (fox, lazy)
Trying out the code for myself, I got similar results until I increased the vocab size to around 3x, which gave unique numbers to each word.
Am I misunderstanding something here, or is there an error in the examples?
Hi Jason, i want to convert text to numbers in the form of their ranking as per most used words, like in the form on IMDb dataset on keras. Is it possible to do it and will doing that give me better results?
man thanks a lot once again was breaking my head as I got tokenizer at character level instead of words. To correct it had to pass it as a list
t = Tokenizer()
t.fit_on_texts([‘Hello world’])
instead of
t = Tokenizer()
t.fit_on_texts(‘Hello world’)
Thank you very much for your informative tutorial!
I am currently doing sentiment analysis of the Twitter project. Inspired by the work emoji2vec[1], I try to add the emoji embedding(which is a 100*1 vector) to the Keras Tokenizer. In this way, I could construct the embedding matrix which contains both word embedding and emoji embedding in sentiment analysis. The constructed embedding matrix could be used as weights in the downstream Embedding layer.
However, the Tokenizer is mostly built by given num_words argument, It is undoubtedly true that the frequency of words is much higher than emoji and if I set num_words=20000, not all the emojis are included. Hence, I think I need to add the emoji manually in the Keras Tokenizer API so as to construct the word-emoji embedding matrix. But is it possible in Keras?
My question is what is the best way to form this text_corpus ? This text_corpus is like some dictionary and which token correspond to which word depends on it. I will get in the future more text for tokenization and I need, for example, that every time word “good” have the same token. So for that, I need always to use the same text_corpus.
Btw. do you have some posts for text preprocessing like removing stop words, lemming etc and is it a good idea to do it before tokenization
The text corpus is the training data that should be representative of the problem.
The Keras API tokenizer is not designed to be updated as far as I know. You may need to use a different API or develop your own tokenizer if you need to update it. Or you can refit the tokenizer and model in the future when new data becomes available.
Hi, I’m working on a text summarizer for an Indian language.
When I use fit_on_texts fn it gives me an attribute error — ‘NoneType’ object has no attribute ‘lower’
Is there any way to do one-hot-encoding without doing tokenization first?
I have a data set which has a column for location. This can contain multi-word string. e.g.-
JERSEY CITY, NEW JERSEY
ST. LOUIS, MISSOURI
MORRISVILLE, NORTH CAROLINA
I want to one-hot-encode these values. How should I do it using Keras?
Hi,
Great tutorial! I’m actually working on some text in a csv file which has some null/empty entries as well. What should I assign such empty entries so it can be processed further? Removal would not be an option. Would assigning them ‘none’ or ’empty’ be a good choice?
I have a dataset , which has 289323 rows
I have a column feature called InstanceDataId which has 25603 unique values,
What is the best way to deal with this one as one of 11 features ?
Thanks Jason for the wonderful tutorials on the Machine Learning theme.
A question on the one-hot encoding. After (one-hot) encoding the text, is it possible via a keras API to get (& print) the mapping between the integer code and the original word. This is to verify if the Thanks.
Thanks Jason. I assume you are referring to Tokenizer.word_index. Wouldn’t that only work if I used Tokenizer.fit_on_texts? How would it work on the text encoded using one_hot function?
You would need to one hot encode yourself or use the tokenizer to do it, or perhaps the sklearn implementation – something you can reuse/save and operate on text consistently in the future.
Hi Jason, thanks for your great work! I’m gonna enter my career in deep learning.
I have questions:
1. If we are working on sentiment analysis model that has lots of review textual sentences data then with method should we prefer ? Is it Tokenizer or hashing_trick ? I think it would be better if we use Tokenizer for features of our model.
2. Which will give better performance ? ‘Tokenizer’ or ‘hashing_trick’ ?
Thanks 🙂
May I Ask you a question concerning: tokenizer.texts_to_sequences method? Does it assign unique integer value to every token in the vocabulary? Do you have any more detailed post on this method? I am trying to create encoded texts inputs for the embedding layer and I wonder is it reasonable to use this method to encode documents.
I already know basics of Deep learning and NLP. But i am looking for any implementation in which we can classify a multi line comments present in document using Deep learning .
I have a question:
I trained a word2vec model on my text data (texts) with min_count = 5, so I only have vectors for words that appear at least 5 times.
Now if I’m going to tokenize my text to feed it into a neural network:
Do I have to fit the tokenizer on the initital text data (text)? And then use this also for the test data (texts_test)? So does it look like the following :
1. Or do I have to fit a new tokenizer object for my test data?
2. Or do I have to fit the tokenizer object on the dictionary from my word2vec Model like:
I read a lot but I’m getting more and more confused about this topic.
And then another and to me most important question: When I get to the point, that my Network is trained: How do I tokenize the so to say future data (texts_future) on which I want to run my predictions? Also with the same tokenizer object that I fitted on the training data in the “first” step? So:
Yes, the first approach would be appropriate. Fit it on train, apply to test. That is how you would use the model in practice.
Once you choose a config, fit the tokenizer and model on all data, save model and tokenizer (and other data prep procedures) and use it to prepare new data in the future. New data must be prepared identically to training data – because that is what the model will expect.
Thanks a lot for the fast reply.
Is there a reason why I should not fit it on the dictionary that I get from the Word2Vec?
I thought that it would make sense as I only tokenize then the words that appear at least X times and the others get marked with an additional token.
In other words, could you maybe explain why I have to fit it on the inital text data and not on the let’s say cleaned initial text (initial_text – words_that_appear_only_X-1_times).
I hope the question is comprehensible.
Surely you share good concept how to apply deep learning in NLP is hot topics!thanks
but how can prepare large documents for text classification my own corpus?
Hi Jason,
When i use set of sentences of different sizes and apply text_to_word_sequence for each sentence, I end up with different vocab_lengths. Should i take max len or average len of these vocab_lengths and apply 130% and use it for OneHotEncoding ?
I have a deep learning project to understand some behaviors of a deternined comentaries database, so I need to transform each word of my database in one specific number, but when the word appears more than one time, it should receive the same number, for example:
I love the musics that you placed in the game
1, 2, 3, 4, 5, 6, 7 , 8 , 3 ,9
Can you please advise me if keras can do this? For the “one hot” sample that you placed, I didn´t understand if it fits to this use case I discribed
Hi Jason,
Do you have any plans to cover word embeddings usine either word2vec or GloVe and how they work with keras?
Yes! I have many posts on word embeddings scheduled for the coming days/weeks.
Hello Jason,
It seems the attributes mentioned for Tokenizer have been typed incorrectly, document_count and word_docs have been inter-changed.
Thanks,
Lalit
Thanks Lalit, in which part of the tutorial exactly?
Once fit, the Tokenizer provides 4 attributes that you can use to query what has been learned about your documents:
How do we further apply a neural network on this data?
Great question, I will have many tutorials about how to do that coming out on the blog in coming weeks.
Hi Jason,
Kudos for all you efforts in these tutorials. I have a question though, in a documnent classification task, say a csv file where each row contains text from a document. How does one realistically determine the appropriate vocabulary size and max_length for word embedding? Thank you
Great question!
Generally, I would recommend testing different sizes/lengths and evaluate their impact on model skill – e.g. a sensitivity analysis.
Hi Jason
Is there anyway to perform a sensitivity analysis using python/keras please? I am looking for the codes for a while know.
Many thanks for you help.
Kim
Sure, pick a variable, then run your analysis.
What is the problem that you’re having exactly?
Thanks for putting this together
You’re welcome.
Hi Jason,
You have a typo(s) in the above paragraph. You repeat sentences.
“Keras provides the one_hot() function that you can use to tokenize and integer encode a text document in one step. The name suggests that it will create a one-hot encoding of the document, which is not the case. Instead, the function is a wrapper for the hashing_trick() function described in the next section. The function returns an integer encoded version of the document. The use of a hash function means that there may be collisions and not all words will be assigned unique integer values.
Instead, the function is a wrapper for the hashing_trick() function described in the next section. The function returns an integer encoded version of the document. The use of a hash function means that there may be collisions and not all words will be assigned unique integer values.”
Fixed, thanks David. It’s my crappy spell check plugin duplicating paragraphs!
Thanks for the great blog Jason,
I am trying to compare how Hashing is done differently in Sklearn and in Keras. with HashingVectorizer (SciKit Learn) you would be bale to convert a document ( array of texts) into a matrix of n dimension. However when it comes to Keras we are more similar to CountVectorizer (SciKit Learn). Isnt this fundamentally different?
Hashing and counts are different. One uses a hash function to project the vocab into a smaller dimensional space, the other has one entry for each word and a count of the words next to it.
Also, keras and sklearn can do both approaches and more.
Try multiple methods on your dataset and use model skill to help choose the one that works best.
Thank you!
You’re welcome.
Could please do text summarization on small datasets using auto encoder and decoder.please provide the code sir
Thanks for the suggestion.
Hi Jason,
Thank you so much for your time and effort for preparing tutorials.
I just have a question regarding to this tutorial, I want to use http://archive.ics.uci.edu/ml/datasets/Adult as my dataset. As you can see, it has both Strings and Int as input.
My question is how to deal with this type of data and what is the best way to prepare my data for training?
Thanks
The strings look like labels or categories.
Perhaps you can encode the string values as integers and/or one hot encode.
so impressive post:)
I have a question.
If my test set include words which are not in train set, how can I handle it?
Does Keras embed OoV words to unknown vector implicitly?
If so, which function cover it?
Thank you!
If you keep the tokenizer, they will be assigned a 0 value I would expect.
Hi Jason,
Thank you for the nice tutorial. I have a question. why is (vocab_size*1.3) 1.3 and not an integer? you said vocab_size should be 25% larger than the vocabulary. So why is it not 8*125/100 = 10
I specified 130% of the size of the vocab. It could have just as easily been 125.
Sir, how can i input a txt file that consists of many paragraphs.
What is the problem you are having exactly?
Sir, i’m inputting
text = open(‘summary.text’ , ‘r’)
words = list(text.read().split)
and while converting text_to_word_sequence i’m getting error
AttributeError: ‘list’ object has no attribute ‘lower’
Sorry to hear that. Did you copy all of the code? Confirm Python3 and all libs are up to date?
Thank you for an excellent tutorial which has helped me enormously.
Just to let you know you have a mistake in the text under the Tokenizer Api…
This bit here:
word_counts: A dictionary of words and their counts.
word_docs: An integer count of the total number of documents that were used to fit the Tokenizer.
word_index: A dictionary of words and their uniquely assigned integers.
document_count: A dictionary of words and how many documents each appeared in.
Think you have the explanations for ‘word_docs’ and the ‘document_count’ mixed up…
Thanks, fixed.
Jason,
Fantastic website. Lots of great information.
I have this unusual case where I have short sentences (think about the size of a tweet) so there are, at most, say 24 tokens. These sentences have to be categorized into 64 categories.
With on the order of 10E+06 sentences, I can’t really use a counting technique, so I am thinking about using the hashing trick on each token going into a zero-padded vector that maps to a one-hot encoding of the label:
[123. 456. 789. 0. 0. 0. 0.] => [0. 0. 0. 1. 0. 0.]
Is this at all a sound approach?
Sounds like a good start!
Let me know how you go.
Thanks for the information. Just a quick question – suppose I am done training this on training data, and I am making predictions using test data.
What if test data has words in text that do not appear in training data? I am dealing with review text data so there are going to be many words that will only be in training data. And same question for making new predictions – what if it encounters words that were not there in original training dataset?
Ideally you want to choose training data that covers your vocab.
If test data contains new words, they will be zero’ed or ignored.
I want to build a text classification model. Where the network will predict Yes or No. based upon the input sentence. I can use the above approaches you mentioned to convert them to the real number but not sure how these will give me fixed-length vectors. For example. Okay and Let’s do it can relate to Yes. But if I use above approach then input vector won’t be fixed size. Can you suggest something?
You can truncate or pad the input sequences of words.
Wonderful site to know NLP coding using python
Thanks.
Hi,
thank you for the informative tutorial!
in the text ‘The quick brown fox jumped over the lazy dog.’, the only repeating word is ‘the’, but in the one_hot and hashing_trick examples, the tokenized output seems to repeat several words:
one hot:
[5, 9, 8, 7, 9, 1, 5, 3, 8]
Here 5 is used twice, corresponding to ‘the’ appearing twice, but 9 appears twice as well (quick, jumped), as does 8 (brown, dog)
hashing trick:
[6, 4, 1, 2, 7, 5, 6, 2, 6]
here 6 seems to represent (the, the, dog), and 2 is (fox, lazy)
Trying out the code for myself, I got similar results until I increased the vocab size to around 3x, which gave unique numbers to each word.
Am I misunderstanding something here, or is there an error in the examples?
Yes, it is hashing rather than encoding.
You could try a true one hot encoding instead if you like.
Even I am getting the same results. Different words are coded with the same integer in the encoding.
Yes, it uses a hash encoding.
Dr. Jason Brownlee, Thank you for works.
Could you give an idea that in which order indexing takes place? In the above example, print(t.word_index))
{‘work’: 1, ‘effort’: 6, ‘done’: 3, ‘great’: 5, ‘good’: 4, ‘excellent’: 8, ‘well’: 2, ‘nice’: 7}
Order within the text I believe.
Keras will use a hash function though, so it is nonlinear. I recommend using the Tokenizer.
Hi Jason, i want to convert text to numbers in the form of their ranking as per most used words, like in the form on IMDb dataset on keras. Is it possible to do it and will doing that give me better results?
It may help in choosing the low frequency words to remove.
Hi Jason
I’m interested in Arabic text summarization,
can I use keras to prepare Arabic text?
I don’t see why not.
Hi, Your blog is amazing.Thank you so much for doing it
Thanks!
Hi! Great blog. I have a question. While using fit_on_text(docs), does docs have to be the training data or both train + test data?
Typically just the training data is used.
Hi sir,
Could you please help me to convert a English word to a tamil?
Sorry, I don’t have the capacity for new projects.
man thanks a lot once again was breaking my head as I got tokenizer at character level instead of words. To correct it had to pass it as a list
t = Tokenizer()
t.fit_on_texts([‘Hello world’])
instead of
t = Tokenizer()
t.fit_on_texts(‘Hello world’)
Perhaps the API has changed?
Hi Jason, in the texts_to_matrix() example, there are 8 words in the vocabulary but the dimension of the generated vectors is 9. Why?
Words start at 1 and an extra space is added for 0 or “unknown”.
How to view the vocabulary created by keras texts_to_matrix function?
encoded_docs = t.texts_to_matrix(docs, mode=’tfidf’)
The tokenizer has a dictionary that can be viewed.
See the section titled “Tokenizer API” for details.
Thank you very much for your informative tutorial!
I am currently doing sentiment analysis of the Twitter project. Inspired by the work emoji2vec[1], I try to add the emoji embedding(which is a 100*1 vector) to the Keras Tokenizer. In this way, I could construct the embedding matrix which contains both word embedding and emoji embedding in sentiment analysis. The constructed embedding matrix could be used as weights in the downstream Embedding layer.
However, the Tokenizer is mostly built by given num_words argument, It is undoubtedly true that the frequency of words is much higher than emoji and if I set num_words=20000, not all the emojis are included. Hence, I think I need to add the emoji manually in the Keras Tokenizer API so as to construct the word-emoji embedding matrix. But is it possible in Keras?
[1] https://arxiv.org/abs/1609.08359
Good question.
Perhaps you can create your own approach to carefully map words/emoji to integers in your word and emoji embeddings.
Hello Jason, thank you for the nice post.
I want to tokenize some text into a sequence of tokens and I’m using
tokenizer.fit_on_texts(text_corpus)
sequences = tokenizer.texts_to_sequences(text)
My question is what is the best way to form this text_corpus ? This text_corpus is like some dictionary and which token correspond to which word depends on it. I will get in the future more text for tokenization and I need, for example, that every time word “good” have the same token. So for that, I need always to use the same text_corpus.
Btw. do you have some posts for text preprocessing like removing stop words, lemming etc and is it a good idea to do it before tokenization
The text corpus is the training data that should be representative of the problem.
The Keras API tokenizer is not designed to be updated as far as I know. You may need to use a different API or develop your own tokenizer if you need to update it. Or you can refit the tokenizer and model in the future when new data becomes available.
Hi, I’m working on a text summarizer for an Indian language.
When I use fit_on_texts fn it gives me an attribute error — ‘NoneType’ object has no attribute ‘lower’
tokenizer_outputs = Tokenizer(num_words=MAX_NUM_WORDS, filters=”)
tokenizer_outputs.fit_on_texts(target_texts + target_texts_inputs)
Does it work only for English language?
Perhaps double check your version of Python is 3.5+?
I’m using Python 3.6.5
Hi Jason,
Is there any way to do one-hot-encoding without doing tokenization first?
I have a data set which has a column for location. This can contain multi-word string. e.g.-
JERSEY CITY, NEW JERSEY
ST. LOUIS, MISSOURI
MORRISVILLE, NORTH CAROLINA
I want to one-hot-encode these values. How should I do it using Keras?
Words must be converted to integers first, then vectors.
You could hash or integer encode for the first step, then one hot encode, bag of words, or use an embedding for the second step.
Hi Jason, Thank you for great post!
It really helps me to understand preprocessing step for text data.
But I can not understand when ‘hashing trick’ is needed.
I think in most of NLP case, such as text classification, I should choose ‘Encoding’ to avoid collision.
Because if positive words and negative words are mapped to same number, there will be some scope to misclassify.
Why and when is ‘hashing trick’ or ‘one-hot’ needed?
Yes, I think so too, but sometimes you don’t have the space/RAM to handle the whole vocab and you may need the hash trick to work around it.
Additionally, the hash trick lets you seamlessly handle new words in the future.
Hi,
Great tutorial! I’m actually working on some text in a csv file which has some null/empty entries as well. What should I assign such empty entries so it can be processed further? Removal would not be an option. Would assigning them ‘none’ or ’empty’ be a good choice?
You can assign them a special word, like [MISSING], or assign them a value of 0 when mapping them to integers.
I have a dataset , which has 289323 rows
I have a column feature called InstanceDataId which has 25603 unique values,
What is the best way to deal with this one as one of 11 features ?
Perhaps compare removing it, integer encode, one hot encode and embedding and use the approach that gives the best model skill.
Thanks Jason for the wonderful tutorials on the Machine Learning theme.
A question on the one-hot encoding. After (one-hot) encoding the text, is it possible via a keras API to get (& print) the mapping between the integer code and the original word. This is to verify if the Thanks.
Yes, it is available in the Tokenizer.
Thanks Jason. I assume you are referring to Tokenizer.word_index. Wouldn’t that only work if I used Tokenizer.fit_on_texts? How would it work on the text encoded using one_hot function?
You would need to one hot encode yourself or use the tokenizer to do it, or perhaps the sklearn implementation – something you can reuse/save and operate on text consistently in the future.
Hi Jason, thanks for your great work! I’m gonna enter my career in deep learning.
I have questions:
1. If we are working on sentiment analysis model that has lots of review textual sentences data then with method should we prefer ? Is it Tokenizer or hashing_trick ? I think it would be better if we use Tokenizer for features of our model.
2. Which will give better performance ? ‘Tokenizer’ or ‘hashing_trick’ ?
Thanks 🙂
A word embedding would be best for text classification:
https://machinelearningmastery.mystagingwebsite.com/what-are-word-embeddings/
If you want to other methods, my best advice in general is to test and let the results guide you.
Hello Jason,
Thank you for your excellent tutorials!
May I Ask you a question concerning: tokenizer.texts_to_sequences method? Does it assign unique integer value to every token in the vocabulary? Do you have any more detailed post on this method? I am trying to create encoded texts inputs for the embedding layer and I wonder is it reasonable to use this method to encode documents.
Yes, fit the Tokenizer on your data, then calls to functions like texts_to_matrix() will transform text to the integer mapping.
You can learn more about the API here:
https://keras.io/preprocessing/text/
The source code is also helpful:
https://github.com/keras-team/keras-preprocessing/blob/master/keras_preprocessing/text.py#L139
HI Jason,
I am trying to classify a special instruction text clause from multiple page document. Please guide me how should i go about it .
Perhaps some of the tutorials here will help you to get started:
https://machinelearningmastery.mystagingwebsite.com/start-here/#nlp
HI Jason,
I already know basics of Deep learning and NLP. But i am looking for any implementation in which we can classify a multi line comments present in document using Deep learning .
You can load multiple line comments as a document and classify them as needed.
Hello Jaso, great topic and also a nice tutorial:
I have a question:
I trained a word2vec model on my text data (texts) with min_count = 5, so I only have vectors for words that appear at least 5 times.
Now if I’m going to tokenize my text to feed it into a neural network:
Do I have to fit the tokenizer on the initital text data (text)? And then use this also for the test data (texts_test)? So does it look like the following :
tokenizer_obj = Tokenizer()
tokenizer_obj.fit_on_texts(texts)
seq = tokenizer_obj.texts_to_sequences(texts)
seq_test = tokenizer_obj.texts_to_sequences(texts_test)
1. Or do I have to fit a new tokenizer object for my test data?
2. Or do I have to fit the tokenizer object on the dictionary from my word2vec Model like:
dictionary = list(model.wv.vocab.keys())
dictionary
tokenizer_obj_dict = Tokenizer()
tokenizer_obj_dict.fit_on_texts(dictionary)
seq = tokenizer_obj_dict.texts_to_sequences(texts)
I read a lot but I’m getting more and more confused about this topic.
And then another and to me most important question: When I get to the point, that my Network is trained: How do I tokenize the so to say future data (texts_future) on which I want to run my predictions? Also with the same tokenizer object that I fitted on the training data in the “first” step? So:
seq_future_text = tokenizer_obj.texts_to_sequences(texts_future)
I hope that you can help me with this topic. That would be awesome!
Thanks a lot Jason.
Yes, the first approach would be appropriate. Fit it on train, apply to test. That is how you would use the model in practice.
Once you choose a config, fit the tokenizer and model on all data, save model and tokenizer (and other data prep procedures) and use it to prepare new data in the future. New data must be prepared identically to training data – because that is what the model will expect.
Thanks a lot for the fast reply.
Is there a reason why I should not fit it on the dictionary that I get from the Word2Vec?
I thought that it would make sense as I only tokenize then the words that appear at least X times and the others get marked with an additional token.
In other words, could you maybe explain why I have to fit it on the inital text data and not on the let’s say cleaned initial text (initial_text – words_that_appear_only_X-1_times).
I hope the question is comprehensible.
You can choose the vocab any way you wish.
Choosing the vocab based on removing infrequent works is an excellent idea.
Does that help?
Surely you share good concept how to apply deep learning in NLP is hot topics!thanks
but how can prepare large documents for text classification my own corpus?
Thanks.
What problem are you having with large documents as opposed to small documents?
What is the role of .texts_to_sequences?
Convert text to sequences of numbers. E.g. each word is assigned a unique number.
Hi Jason,
When i use set of sentences of different sizes and apply text_to_word_sequence for each sentence, I end up with different vocab_lengths. Should i take max len or average len of these vocab_lengths and apply 130% and use it for OneHotEncoding ?
The vocab should based on the entire training dataset.
Say, X_train is the entire training set with say 1000 sentences/doc
X_unique_wordseq = [ set ( text_to_word_sequence ( x ) ) for x in X_train]
word_seq_len = [ len(x) for x in X_wordseq ]
vocab_size = ? #How do we get the vocab size ?
[ vocab_size = np.max(X_train_words_len) ] #Like this ??
X_train_oneH = [one_hot (x, vocab_size) for x in X_train]
The vocab size is the total number of unique words in the training dataset.
The tutorials here will help to get you started:
https://machinelearningmastery.mystagingwebsite.com/start-here/#nlp
Dear Jason,
Thanks for the post, is Tokenizer API provide a bag-of-word model or a word embedding?
No, not quite. It does basic cleaning and assigns numbers to words.
I see, thanks. What about TF-IDF? is it a Bag-of-Word model?
tf-idf is a more advanced representation than bag of words for text.
hello jaso,
is there any solution to get encoded string after using one_hot?????
Yes, you can encode a single strng by calling:
Thank you, Jason I was looking for that, great explanation
You’re welcome.
Hi Jason, thank you very much for the tutorial!
I have a deep learning project to understand some behaviors of a deternined comentaries database, so I need to transform each word of my database in one specific number, but when the word appears more than one time, it should receive the same number, for example:
I love the musics that you placed in the game
1, 2, 3, 4, 5, 6, 7 , 8 , 3 ,9
Can you please advise me if keras can do this? For the “one hot” sample that you placed, I didn´t understand if it fits to this use case I discribed
Hi Carol…Please clarify what you are trying accomplish so that we may better assist you.