Multi-label classification involves predicting zero or more class labels.
Unlike normal classification tasks where class labels are mutually exclusive, multi-label classification requires specialized machine learning algorithms that support predicting multiple mutually non-exclusive classes or “labels.”
Deep learning neural networks are an example of an algorithm that natively supports multi-label classification problems. Neural network models for multi-label classification tasks can be easily defined and evaluated using the Keras deep learning library.
In this tutorial, you will discover how to develop deep learning models for multi-label classification.
After completing this tutorial, you will know:
Multi-label classification is a predictive modeling task that involves predicting zero or more mutually non-exclusive class labels.
Neural network models can be configured for multi-label classification tasks.
How to evaluate a neural network for multi-label classification and make a prediction for new data.
Let’s get started.
Multi-Label Classification with Deep Learning Photo by Trevor Marron, some rights reserved.
Tutorial Overview
This tutorial is divided into three parts; they are:
Multi-Label Classification
Neural Networks for Multiple Labels
Neural Network for Multi-Label Classification
Multi-Label Classification
Classification is a predictive modeling problem that involves outputting a class label given some input
It is different from regression tasks that involve predicting a numeric value.
Typically, a classification task involves predicting a single label. Alternately, it might involve predicting the likelihood across two or more class labels. In these cases, the classes are mutually exclusive, meaning the classification task assumes that the input belongs to one class only.
Some classification tasks require predicting more than one class label. This means that class labels or class membership are not mutually exclusive. These tasks are referred to as multiple label classification, or multi-label classification for short.
In multi-label classification, zero or more labels are required as output for each input sample, and the outputs are required simultaneously. The assumption is that the output labels are a function of the inputs.
Our dataset will have 1,000 samples with 10 input features. The dataset will have three class label outputs for each sample and each class will have one or two values (0 or 1, e.g. present or not present).
The complete example of creating and summarizing the synthetic multi-label classification dataset is listed below.
1
2
3
4
5
6
7
8
9
# example of a multi-label classification task
from sklearn.datasets import make_multilabel_classification
Running the example creates the dataset and summarizes the shape of the input and output elements.
We can see that, as expected, there are 1,000 samples, each with 10 input features and three output features.
The first 10 rows of inputs and outputs are summarized and we can see that all inputs for this dataset are numeric and that output class labels have 0 or 1 values for each of the three class labels.
1
2
3
4
5
6
7
8
9
10
11
12
(1000, 10) (1000, 3)
[ 3. 3. 6. 7. 8. 2. 11. 11. 1. 3.] [1 1 0]
[7. 6. 4. 4. 6. 8. 3. 4. 6. 4.] [0 0 0]
[ 5. 5. 13. 7. 6. 3. 6. 11. 4. 2.] [1 1 0]
[1. 1. 5. 5. 7. 3. 4. 6. 4. 4.] [1 1 1]
[ 4. 2. 3. 13. 7. 2. 4. 12. 1. 7.] [0 1 0]
[ 4. 3. 3. 2. 5. 2. 3. 7. 2. 10.] [0 0 0]
[ 3. 3. 3. 11. 6. 3. 4. 14. 1. 3.] [0 1 0]
[ 2. 1. 7. 8. 4. 5. 10. 4. 6. 6.] [1 1 1]
[ 5. 1. 9. 5. 3. 4. 11. 8. 1. 8.] [1 1 1]
[ 2. 11. 7. 6. 2. 2. 9. 11. 9. 3.] [1 1 1]
Next, let’s look at how we can develop neural network models for multi-label classification tasks.
Neural Networks for Multiple Labels
Some machine learning algorithms support multi-label classification natively.
Neural network models can be configured to support multi-label classification and can perform well, depending on the specifics of the classification task.
Multi-label classification can be supported directly by neural networks simply by specifying the number of target labels there is in the problem as the number of nodes in the output layer. For example, a task that has three output labels (classes) will require a neural network output layer with three nodes in the output layer.
Each node in the output layer must use the sigmoid activation. This will predict a probability of class membership for the label, a value between 0 and 1. Finally, the model must be fit with the binary cross-entropy loss function.
In summary, to configure a neural network model for multi-label classification, the specifics are:
Number of nodes in the output layer matches the number of labels.
Sigmoid activation for each node in the output layer.
Binary cross-entropy loss function.
We can demonstrate this using the Keras deep learning library.
We will define a Multilayer Perceptron (MLP) model for the multi-label classification task defined in the previous section.
Each sample has 10 inputs and three outputs; therefore, the network requires an input layer that expects 10 inputs specified via the “input_dim” argument in the first hidden layer and three nodes in the output layer.
We will use the popular ReLU activation function in the hidden layer. The hidden layer has 20 nodes that were chosen after some trial and error. We will fit the model using binary cross-entropy loss and the Adam version of stochastic gradient descent.
The definition of the network for the multi-label classification task is listed below.
You may want to adapt this model for your own multi-label classification task; therefore, we can create a function to define and return the model where the number of input and output variables is provided as arguments.
Now that we are familiar with how to define an MLP for multi-label classification, let’s explore how this model can be evaluated.
Neural Network for Multi-Label Classification
If the dataset is small, it is good practice to evaluate neural network models repeatedly on the same dataset and report the mean performance across the repeats.
This is because of the stochastic nature of the learning algorithm.
Additionally, it is good practice to use k-fold cross-validation instead of train/test splits of a dataset to get an unbiased estimate of model performance when making predictions on new data. Again, only if there is not too much data that the process can be completed in a reasonable time.
Taking this into account, we will evaluate the MLP model on the multi-output regression task using repeated k-fold cross-validation with 10 folds and three repeats.
The MLP model will predict the probability for each class label by default. This means it will predict three probabilities for each sample. These can be converted to crisp class labels by rounding the values to either 0 or 1. We can then calculate the classification accuracy for the crisp class labels.
1
2
3
4
5
6
7
...
# make a prediction on the test set
yhat=model.predict(X_test)
# round probabilities to class labels
yhat=yhat.round()
# calculate accuracy
acc=accuracy_score(y_test,yhat)
The scores are collected and can be summarized by reporting the mean and standard deviation across all repeats and cross-validation folds.
The evaluate_model() function below takes the dataset, evaluates the model, and returns a list of evaluation scores, in this case, accuracy scores.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# evaluate a model using repeated k-fold cross-validation
Running the example reports the classification accuracy for each fold and each repeat, to give an idea of the evaluation progress.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
At the end, the mean and standard deviation accuracy is reported. In this case, the model is shown to achieve an accuracy of about 81.2 percent.
You can use this code as a template for evaluating MLP models on your own multi-label classification tasks. The number of nodes and layers in the model can easily be adapted and tailored to the complexity of your dataset.
1
2
3
4
5
6
7
...
>0.780
>0.820
>0.790
>0.810
>0.840
Accuracy: 0.812 (0.032)
Once a model configuration is chosen, we can use it to fit a final model on all available data and make a prediction for new data.
The example below demonstrates this by first fitting the MLP model on the entire multi-label classification dataset, then calling the predict() function on the saved model in order to make a prediction for a new row of data.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# use mlp for prediction on multi-label classification
from numpy import asarray
from sklearn.datasets import make_multilabel_classification
Running the example fits the model and makes a prediction for a new row. As expected, the prediction contains three output variables required for the multi-label classification task: the probabilities of each class label.
1
Predicted: [0.9998627 0.9849341 0.00208042]
Further Reading
This section provides more resources on the topic if you are looking to go deeper.
Use label based macro or micro averaged F-Score and not accuracy or hamming loss to evaluate. It’s a good idea to perform a per-label performance analysis which is often not done in the multi-label literature.
Even f1-score might not show the actual status of the model. So, I think It is better to use Precision and recall also.
Nowadays, I am doing a project on “SafeCity: Stories classification”(a Multi-label problem). and I am using these metrics below to evaluate my model.
1. Accuracy(Exact match): Simply, not a good metric to judge a model But used in a research paper.
2. macro f1-score, and also per label f1-score using Classification report.
3. macro recall, and also per label recall using Classification report.
4. macro precision (you can also use ‘micro’ but there is a problem, you can Google it)
5. Hamming loss
6. Hamming accuracy (not any official metrics, code written by self, no sklearn/tf support)
7. AUC per label
And finally, I am judging my model using all the above seven metrics but I prefer precision and recall per label.
The last thing, I evaluate my model for training data to see whether I am overfitting.
I do hope that this will help you to judge your multi-label classification model.
Thanks!
Had some trouble installing them on Anaconda my solution is the instruction in Autokeras website.To install the package, please use the pip installation as follows:
Hi Jason,
Can Random Forest or XGBoost be used for similar problem of multi-label classification? How viable would that approach be? And are you planning to do any such article in the near future?
Hi Saad, I think if you can transform the problem (using Binary Relevance), you can use classifier chains to perform multi label classification (that can use RF/DT, KNN, naive bayes, (you name it) etc.as base classifier).
and the choice of the classifier depends on how you want to exploit (capture) the correlation among the multiple labels.
This is really an awesome tutorial.
The hidden layer has 20 nodes . Is there any particular logic to choose the number of nodes based on number of input dimensions which is 10 in this case ?
There is some vague understanding.
If you have a large amount of data then you can create a complex model that means many hidden layers with a large no .of neurons, but it is not mandatory. But the model should be complex to learn from a large amount of data, otherwise, you would underfit.
If you have less amount of data then create shallow networks. Otherwise, your model would overfit to train data means will remember all the weights instead of learning something out of your training data.
I hope this will help.
Thanks Jason
Does that mean in this also
array([[0.4774732 , 0.04919493, 0.47333184]], dtype=float32)
Index 0 belong to class 0, Index 1 belong to Class 1, etc
Dear Dr Jason,
Given that there are multi-label outputs consisting of only 0 or 1, are there multilabel categories regression models whose outputs belong to the set of integers?
By set of integers I mean numbers that are like 0,1,2,3,4? NOT 0, 1.39, 2.141, 3.142, 4.23? I mean multi-label integers output categories
Dear Dr Jason,
Thank you for the reply by averting me to the FAQ on adding lawyers and nodes.
I have a further question not addressed by the FAQ.
Is it possible that even if you add layers and nodes, would an evaluiation score or accuracy score get worse.
Put it another way, could an evaluation or accuracy score peak as you add more layers and/or nodes then as you add more layers and/or nodes, the evaluation or accuracy score drops?
Yes, it is common that adding layers and nodes results in worse performance at some point, particularly if you do not also tune the hyperparametres for the learning algorithm.
Hi Dr Jason,
Very well explained, Thank you.
Please , I have multi label classification dataset with large number of labels , labels equal to 1385
When use this model on my dataset
The accuracy of training data equal to 15 and
The accuracy of testing data equal to zero
How can I do to deal with my multi label dataset with this number of labels ?
Thanks alot.
hi Jason,
very well explained!
Just wanted to understand how to achieve following –
my data looks like below
Order No Item ID Item Type Box Size
X A APP C1
B APP C2
C FTW C3
D FTW
Y B HAZ C1
C FTW C2
E APP C3
Basically, I have orders which can contain multiple products. The products in one order can be grouped into one or multiple boxes, based on certain parameters. my algorithm should be able to predict which products can go into what size of box based on historical data. Is there anyway we can achieve this?
Jason,
Take the example of a binary classification problem of whether an image has a human in it or not. Here the outputs are obviously mutually exclusive. Then why do we use sigmoid functions for such a problem?
The problem that I stated here it’s outputs are mutually exclusive i.e in an image we can either have a human or not. Then why in this binary problem I’m supposed to use sigmoid instead of softmax?
What i’m trying to ask is that binary classification problems like whether there is a human in an image or not, this problems output are mutually exclusive. One can either have a human or not have. Why are we using using sigmoid here? Sigmoids don’t sum to one. Hence probability(having a human in the image)+ p(not having human in the image) not equalls to 1. Perhaps use softmax instead for binary classification problems?
Very nice tutorial. Thank you. I ran it and did some mods experimenting, did try weighted F1 and it was better than accuracy on my mods (double the features and classes — I’m looking to go to x10 classes). Any suggestions on where the classes are not all independent?
Hi Jason, in this tutorial is it possible to use time series as a data set to do a classification for future time steps? I mean, is there any other consideration to take into account?
Hi, can you help me with my research problem, I’m doing multi-label classification on research papers (RP) textual content, and RP content is too large and there are many classes in which RP can lie, can you suggest the model for multi-label large scale textual content classification problem.
Hi Jason, Awesome article & am happy to see that you answering everyone’s queries.
I am working on my semester project of ‘Fingerprint prediction’. I have 366 individual classes and ~3000 image dataset. I want to build a model that would take two inputs fingerprints images and predict how “accurately same” they are.
If I teach my model with these 366 categorical multi-class, it will not be able to detect or compare any new fingerprints outside the dataset right?
Sounds like you want a distance measure between two images.
Another approach might be to treat it as binary classification – e.g. how likely are the two images of the same fingerprint.
Perhaps experiment to see what works well?
Perhaps check the literature to see what is a common approach?
Perhaps check similar fields (face recognition) to see if you can use a similar approach?
Thanks for this article Jason.
I am interested in learning about using other multi label classification algorithms like decision tree based models for making predictions on structured tabular data (an example would be tabular healthcare data mostly containing categorical or quantitative values used to predict multiple chronic disease risks). I was searching your blog for similar examples. Can you please recommend any good resources to learn from? I appreciate your help!
I want to get multiple labels for a given review. In that case, my input data is textual data. So in this model, you have used numeric data as input data. So can I use this kind of model to clarify reviews into multiple labels?
Thanks for the great work! This site is the one I mostly rely to understand ML concepts clearly.
I have a question – I have around 2 Million rows with 10K labels in the form of ids i.e. like 1534, 67322,592874 etc. Based on a text variable, I need to predict those numbers. Would you suggest regression(after vectorization) or a classification to be applied on this problem?
I have tried classification with MLP using Keras but got stuck at the point where to_categorical() applied on the highly cardinal label(on the label encoded values) throws –
“MemoryError: Unable to allocate 247. GiB for an array with shape (257483, 257483) and data type int32”
Appreciate any pointers(I am reaching out to you after googling a lot on this problem).
Perhaps you can prototype a few different framing of the problem and discover what works well or best for your project goals or stakeholder requirements.
if the ground-truth labels, for each class label, are not binary (multi-label multi-class classification), the sigmoid would not be the suitable option. Have you any suggestion to deal with this problem.
Thank you for the reply. I did not really understand your response. I changed the gt-labels into categorical so that i can obtain the one hot encode. At this point I will obtain a 3D array of labels (e.g. (5000, 4, 3), where rows are instances, columns are the class labels and the third demension represent the categorical labels (3). The problem is that when I try to train the model there is a mismatch of logits and labels shapes ( (None, 4) vs (None, 4, 3)). Should I train with each class label solely, which will omit the correlation between class labels, or there exists any other solution.
I am working on a very similar problem where I have 20 classes (10 shapes, 10 colors) and each sample has a ground truth and known label for the shape and color it belongs to – so [1,0,1..etc] – the sigmoid and binary_crossentropy approach works really well – I generate a confusion matrix for individual subclasses (cm for shape only and cm for color only) – this gives me a better idea of the per category classification – but one thing I can not find anywhere in the literature, is how to treat this dataset with respect to metrics after training – i.e – FPR, TPR, sensitivity etc..
Is it valid to split the confusion matrices as above and THEN calculate FPR TPR recal etc on a per category basis (for shape only and for color only) – given that the labels are now binary – i don’t see how this approach would be invalid per se – the idea being that there are two main subcategories so i would like to see how it performed at each category – an overall FPR etc approach might mask how well it can recognise color over shape for example.. but i am not aware of any statistical ‘laws’ against this?
Those metrics only make sense if you can separate classes into “positive” and “negative” cases (e.g. cancer and no cancer, etc.). If not, then the metrics are not measningful.
ok thank you – so would something like age and organ qualify here – so if i have 5 organs for which i also know age – can i see how well the split has worked based on organ only and then age only – would this also fit with the analogy you used? – i.e. within organ it would be – positive organ relative to all other organs.. and the same for age…?
Deep Learning is most important word in each work. Anyone can grow up business with deep learning. Anyway thanks for sharing this kind of google article
Thanks, Jason, great tutorial! Now I’m using Keras to implement a multi-label classification model. The label of data has 8-bit, for example, [0,1,0,0,1,0,1,1]. It means totally the label should have 2^8=256 combinations. Now I only collected part of the labels (about 20) in data for the model training. Although the model has a good performance in seen data (95%), it performs badly (30%) for the data with unseen labels (not presented in the training). I wonder how I can improve the model performance? or I have to train this model with as much as data with different labels? I think it will cause a combinatorial explosion for the data collecting workload.
I do not distinguish clearly within the arguments of SKLEARN API “make_multilabel_classification()’ if you set up the n_classes = 3..and I guess it could be 1, 2, or 3 labels presents at he same time ..what is the meaning of n_labels= 2 in the argument?
Because when you explained (as present =1 or not present =0, so total = 2) it is valid every time and always must be n_labels= 2?
# get model
model = get_model(n_inputs, n_outputs)
# fit the model on all data
model.fit(X, y, verbose=0, epochs=100)
# make a prediction for new data
row = [3, 3, 6, 7, 8, 2, 11, 11, 1, 3]
After you do model evaluation, how do you choose the best model for the “model.fit” as shown above? I don’t see how the CV is choosing the right model and use that model for the ‘fit’ as shown above.
Good question. Generally, you evaluate a suite of models and model configurations, choose a model that best addresses your problem, then fit a final model on data to use for making predictions in the future.
Hi Jason, Thank you for the post. I wanted your opinion on one type of problem:
My objective is to classify instances into labels where the labels follow a distribution (distribution sum to 1) instead of binary 0 or 1. I am not sure in this case to use binary cross-entropy loss with sigmoid OR categorical cross-entropy loss with softmax. Also, in this case, will MLP work as a NN architecture? Please guide.
I believe you want to predict class probabilities instead of class labels.
If you have two classes, you would use binary cross entropy loss with sigmoid activation in the output layer, otherwise use categorical cross entropy loss with softmax activation in the output layer.
Thanks Jason for the response. My outputs are percentage distribution across different labels. For every instance in input data, I have a %distribution across labels. And I want to predict that. I have 2 questions:
1. I tried replacing in your code to categorical cross entropy loss with softmax activation in the output layer, but its giving probabilities across different labels which are far from actual percentages. Am I missing anything here?
2. I want to perform entity embeddings for my input categorical features, however, most of the literature talk about embeddings where they use target variable as a single output and not multi-label output. Do you have any reference/implementation for doing entity embeddings for Multi-label Classification?
If your predicted probability are far from actual, you may see if your model is wrong. For example, are you using the right metric to train the model? One issue here is that your design here is quite different from ordinary. MSE may not work, but max error across distribution may be a better metric.
Hi Jason,
Thanks for the post,it was straightforward and easy to understand.
I’m working on a project that deals with multi-label, multi-class (more than 2) outputs. we usually apply smote to balance the output for testing and training data in classification problems.
What can we do if we have imbalanced data in a multi-label classification problem?
Thank you so much for putting together this simple and effective tutorial. I am trying to adapt this idea to per pixel semantic segmentation. I have n samples encoded as RGB image (512,512,3) and the ground truth encoded in two channels (512,512,2) where the first channel provides a more general segmentation (3 classes) and the second one into more specific classes (7 classes). The total number of classes is therefore 10, and some one of the class in the first channel encompass several others in the second one.
I have successfully trained a U-Net using simple (512,512,1) ground truth, but I am having a hard time extending this to multilabel. I believe that creating that two step classification should improve the performance of the model.
I am not sure I follow. But for two channels, first channel has 3 classes, second channel has 7 classes, there should be 3×7=21 classes in total in your model.
If I understand true your note
I have n_samples=11445 , n_features=904 (without preprocessing) , n_classes=4 , ( I don’t have any duplicates or zero rows .
One label column is binary ( True/False) , but three others have 3 state (1 , 2 , 3 => like : “3-other , 2-other , 2-samba, 3-other , 1-sql … “) .
My Question:
Is it good that I use this article’ way for my dataset to do neural network model ?
2- if you say yes , what number should I use for n_label ?
3- Two of my label columns have categorical type, and for scaling my dataset I use z_score and I encoding my these 2 column with ordinal encoding, because their value were too different .
and on-hot encoding isn’t good for them.
Is it correct?
P.S
The article reference doesn’t split dataset to train-test.
Thank you very much if you answer me. Because no body good guide me and I become confused 🤷🏻♀️
1. You need to check but I think it can work.
2. You have 4 columns, one is binary, three are in [1,2,3], so total label would be 2x3x3x3=54
3. It doesn’t seem correct to me. Why one-hot isn’t good for categorical type?
You’re not doing classification among N classes in this case. But you’re doing N binary classifications instead (i.e., whether or not it belongs to a class, for N different classes).
In this case, you need two things: (1) the output of last layer is N sigmoid activation functions so the output of each is from 0 to 1 (like a probability) and (2) you don’t use softmax at the output layer as you don’t want to normalize the value into probability that sum to 1.
Firstly, these tutorials are fantastic and thank you for taking the time and care to write such detailed lessons and code.
Second, this is likely a silly question with an easy answer, but I’m trying to build off the ‘Neural Network for Multi-Label Classification’ code with a CSV as an input. I’ve tried a variety of approaches where I preprocess the CSV outside the code, pop out the class/target var and treat it as y, etc., but keep getting a variety of errors.
Do you have an example code that would include the use of a read CSV instead of the ‘make_multilabel_classification’ seen at the beggining of the code? Appreciate it.
Pathik Prashant GhugareNovember 13, 2021 at 7:55 pm#
I have a doubt,
why are we initializing the model inside the cv.split() loop (in evaluate() function)?
It should be outside the cv.split() loop right ?
Since we are initializing it again and again so there is no point in using the cross-validation method.
No, the code is correct. CV is to evaluate a model configuration. Hence you should initialize it from scratch and train it using one fold. Taking average of K such folds, you get the average score for this particular model configuration.
My project is a power system consisting of four areas in which their condition involves [stable, alert, emergency]. I was wondering that is it right to utilize 4 neurons in output which the value of each of them is (-1 or 0 or 1) and use tanh instead of sigmoid for the last layer? if i am right what loss function do you suggest in this case?
tanh seems better fit, or you may consider this as a multi-class classification problem which each of the stable, alert, emergency state are one-hot encoded.
Hi Mahmoud…Thank you for your question! Often it is advantageous to experiment with various activation functions. The following resource will provide you some guidance on selection of various activation functions.
For multi-class problems, there are some problems for which every example falls in one of the classes and there are other problems where an example falls in either 0 or 1 class. This situation sort of then has a “None of the Above” label, which is in its own way a class.
If we can help it, how many “None of the Above” labels compared with the other examples should we strive for? In NER, for instance, we cannot help it, the ‘O’ label is going to create an unbalanced dataset. But other cases we might.
So it comes down to, how much do we train for the interesting label and how many counterexamples do we provide?
Hi Greg…My recommendation would be to make adjustments based upon the performance on data never seen by the network…that is through validation if you have the data available.
How can we alter this model for multi label image classification.
The input layer will be 224 as the images will of dimension 224×224.
The number of features will not be available, as these are images.
Hi Jason,
How to deal if one or more labels are heavily imbalanced?
We might have high accuracy but F1 score would be low for those.
Use label based macro or micro averaged F-Score and not accuracy or hamming loss to evaluate. It’s a good idea to perform a per-label performance analysis which is often not done in the multi-label literature.
Great suggestion.
This article is nice
Thanks!
Even f1-score might not show the actual status of the model. So, I think It is better to use Precision and recall also.
Nowadays, I am doing a project on “SafeCity: Stories classification”(a Multi-label problem). and I am using these metrics below to evaluate my model.
1. Accuracy(Exact match): Simply, not a good metric to judge a model But used in a research paper.
2. macro f1-score, and also per label f1-score using Classification report.
3. macro recall, and also per label recall using Classification report.
4. macro precision (you can also use ‘micro’ but there is a problem, you can Google it)
5. Hamming loss
6. Hamming accuracy (not any official metrics, code written by self, no sklearn/tf support)
7. AUC per label
And finally, I am judging my model using all the above seven metrics but I prefer precision and recall per label.
The last thing, I evaluate my model for training data to see whether I am overfitting.
I do hope that this will help you to judge your multi-label classification model.
Thanks!
Perhaps you can try a class weighting, e.g. cost-sensitive learning:
https://machinelearningmastery.com/cost-sensitive-neural-network-for-imbalanced-classification/
Hello,
This is really very informative article and thanks for sharing this usefull post.
You’re welcome. I’m happy to hear that!
Sir you are a gem! Great article!
Thanks.
For larger datasets where we would use train/test/validate would we still use average accuracy to evaluate?
No, you would estimate directly from your test set.
Hi Jason,
Very well explained.
Thank you
You’re welcome.
Jason I appreciate your tutorials
Thank you!
Had some trouble installing them on Anaconda my solution is the instruction in Autokeras website.To install the package, please use the pip installation as follows:
pip3 install git+https://github.com/keras-team/[email protected]
pip3 install autokeras
Thanks for this Doc Jason will buy the book over the weekend on a Student Discount
Great tip, thanks for sharing!
Hi Jason,
Can Random Forest or XGBoost be used for similar problem of multi-label classification? How viable would that approach be? And are you planning to do any such article in the near future?
I’m not sure off the cuff, sorry. Perhaps try it and see.
Hi Saad, I think if you can transform the problem (using Binary Relevance), you can use classifier chains to perform multi label classification (that can use RF/DT, KNN, naive bayes, (you name it) etc.as base classifier).
and the choice of the classifier depends on how you want to exploit (capture) the correlation among the multiple labels.
This is really an awesome tutorial.
The hidden layer has 20 nodes . Is there any particular logic to choose the number of nodes based on number of input dimensions which is 10 in this case ?
Thanks!
No, it was configured via trial and error.
There is some vague understanding.
If you have a large amount of data then you can create a complex model that means many hidden layers with a large no .of neurons, but it is not mandatory. But the model should be complex to learn from a large amount of data, otherwise, you would underfit.
If you have less amount of data then create shallow networks. Otherwise, your model would overfit to train data means will remember all the weights instead of learning something out of your training data.
I hope this will help.
Hi Jason,
In MultiLabel if the prediction gives as
array([[0.4774732 , 0.04919493, 0.47333184]], dtype=float32)
and data has 3 classes say ‘0’,’1′,’2′
How we know which class probability belong to which class?
Thank You
Also, How to get model.classes_ for keras functional model, for the multi label we do to_categorical of y values like
y_train = tensorflow.keras.utils.to_categorical(y_train, num_classes= 3)
y_valid = tensorflow.keras.utils.to_categorical(y_valid, num_classes= 3)
whether y_train column names return classes in order of prediction?
You can call model.predict_classes(), more here:
https://machinelearningmastery.com/how-to-make-classification-and-regression-predictions-for-deep-learning-models-in-keras/
Index 0 is always class 0, index 1 is always class 1, etc.
You can use argmax as a short cut:
https://machinelearningmastery.com/argmax-in-machine-learning/
Thanks Jason
Does that mean in this also
array([[0.4774732 , 0.04919493, 0.47333184]], dtype=float32)
Index 0 belong to class 0, Index 1 belong to Class 1, etc
Yes.
Because model.predict_classes(), argmax returns the max probability class
Yes.
predict_classes() does an argmax for you.
Dear Dr Jason,
The predicted yhat[0].round() returns
* Is the above example predicting multi-variate output
Thank you,
Anthony of Sydney
A multi-label output. I guess you could call it multivariate, but I would not use that phrase.
Dear Dr Jason,
Thank you,
Anthony of Sydney
Dear Dr Jason,
Given that there are multi-label outputs consisting of only 0 or 1, are there multilabel categories regression models whose outputs belong to the set of integers?
By set of integers I mean numbers that are like 0,1,2,3,4? NOT 0, 1.39, 2.141, 3.142, 4.23? I mean multi-label integers output categories
Thank you,
Anthony of Sydney
If you require a model to predict integers, you can round/scale the output of a model to meet your needs.
Dear Dr Jason,
Thank you for your reply.
Could you elaborate what kind of multivariate Y models I could generate please.
Thank you,
Anthony of Sydney
An MLP for regression can output real values in 0-1 that can be scaled by the desired range and rounded.
Target values in the training data must be prepared in the same way.
Dear Dr Jason,
In the #define model
* We know that ’20’ means 20 neurons in the first hidden layer.
* Is there a rule of thumb to determine how many neurons are in the hidden layer?
Thank you,
Anthony of Sydney
Good question, no, see this:
https://machinelearningmastery.com/faq/single-faq/how-many-layers-and-nodes-do-i-need-in-my-neural-network
Dear Dr Jason,
Thank you for the reply by averting me to the FAQ on adding lawyers and nodes.
I have a further question not addressed by the FAQ.
Is it possible that even if you add layers and nodes, would an evaluiation score or accuracy score get worse.
Put it another way, could an evaluation or accuracy score peak as you add more layers and/or nodes then as you add more layers and/or nodes, the evaluation or accuracy score drops?
Thank you,
Anthony of Sydney
Yes, it is common that adding layers and nodes results in worse performance at some point, particularly if you do not also tune the hyperparametres for the learning algorithm.
Hi Dr Jason,
Very well explained, Thank you.
Please , I have multi label classification dataset with large number of labels , labels equal to 1385
When use this model on my dataset
The accuracy of training data equal to 15 and
The accuracy of testing data equal to zero
How can I do to deal with my multi label dataset with this number of labels ?
Thanks alot.
Thanks.
Good question, this may give you ideas:
https://machinelearningmastery.com/faq/single-faq/how-do-i-handle-a-large-number-of-categories
hi Jason,
very well explained!
Just wanted to understand how to achieve following –
my data looks like below
Order No Item ID Item Type Box Size
X A APP C1
B APP C2
C FTW C3
D FTW
Y B HAZ C1
C FTW C2
E APP C3
Basically, I have orders which can contain multiple products. The products in one order can be grouped into one or multiple boxes, based on certain parameters. my algorithm should be able to predict which products can go into what size of box based on historical data. Is there anyway we can achieve this?
This might gives you some ideas (replace sites with products):
https://machinelearningmastery.com/faq/single-faq/how-to-develop-forecast-models-for-multiple-sites
Jason,
Take the example of a binary classification problem of whether an image has a human in it or not. Here the outputs are obviously mutually exclusive. Then why do we use sigmoid functions for such a problem?
Isn’t sigmoid and softmax same for binary classification problems?
Sigmoid is for binary classification.
Softmax is for multi-class classification.
Here we are doing “multi-label” classification, which is like multiple binary classification problems. So we use sigmoid.
The problem that I stated here it’s outputs are mutually exclusive i.e in an image we can either have a human or not. Then why in this binary problem I’m supposed to use sigmoid instead of softmax?
Sorry, I don’t understand your question. Could you please rephrase or elaborate?
Classification labels are mutually exclusive.
In multi-label classification, they are not mutually exclusive.
See this:
https://machinelearningmastery.com/types-of-classification-in-machine-learning/
What i’m trying to ask is that binary classification problems like whether there is a human in an image or not, this problems output are mutually exclusive. One can either have a human or not have. Why are we using using sigmoid here? Sigmoids don’t sum to one. Hence probability(having a human in the image)+ p(not having human in the image) not equalls to 1. Perhaps use softmax instead for binary classification problems?
Your problem may be a mutually exclusive classification problem.
We use sigmoid above because it is a different kind of classification problem where labels are NOT mutually exclusive.
Is that clearer?
Then for binary classification problems where labels are mutually exclusive we should use softmax instead of sigmoid?
No, mutually exclusive binary classification has a single output and should use sigmoid.
Softmax is only used for more than two mutually exclusive classes.
thanks Jason
You’re welcome.
Very nice tutorial. Thank you. I ran it and did some mods experimenting, did try weighted F1 and it was better than accuracy on my mods (double the features and classes — I’m looking to go to x10 classes). Any suggestions on where the classes are not all independent?
Nice work!
It might be worth reviewing the literature to see how vast numbers of labels are supported.
If they are not independent, then perhaps a multi-pass/hierarchical approach can be used.
Hi Jason,
In MultiLabel if the prediction gives as
array([[0.4774732 , 0.04919493, 0.47333184]], dtype=float32)
and data has 3 classes say ‘credit’,’debit′,’loan′
How we know which class probability belong to which class?
Thank You
Multi-label classes are not mutually exclusive – meaning we can apply multiple labels to a given example.
The cut-off is 0.5. You can call model.predict_classes() to get labels instead of probabilities or call round().
Hi Jason, in this tutorial is it possible to use time series as a data set to do a classification for future time steps? I mean, is there any other consideration to take into account?
Yes, see the HAR examples of time series classification here:
https://machinelearningmastery.com/start-here/#deep_learning_time_series
Dear Jason,
Thank you for this tutorial. Could you please give an advice, how to deal with partial/missing labels of training samples?
Examples with missing labels/targets cannot be used for training.
Hi, can you help me with my research problem, I’m doing multi-label classification on research papers (RP) textual content, and RP content is too large and there are many classes in which RP can lie, can you suggest the model for multi-label large scale textual content classification problem.
I would expect a deep learning language model would play an important part in this application.
Hello Jason, thanks for the tutorial.
And what about multi-label multi-class classification? That is when every class has more than 2 label options? How to deal with such a task?
Good question, I hope to write about this topic in the future.
Hello Dr,
Nice article with extensive explanation..can you please tell me which keras,tensorflow and python version did you use for this code ?
Thnaks.
It works with the latest version of Keras and TensorFlow, but I expect it works with most if not all versions.
How can i prepare data for multi-label classification?
Good question.
Create a target vector with a length of the number of labels/classes, then mark a 1 if a label is present for the sample, otherwise mark a 0.
An example please!
Thanks for the suggestion.
Hi Jason, Awesome article & am happy to see that you answering everyone’s queries.
I am working on my semester project of ‘Fingerprint prediction’. I have 366 individual classes and ~3000 image dataset. I want to build a model that would take two inputs fingerprints images and predict how “accurately same” they are.
If I teach my model with these 366 categorical multi-class, it will not be able to detect or compare any new fingerprints outside the dataset right?
What do you suggest I can do here.
Sounds like you want a distance measure between two images.
Another approach might be to treat it as binary classification – e.g. how likely are the two images of the same fingerprint.
Perhaps experiment to see what works well?
Perhaps check the literature to see what is a common approach?
Perhaps check similar fields (face recognition) to see if you can use a similar approach?
Thank you so much! Will see!
You’re welcome.
Thanks for this article Jason.
I am interested in learning about using other multi label classification algorithms like decision tree based models for making predictions on structured tabular data (an example would be tabular healthcare data mostly containing categorical or quantitative values used to predict multiple chronic disease risks). I was searching your blog for similar examples. Can you please recommend any good resources to learn from? I appreciate your help!
Sorry, I don’t have a so specific example. Perhaps you can try applying a decision tree directly to your problem?
Hi,
I want to get multiple labels for a given review. In that case, my input data is textual data. So in this model, you have used numeric data as input data. So can I use this kind of model to clarify reviews into multiple labels?
Yes, perhaps adapt it for your own input data.
Hi Jason, I am working on my project of AI dietian where I have to predict a list of foods on the basis of Age, bmi, gender and disease.
I have more than 6 dependant variables, what algorithm should I apply? Multi output classification or multi label classification.
And plz recommend me some sources from where I can get help. Thank you
I recommend testing a suite of different algorithms in order to discover what works well or best for your dataset.
Hi Jason,
Thanks for the great work! This site is the one I mostly rely to understand ML concepts clearly.
I have a question – I have around 2 Million rows with 10K labels in the form of ids i.e. like 1534, 67322,592874 etc. Based on a text variable, I need to predict those numbers. Would you suggest regression(after vectorization) or a classification to be applied on this problem?
I have tried classification with MLP using Keras but got stuck at the point where to_categorical() applied on the highly cardinal label(on the label encoded values) throws –
“MemoryError: Unable to allocate 247. GiB for an array with shape (257483, 257483) and data type int32”
Appreciate any pointers(I am reaching out to you after googling a lot on this problem).
Thanks.
Perhaps you can prototype a few different framing of the problem and discover what works well or best for your project goals or stakeholder requirements.
Regarding large datasets:
https://machinelearningmastery.com/faq/single-faq/how-to-i-work-with-a-very-large-dataset
Hi Jason,
if the ground-truth labels, for each class label, are not binary (multi-label multi-class classification), the sigmoid would not be the suitable option. Have you any suggestion to deal with this problem.
Ouch.
Perhaps one hot encode the multi-class labels, e.g. denormalize all class labels.
Dear Jason
Thank you for the reply. I did not really understand your response. I changed the gt-labels into categorical so that i can obtain the one hot encode. At this point I will obtain a 3D array of labels (e.g. (5000, 4, 3), where rows are instances, columns are the class labels and the third demension represent the categorical labels (3). The problem is that when I try to train the model there is a mismatch of logits and labels shapes ( (None, 4) vs (None, 4, 3)). Should I train with each class label solely, which will omit the correlation between class labels, or there exists any other solution.
Thank you
You may need to experiment, I have not tried this before.
Perhaps you can use a different output model for each class label?
Great read!
I am working on a very similar problem where I have 20 classes (10 shapes, 10 colors) and each sample has a ground truth and known label for the shape and color it belongs to – so [1,0,1..etc] – the sigmoid and binary_crossentropy approach works really well – I generate a confusion matrix for individual subclasses (cm for shape only and cm for color only) – this gives me a better idea of the per category classification – but one thing I can not find anywhere in the literature, is how to treat this dataset with respect to metrics after training – i.e – FPR, TPR, sensitivity etc..
Is it valid to split the confusion matrices as above and THEN calculate FPR TPR recal etc on a per category basis (for shape only and for color only) – given that the labels are now binary – i don’t see how this approach would be invalid per se – the idea being that there are two main subcategories so i would like to see how it performed at each category – an overall FPR etc approach might mask how well it can recognise color over shape for example.. but i am not aware of any statistical ‘laws’ against this?
thanks!
Those metrics only make sense if you can separate classes into “positive” and “negative” cases (e.g. cancer and no cancer, etc.). If not, then the metrics are not measningful.
ok thank you – so would something like age and organ qualify here – so if i have 5 organs for which i also know age – can i see how well the split has worked based on organ only and then age only – would this also fit with the analogy you used? – i.e. within organ it would be – positive organ relative to all other organs.. and the same for age…?
thanks!
Perhaps try it.
Deep Learning is most important word in each work. Anyone can grow up business with deep learning. Anyway thanks for sharing this kind of google article
Thanks.
Thanks, Jason, great tutorial! Now I’m using Keras to implement a multi-label classification model. The label of data has 8-bit, for example, [0,1,0,0,1,0,1,1]. It means totally the label should have 2^8=256 combinations. Now I only collected part of the labels (about 20) in data for the model training. Although the model has a good performance in seen data (95%), it performs badly (30%) for the data with unseen labels (not presented in the training). I wonder how I can improve the model performance? or I have to train this model with as much as data with different labels? I think it will cause a combinatorial explosion for the data collecting workload.
Perhaps some of these suggestions will help:
https://machinelearningmastery.com/improve-deep-learning-performance/
Thanks! I will check it.
Hi Jason:
nice tutorial!
I do not distinguish clearly within the arguments of SKLEARN API “make_multilabel_classification()’ if you set up the n_classes = 3..and I guess it could be 1, 2, or 3 labels presents at he same time ..what is the meaning of n_labels= 2 in the argument?
Because when you explained (as present =1 or not present =0, so total = 2) it is valid every time and always must be n_labels= 2?
thanks
Off the cuff, I think the whole problem of multi-label classification becomes simpler to model if all labels are transformed to binary.
thks
You’re welcome!
Hi Jason.
Why do you use BinaryCrossEntropy for a multi label problem? Shouldn’t we use CategoricalCrossEntropy?
Sorry just misinterpreted the whole scenario… Very cool article Jason
No problem at all, it’s a good question.
Good question, because we have a series of binary predictions to make, not a single multinomial probability distribution.
Thanks for the clarification Jason
You’re welcome.
Hi, Jason:
# get model
model = get_model(n_inputs, n_outputs)
# fit the model on all data
model.fit(X, y, verbose=0, epochs=100)
# make a prediction for new data
row = [3, 3, 6, 7, 8, 2, 11, 11, 1, 3]
After you do model evaluation, how do you choose the best model for the “model.fit” as shown above? I don’t see how the CV is choosing the right model and use that model for the ‘fit’ as shown above.
Good question. Generally, you evaluate a suite of models and model configurations, choose a model that best addresses your problem, then fit a final model on data to use for making predictions in the future.
Perhaps this will help:
https://machinelearningmastery.com/make-predictions-scikit-learn/
Hi Jason, Thank you for the post. I wanted your opinion on one type of problem:
My objective is to classify instances into labels where the labels follow a distribution (distribution sum to 1) instead of binary 0 or 1. I am not sure in this case to use binary cross-entropy loss with sigmoid OR categorical cross-entropy loss with softmax. Also, in this case, will MLP work as a NN architecture? Please guide.
I believe you want to predict class probabilities instead of class labels.
If you have two classes, you would use binary cross entropy loss with sigmoid activation in the output layer, otherwise use categorical cross entropy loss with softmax activation in the output layer.
Thanks Jason for the response. My outputs are percentage distribution across different labels. For every instance in input data, I have a %distribution across labels. And I want to predict that. I have 2 questions:
1. I tried replacing in your code to categorical cross entropy loss with softmax activation in the output layer, but its giving probabilities across different labels which are far from actual percentages. Am I missing anything here?
2. I want to perform entity embeddings for my input categorical features, however, most of the literature talk about embeddings where they use target variable as a single output and not multi-label output. Do you have any reference/implementation for doing entity embeddings for Multi-label Classification?
If your predicted probability are far from actual, you may see if your model is wrong. For example, are you using the right metric to train the model? One issue here is that your design here is quite different from ordinary. MSE may not work, but max error across distribution may be a better metric.
Hope this helps.
Sure, thanks Jason. Will give it a try. About the second question, I had, do you have any inputs there?
What prevent you to use embedding in multi-level output?
Very nice tutorial, well done dear Jason
Thanks.
Thanks for the great explanation,
but there is a code on R for this? I am not familiar with python.
Not exactly but you may find this post useful: https://machinelearningmastery.com/non-linear-classification-in-r/
Hi Jason,
Thanks for the post,it was straightforward and easy to understand.
I’m working on a project that deals with multi-label, multi-class (more than 2) outputs. we usually apply smote to balance the output for testing and training data in classification problems.
What can we do if we have imbalanced data in a multi-label classification problem?
Do you think this post answers your questions: https://machinelearningmastery.com/multi-class-imbalanced-classification/
Good morning Jason,
Thank you so much for putting together this simple and effective tutorial. I am trying to adapt this idea to per pixel semantic segmentation. I have n samples encoded as RGB image (512,512,3) and the ground truth encoded in two channels (512,512,2) where the first channel provides a more general segmentation (3 classes) and the second one into more specific classes (7 classes). The total number of classes is therefore 10, and some one of the class in the first channel encompass several others in the second one.
I have successfully trained a U-Net using simple (512,512,1) ground truth, but I am having a hard time extending this to multilabel. I believe that creating that two step classification should improve the performance of the model.
Any idea/guidance is more than welcome.
Thanks,
Hector
I am not sure I follow. But for two channels, first channel has 3 classes, second channel has 7 classes, there should be 3×7=21 classes in total in your model.
Hello dear Dr Jason. Thanks for this article.
I have some questions.
I have sample=11445 and 4 columns as label.
If I understand true your note
I have n_samples=11445 , n_features=904 (without preprocessing) , n_classes=4 , ( I don’t have any duplicates or zero rows .
One label column is binary ( True/False) , but three others have 3 state (1 , 2 , 3 => like : “3-other , 2-other , 2-samba, 3-other , 1-sql … “) .
My Question:
Is it good that I use this article’ way for my dataset to do neural network model ?
2- if you say yes , what number should I use for n_label ?
3- Two of my label columns have categorical type, and for scaling my dataset I use z_score and I encoding my these 2 column with ordinal encoding, because their value were too different .
and on-hot encoding isn’t good for them.
Is it correct?
P.S
The article reference doesn’t split dataset to train-test.
Thank you very much if you answer me. Because no body good guide me and I become confused 🤷🏻♀️
1. You need to check but I think it can work.
2. You have 4 columns, one is binary, three are in [1,2,3], so total label would be 2x3x3x3=54
3. It doesn’t seem correct to me. Why one-hot isn’t good for categorical type?
Hi,
Thank you for informative article.
Need your suggestion on this if three class available and probability within class should be like
[x,y,z]
[.98 .97,.96]
we don’t need probability with sum of 1,we need individual probability, if particular object belong to three different class.
You’re not doing classification among N classes in this case. But you’re doing N binary classifications instead (i.e., whether or not it belongs to a class, for N different classes).
In this case, you need two things: (1) the output of last layer is N sigmoid activation functions so the output of each is from 0 to 1 (like a probability) and (2) you don’t use softmax at the output layer as you don’t want to normalize the value into probability that sum to 1.
Please suggest how to use Deep Learning techniques to classify QoS Parameters
Did you tried anything? It would be easier to comment if you have a concrete example.
Hello,
Firstly, these tutorials are fantastic and thank you for taking the time and care to write such detailed lessons and code.
Second, this is likely a silly question with an easy answer, but I’m trying to build off the ‘Neural Network for Multi-Label Classification’ code with a CSV as an input. I’ve tried a variety of approaches where I preprocess the CSV outside the code, pop out the class/target var and treat it as y, etc., but keep getting a variety of errors.
Do you have an example code that would include the use of a read CSV instead of the ‘make_multilabel_classification’ seen at the beggining of the code? Appreciate it.
Yes, a lot. For example here: https://machinelearningmastery.com/neural-network-for-cancer-survival-dataset/
Search for the read_csv() function and you should see how to do it.
I have a doubt,
why are we initializing the model inside the cv.split() loop (in evaluate() function)?
It should be outside the cv.split() loop right ?
Since we are initializing it again and again so there is no point in using the cross-validation method.
No, the code is correct. CV is to evaluate a model configuration. Hence you should initialize it from scratch and train it using one fold. Taking average of K such folds, you get the average score for this particular model configuration.
Hi Jason,
Thanks for this nice article.
To explain “round probabilities to class labels”, you could added a reference link to “Converting Probabilities to Class Labels” section of https://machinelearningmastery.com/threshold-moving-for-imbalanced-classification/
Thanks for the suggestion!
Dear Jason,
Thanks for your great article.
My project is a power system consisting of four areas in which their condition involves [stable, alert, emergency]. I was wondering that is it right to utilize 4 neurons in output which the value of each of them is (-1 or 0 or 1) and use tanh instead of sigmoid for the last layer? if i am right what loss function do you suggest in this case?
tanh seems better fit, or you may consider this as a multi-class classification problem which each of the stable, alert, emergency state are one-hot encoded.
Hi Mahmoud…Thank you for your question! Often it is advantageous to experiment with various activation functions. The following resource will provide you some guidance on selection of various activation functions.
https://machinelearningmastery.com/choose-an-activation-function-for-deep-learning/
Regards,
For multi-class problems, there are some problems for which every example falls in one of the classes and there are other problems where an example falls in either 0 or 1 class. This situation sort of then has a “None of the Above” label, which is in its own way a class.
If we can help it, how many “None of the Above” labels compared with the other examples should we strive for? In NER, for instance, we cannot help it, the ‘O’ label is going to create an unbalanced dataset. But other cases we might.
So it comes down to, how much do we train for the interesting label and how many counterexamples do we provide?
Hi Greg…My recommendation would be to make adjustments based upon the performance on data never seen by the network…that is through validation if you have the data available.
Hi Jason,
How can we alter this model for multi label image classification.
The input layer will be 224 as the images will of dimension 224×224.
The number of features will not be available, as these are images.
Can you help Json?
Regards,
Haris.