How to Handle Missing Data with Python

Last Updated on

Real-world data often has missing values.

Data can have missing values for a number of reasons such as observations that were not recorded and data corruption.

Handling missing data is important as many machine learning algorithms do not support data with missing values.

In this tutorial, you will discover how to handle missing data for machine learning with Python.

Specifically, after completing this tutorial you will know:

  • How to marking invalid or corrupt values as missing in your dataset.
  • How to remove rows with missing data from your dataset.
  • How to impute missing values with mean values in your dataset.

Discover how to prepare data with pandas, fit and evaluate models with scikit-learn, and more in my new book, with 16 step-by-step tutorials, 3 projects, and full python code.

Let’s get started.

Note: The examples in this post assume that you have Python 2 or 3 with Pandas, NumPy and Scikit-Learn installed, specifically scikit-learn version 0.18 or higher.

  • Update March/2018: Added alternate link to download the dataset as the original appears to have been taken down.
How to Handle Missing Values with Python

How to Handle Missing Values with Python
Photo by CoCreatr, some rights reserved.


This tutorial is divided into 6 parts:

  1. Pima Indians Diabetes Dataset: where we look at a dataset that has known missing values.
  2. Mark Missing Values: where we learn how to mark missing values in a dataset.
  3. Missing Values Causes Problems: where we see how a machine learning algorithm can fail when it contains missing values.
  4. Remove Rows With Missing Values: where we see how to remove rows that contain missing values.
  5. Impute Missing Values: where we replace missing values with sensible values.
  6. Algorithms that Support Missing Values: where we learn about algorithms that support missing values.

First, let’s take a look at our sample dataset with missing values.

1. Pima Indians Diabetes Dataset

The Pima Indians Diabetes Dataset involves predicting the onset of diabetes within 5 years in Pima Indians given medical details.

It is a binary (2-class) classification problem. The number of observations for each class is not balanced. There are 768 observations with 8 input variables and 1 output variable. The variable names are as follows:

  • 0. Number of times pregnant.
  • 1. Plasma glucose concentration a 2 hours in an oral glucose tolerance test.
  • 2. Diastolic blood pressure (mm Hg).
  • 3. Triceps skinfold thickness (mm).
  • 4. 2-Hour serum insulin (mu U/ml).
  • 5. Body mass index (weight in kg/(height in m)^2).
  • 6. Diabetes pedigree function.
  • 7. Age (years).
  • 8. Class variable (0 or 1).

The baseline performance of predicting the most prevalent class is a classification accuracy of approximately 65%. Top results achieve a classification accuracy of approximately 77%.

A sample of the first 5 rows is listed below.

This dataset is known to have missing values.

Specifically, there are missing observations for some columns that are marked as a zero value.

We can corroborate this by the definition of those columns and the domain knowledge that a zero value is invalid for those measures, e.g. a zero for body mass index or blood pressure is invalid.

Download the dataset from here and save it to your current working directory with the file name pima-indians-diabetes.csv .

2. Mark Missing Values

In this section, we will look at how we can identify and mark values as missing.

We can use plots and summary statistics to help identify missing or corrupt data.

We can load the dataset as a Pandas DataFrame and print summary statistics on each attribute.

Running this example produces the following output:

This is useful.

We can see that there are columns that have a minimum value of zero (0). On some columns, a value of zero does not make sense and indicates an invalid or missing value.

Specifically, the following columns have an invalid zero minimum value:

  • 1: Plasma glucose concentration
  • 2: Diastolic blood pressure
  • 3: Triceps skinfold thickness
  • 4: 2-Hour serum insulin
  • 5: Body mass index

Let’ confirm this my looking at the raw data, the example prints the first 20 rows of data.

Running the example, we can clearly see 0 values in the columns 2, 3, 4, and 5.

We can get a count of the number of missing values on each of these columns. We can do this my marking all of the values in the subset of the DataFrame we are interested in that have zero values as True. We can then count the number of true values in each column.

We can do this my marking all of the values in the subset of the DataFrame we are interested in that have zero values as True. We can then count the number of true values in each column.

Running the example prints the following output:

We can see that columns 1,2 and 5 have just a few zero values, whereas columns 3 and 4 show a lot more, nearly half of the rows.

This highlights that different “missing value” strategies may be needed for different columns, e.g. to ensure that there are still a sufficient number of records left to train a predictive model.

In Python, specifically Pandas, NumPy and Scikit-Learn, we mark missing values as NaN.

Values with a NaN value are ignored from operations like sum, count, etc.

We can mark values as NaN easily with the Pandas DataFrame by using the replace() function on a subset of the columns we are interested in.

After we have marked the missing values, we can use the isnull() function to mark all of the NaN values in the dataset as True and get a count of the missing values for each column.

Running the example prints the number of missing values in each column. We can see that the columns 1:5 have the same number of missing values as zero values identified above. This is a sign that we have marked the identified missing values correctly.

We can see that the columns 1 to 5 have the same number of missing values as zero values identified above. This is a sign that we have marked the identified missing values correctly.

This is a useful summary. I always like to look at the actual data though, to confirm that I have not fooled myself.

Below is the same example, except we print the first 20 rows of data.

Running the example, we can clearly see NaN values in the columns 2, 3, 4 and 5. There are only 5 missing values in column 1, so it is not surprising we did not see an example in the first 20 rows.

It is clear from the raw data that marking the missing values had the intended effect.

Before we look at handling missing values, let’s first demonstrate that having missing values in a dataset can cause problems.

3. Missing Values Causes Problems

Having missing values in a dataset can cause errors with some machine learning algorithms.

In this section, we will try to evaluate a the Linear Discriminant Analysis (LDA) algorithm on the dataset with missing values.

This is an algorithm that does not work when there are missing values in the dataset.

The below example marks the missing values in the dataset, as we did in the previous section, then attempts to evaluate LDA using 3-fold cross validation and print the mean accuracy.

Running the example results in an error, as follows:

This is as we expect.

We are prevented from evaluating an LDA algorithm (and other algorithms) on the dataset with missing values.

Now, we can look at methods to handle the missing values.

4. Remove Rows With Missing Values

The simplest strategy for handling missing data is to remove records that contain a missing value.

We can do this by creating a new Pandas DataFrame with the rows containing missing values removed.

Pandas provides the dropna() function that can be used to drop either columns or rows with missing data. We can use dropna() to remove all rows with missing data, as follows:

Running this example, we can see that the number of rows has been aggressively cut from 768 in the original dataset to 392 with all rows containing a NaN removed.

We now have a dataset that we could use to evaluate an algorithm sensitive to missing values like LDA.

The example runs successfully and prints the accuracy of the model.

Removing rows with missing values can be too limiting on some predictive modeling problems, an alternative is to impute missing values.

5. Impute Missing Values

Imputing refers to using a model to replace missing values.

There are many options we could consider when replacing a missing value, for example:

  • A constant value that has meaning within the domain, such as 0, distinct from all other values.
  • A value from another randomly selected record.
  • A mean, median or mode value for the column.
  • A value estimated by another predictive model.

Any imputing performed on the training dataset will have to be performed on new data in the future when predictions are needed from the finalized model. This needs to be taken into consideration when choosing how to impute the missing values.

For example, if you choose to impute with mean column values, these mean column values will need to be stored to file for later use on new data that has missing values.

Pandas provides the fillna() function for replacing missing values with a specific value.

For example, we can use fillna() to replace missing values with the mean value for each column, as follows:

Running the example provides a count of the number of missing values in each column, showing zero missing values.

The scikit-learn library provides the Imputer() pre-processing class that can be used to replace missing values.

It is a flexible class that allows you to specify the value to replace (it can be something other than NaN) and the technique used to replace it (such as mean, median, or mode). The Imputer class operates directly on the NumPy array instead of the DataFrame.

The example below uses the Imputer class to replace missing values with the mean of each column then prints the number of NaN values in the transformed matrix.

Running the example shows that all NaN values were imputed successfully.

In either case, we can train algorithms sensitive to NaN values in the transformed dataset, such as LDA.

The example below shows the LDA algorithm trained in the Imputer transformed dataset.

Running the example prints the accuracy of LDA on the transformed dataset.

Try replacing the missing values with other values and see if you can lift the performance of the model.

Maybe missing values have meaning in the data.

Next we will look at using algorithms that treat missing values as just another value when modeling.

6. Algorithms that Support Missing Values

Not all algorithms fail when there is missing data.

There are algorithms that can be made robust to missing data, such as k-Nearest Neighbors that can ignore a column from a distance measure when a value is missing.

There are also algorithms that can use the missing value as a unique and different value when building the predictive model, such as classification and regression trees.

Sadly, the scikit-learn implementations of decision trees and k-Nearest Neighbors are not robust to missing values. Although it is being considered.

Nevertheless, this remains as an option if you consider using another algorithm implementation (such as xgboost) or developing your own implementation.

Further Reading


In this tutorial, you discovered how to handle machine learning data that contains missing values.

Specifically, you learned:

  • How to mark missing values in a dataset as numpy.nan.
  • How to remove rows from the dataset that contain missing values.
  • How to replace missing values with sensible values.

Do you have any questions about handling missing values?
Ask your questions in the comments and I will do my best to answer.

Discover Fast Machine Learning in Python!

Master Machine Learning With Python

Develop Your Own Models in Minutes

...with just a few lines of scikit-learn code

Learn how in my new Ebook:
Machine Learning Mastery With Python

Covers self-study tutorials and end-to-end projects like:
Loading data, visualization, modeling, tuning, and much more...

Finally Bring Machine Learning To
Your Own Projects

Skip the Academics. Just Results.

See What's Inside

73 Responses to How to Handle Missing Data with Python

  1. Mike March 20, 2017 at 3:16 pm #

    Fancy impute is a library i’ve turned too for imputation:

    Also missingno is great for visualizations!

    • Jason Brownlee March 21, 2017 at 8:37 am #

      Thanks for the tip Mike.

    • bakyalakshmi September 27, 2017 at 2:56 pm #

      please tell me about how to impute median using one dataset

    • Trung Nguyen Thanh July 8, 2018 at 8:42 pm #

      please tell me, in case use Fancy impute library, how to predict for X_test?

  2. Jozo Kovac April 1, 2017 at 8:06 am #

    Thanks for pointing on interesting problem. I would love another one about how to deal with categorical attributes in Python.

    And dear reader, please never ever remove rows with missing values. It changes the distribution of your data and your analyses may become worthless. Learn from mistakes of others and don’t repeat them 🙂

  3. Tommy Carstensen April 4, 2017 at 3:56 am #

    Super duper! Thanks for writing! Would it have been worth mentioning interpolate of Pandas?

  4. Aswathy April 14, 2017 at 12:10 pm #

    Hi Jason,

    I was just wondering if there is a way to use a different imputation strategy for each column. Say, for a categorical feature you want to impute using the mode but for a continuous attribute, you want to impute using mean.

    • Jason Brownlee April 15, 2017 at 9:30 am #

      Yes, try lots of techniques, go with whatever results in the most accurate models.

  5. Salu Khadka June 11, 2017 at 12:29 am #

    thanks for your tutorial sir.
    I would also seek help from you for multi label classification of a textual data , if possible.

    For example, categorizing a twitter post as related to sports, business , tech , or others.

  6. Ali Gabriel Lara June 13, 2017 at 4:51 am #

    Hello Mr. Brownlee. Thank you so much for your post.

    Do you know any approach to recognize the pattern of missing data? I mean, I am interested in discovering the pattern of missing data on a time series data.

    The database is historical data of a chemical process. I think I should apply some pattern recognition approach columnwise because each column represents a process variable and the value coming from a transmisor.

    My goal is to predict if the missing data is for a mechanical fault or a desviation in registration process or for any other causes. Then I should apply a kind of filling methods if it is required.

    Have you any advice? Thanks in advance

    • Jason Brownlee June 13, 2017 at 8:25 am #

      I would invert the problem and model the series of missing data and mark all data you do have with a special value “0” and all missing instances as “1”.

      Great problem!

      Let me know how you go.

  7. Patricia Villa October 5, 2017 at 3:45 pm #

    You helped me keep my sanity. THANK YOU!!

  8. Sachin Raj October 6, 2017 at 7:58 pm #

    How to know whether to apply mean or to replace it with mode?

    • Jason Brownlee October 7, 2017 at 5:54 am #

      Try both and see what results in the most skillful models.

    • Naga May 24, 2018 at 10:35 pm #

      Hi Sachin,
      Mode is effected by outliers whereas Mean is less effected by outliers.
      Please correct me if i am wrong@Jason

      • Jason Brownlee May 25, 2018 at 9:26 am #

        I think you meant “Median” is not affected by outliers. “Mode” is just the most common value.

  9. Patrick October 26, 2017 at 5:06 am #

    If I have a 11×11 table and there are 20 missing values in there, is there a way for me to make a code that creates a list after identifying these values?

    Let us say that the first column got names and the first row has Day 1 to 10. Some of the names does not show up all of the days and therefore there are missing gaps. I put this table into the code and rather than reading the table I get a list with:

    Name, day 2, day 5, day 7
    Name, Day 1, day 6

    I understand that this could take some time to answer, but if you are able to just tell me that this is possible and maybe know of good place to start on how to start on this project that would be of great help!

    • Jason Brownlee October 26, 2017 at 5:35 am #

      Sure, if the missing values are marked with a nan or similar, you can retrieve rows with missing values using Pandas.

  10. Nivetha December 22, 2017 at 7:46 pm #

    can we code our own algorithms to impute the missing values????
    if it is possible then how can i implement it??

    • Jason Brownlee December 23, 2017 at 5:15 am #


      You can write some if-statements and fill in the n/a values in the Pandas dataframe.

      I would recommend using statistics or a model as well and compare results.

  11. Amit December 29, 2017 at 5:33 pm #

    Hi Jason,

    I am trying to prepare data for the TITANIC dataset. One of the columns is CABIN which has values like ‘A22′,’B56’ and so on. This column has maximum number of missing values. First I thought to delete this column but I think this could be an important variable for predicting survivors.

    I am trying to find a strategy to fill these null values. Is there a way to fill alphanumeric blank values?

    If there is no automatic way, I was thinking of fill these records based on Name, number of sibling, parent child and class columns. E.g. for a missing value, try to see if there are any relatives and use their cabin number to replace missing value.

    Similar case is for AGE column which is missing. Any thoughts?

    • Jason Brownlee December 30, 2017 at 5:19 am #

      Sounds like a categorical variable. You could encode them as integers. You could also assign an “unknown” integer value (e.g. -999) for the missing value.

      Perhaps you can develop a model to predict the cabin number from other details and see if that is skilful.

  12. Chidoooo February 9, 2018 at 10:15 pm #

    Good day, I ran this file code pd.read_csv(r’C:\Users\Public\Documents\SP_dow_Hist_stock.csv’,sep=’,’).pct_change(252)
    and it gave me missing values (NAN) of return of stock. How do I resolve it.

    Unnamed: 0 S&P500 Dow Jones
    0 Date close Close
    1 1-Jan-17 2,275.12 24719.22
    2 1-Jan-16 1,918.60 19762.60
    3 1-Jan-15 2,028.18 17425.03
    4 1-Jan-14 1,822.36 17823.07
    5 1-Jan-13 1,480.40 16576.66
    6 1-Jan-12 1,300.58 13104.14
    7 1-Jan-11 1,282.62 12217.56
    8 1-Jan-10 1,123.58 11577.51
    9 1-Jan-09 865.58 10428.05
    10 1-Jan-08 1,378.76 8776.39
    11 1-Jan-07 1,424.16 13264.82
    12 1-Jan-06 1,278.73 12463.15
    13 1-Jan-05 1,181.41 10717.50
    14 1-Jan-04 1,132.52 10783.01
    15 1-Jan-03 895.84 10453.92
    16 1-Jan-02 1,140.21 8341.63
    17 1-Jan-01 1,335.63 10021.57
    18 1-Jan-00 1,425.59 10787.99
    19 1-Jan-99 1,248.77 11497.12
    20 1-Jan-98 963.36 9181.43
    21 1-Jan-97 766.22 7908.25
    22 1-Jan-96 614.42 6448.27
    23 1-Jan-95 465.25 5117.12
    24 1-Jan-94 472.99 3834.44
    25 1-Jan-93 435.23 3754.09
    26 1-Jan-92 416.08 3301.11
    27 1-Jan-91 325.49 3168.83
    28 1-Jan-90 339.97 2633.66
    29 1-Jan-89 285.4 2753.20
    .. … … …
    68 1-Jan-50 16.88 235.42
    69 1-Jan-49 15.36 200.52
    70 1-Jan-48 14.83 177.30
    71 1-Jan-47 15.21 181.16
    72 1-Jan-46 18.02 177.20
    73 1-Jan-45 13.49 192.91
    74 1-Jan-44 11.85 151.93
    75 1-Jan-43 10.09 135.89
    76 1-Jan-42 8.93 119.40
    77 1-Jan-41 10.55 110.96
    78 1-Jan-40 12.3 131.13
    79 1-Jan-39 12.5 149.99
    80 1-Jan-38 11.31 154.36
    81 1-Jan-37 17.59 120.85
    82 1-Jan-36 13.76 179.90
    83 1-Jan-35 9.26 144.13
    84 1-Jan-34 10.54 104.04
    85 1-Jan-33 7.09 98.67
    86 1-Jan-32 8.3 60.26
    87 1-Jan-31 15.98 77.90
    88 1-Jan-30 21.71 164.58
    89 1-Jan-29 24.86 248.48
    90 1-Jan-28 17.53 300.00
    91 1-Jan-27 13.4 200.70
    92 1-Jan-26 12.65 157.20
    93 1-Jan-25 10.58 156.66
    94 1-Jan-24 8.83 120.51
    95 1-Jan-23 8.9 95.52
    96 1-Jan-22 7.3 98.17
    97 1-Jan-21 7.11 80.80

    [98 rows x 3 columns]

    Unnamed: 0 S&P500 Dow Jones
    0 NaN NaN NaN
    1 NaN NaN NaN
    2 NaN NaN NaN
    3 NaN NaN NaN
    4 NaN NaN NaN
    5 NaN NaN NaN
    6 NaN NaN NaN
    7 NaN NaN NaN
    8 NaN NaN NaN
    9 NaN NaN NaN
    10 NaN NaN NaN
    11 NaN NaN NaN
    12 NaN NaN NaN
    13 NaN NaN NaN
    14 NaN NaN NaN
    15 NaN NaN NaN
    16 NaN NaN NaN
    17 NaN NaN NaN
    18 NaN NaN NaN
    19 NaN NaN NaN
    20 NaN NaN NaN
    21 NaN NaN NaN
    22 NaN NaN NaN
    23 NaN NaN NaN
    24 NaN NaN NaN
    25 NaN NaN NaN
    26 NaN NaN NaN
    27 NaN NaN NaN
    28 NaN NaN NaN
    29 NaN NaN NaN
    .. … … …
    68 NaN NaN NaN
    69 NaN NaN NaN
    70 NaN NaN NaN
    71 NaN NaN NaN
    72 NaN NaN NaN
    73 NaN NaN NaN
    74 NaN NaN NaN
    75 NaN NaN NaN
    76 NaN NaN NaN
    77 NaN NaN NaN
    78 NaN NaN NaN
    79 NaN NaN NaN
    80 NaN NaN NaN
    81 NaN NaN NaN
    82 NaN NaN NaN
    83 NaN NaN NaN
    84 NaN NaN NaN
    85 NaN NaN NaN
    86 NaN NaN NaN
    87 NaN NaN NaN
    88 NaN NaN NaN
    89 NaN NaN NaN
    90 NaN NaN NaN
    91 NaN NaN NaN
    92 NaN NaN NaN
    93 NaN NaN NaN
    94 NaN NaN NaN
    95 NaN NaN NaN
    96 NaN NaN NaN
    97 NaN NaN NaN

    [98 rows x 3 columns]

    • Jason Brownlee February 10, 2018 at 8:57 am #

      Perhaps post your code and issue to stackoverflow?

  13. Ravi March 13, 2018 at 10:59 pm #

    Hi Jason,

    Thanks for your valuable writing.
    I have one question :-
    We can also replace NaN values with Pandas fillna() function. In my opinion this is more versatile than Imputer class because in a single statement we can take different strategies on different column.

    What is your opinion? Is there any performance difference between two?

    • Jason Brownlee March 14, 2018 at 6:23 am #

      Great tip.

      No idea, on the performance difference.

  14. annusin0_0 March 26, 2018 at 4:31 am #

    Is there a recommended ratio on the number of NaN values to valid values , when any corrective action like imputing can be taken?
    If we have a column with most of the values as null, then it would be better off to ignore that column altogether for feature?

    • Jason Brownlee March 26, 2018 at 10:04 am #

      No, it is problem specific. Perhaps run some experiments to see how sensitive the model is to missing values.

  15. Ammar Hasan March 31, 2018 at 2:35 pm #

    Hi Jason,

    Thanks for this post, I wanted to ask, how do we impute missing text values in a column which has either text labels or blanks.

    • Jason Brownlee April 1, 2018 at 5:44 am #

      Good question, I’m not sure off hand. Perhaps start with simple masking of missing values.

    • rakend dubba May 27, 2018 at 6:05 pm #

      To fill the nan for a categorical column

      df = df.fillna(df[‘column’].value_counts().index[0])
      This fills the missing values in all columns with the most frequent categorical value

  16. Gabriel April 6, 2018 at 9:44 pm #

    Thanks a lot Jason ! but I have a little question, how about if we want to replace missing values with the mean of each ROW not column ? how to do that ? if you have any clue, please tell me.. Thank you again Jason.

    • Jason Brownlee April 7, 2018 at 6:32 am #

      Why would you do this?

      numpy.mean() allows you to specify the axis on which to calculate the mean. It will do it for you.

  17. Adil April 13, 2018 at 8:14 am #

    Hi Jason,

    I wanted to ask you how you would deal with missing timestamps (date-time values), which are one set of predictor variables in a classification problem. Would you flag and mark them as missing or impute them as the mode of the rest of the timestamps?

  18. quan April 20, 2018 at 9:19 pm #

    Hi Jason,

    A big fan of yours.

    I have a question about imputing missing numerical values. I don’t really want to remove them and I want to impute them to a value that is like Nan but a numerical type? Would say coding it to -1 work? (0 is already being used).

    I guess I am trying to achieve the same thing as categorising an nan category variable to unknown and creating another feature column to indicate that it is missing.


    • Jason Brownlee April 21, 2018 at 6:49 am #

      NaN is a numerical type. It is a valid float.

      You could use -999 or whatever you like.

      Be careful that your model can support them, or normalize values prior to modeling.

  19. Ravi July 12, 2018 at 1:29 pm #

    Hello Jason,

    You mentioned this here: “if you choose to impute with mean column values, these mean column values will need to be stored to file for later use on new data that has missing values.”, but I wanted to ask:

    Would imputing the data before creating the training and test set (with the data set’s mean) cause data leakage? What would be the best approach to tackle missing data within the data pipeline for a machine learning project.

    Let’s say I’m imputing by filling in with the mean. For the model tuning am I imputing values in the test set with the training set’s mean?

  20. Tobias August 8, 2018 at 7:36 pm #

    Hi Jason,

    Thanks again for that huge nice article!

    is there a neat way to clean away all those rows that happen to be filled with text (i.e. strings) in a certain column, i.e. List.ImportantColumn .

    This destroys my plotting with “could not convert string to float”

    Thanks already in advance!

    • Jason Brownlee August 9, 2018 at 7:38 am #

      Yes, you can remove or replace those values with simple NumPy array indexing.

      For example, if you have ‘?’ you can do:

  21. Anantha Krishnan S September 15, 2018 at 5:45 pm #

    Hi Jason,

    I tried using this dropna to delete the entire row that has missing values in my dataset and after which the isnull().sum() on the dataset also showed zero null values. But the problem arises when i run an algorithm and i am getting an error.

    Error : Input contains NaN, infinity or a value too large for dtype(‘float64’)

    This clearly shows there still exists some null values.

    How do i proceed with this thanks in advance

    • Jason Brownlee September 16, 2018 at 5:57 am #

      Perhaps print the contents of the prepared data to confirm that the nans were indeed removed?

  22. fatma October 27, 2018 at 2:23 am #

    Hi Jason,

    Thanks for this post, I’m using CNN for regression and after data normalization I found some NaN values on training samples. How can I use imputer to fill missing values in the data after normalization.

    • Jason Brownlee October 27, 2018 at 6:03 am #

      Does the above tutorial not help?

      • fatma October 29, 2018 at 4:19 pm #

        should I apply Imputer function for both training and testing dataset?

        • Jason Brownlee October 30, 2018 at 5:52 am #

          Yes, but if the imputer has to learn/estimate, it should be developed from the training data and aplied to the train and test sets, in order to avoid data leakage.

          • fatma October 30, 2018 at 6:24 pm #

            I feel that Imputer remove the Nan values and doesn’t replace them. For example the vector features length in my case is 14 and there are 2 Nan values after applying Imputer function the vector length is 12. This means the 2 Nan values are removed. However I used the following setting:
            imputer = Imputer(missing_values=np.nan, strategy=’mean’, axis=0)

          • Jason Brownlee October 31, 2018 at 6:22 am #

            I don’t know what is happening in your case, perhaps post/search on stackoverflow?

          • fatma November 14, 2018 at 5:22 pm #

            You mean I should fit it on training data then applied to the train and test sets as follow :

            imputer = Imputer(strategy=”mean”, axis=0)
            X_train = imputer.transform(X_train)
            X_test = imputer.transform(X_test)

          • Jason Brownlee November 15, 2018 at 5:27 am #

            Looks good.

  23. Manik Chand October 28, 2018 at 8:58 pm #

    Thanks for this post!!!
    A dataSet having more than 4000 rows and rows can be groupby their 1st columns and let there is many columns (assume 20 columns) and few columns(let 14 columns) contains NaN(missing value).
    How we populate NaN with mean of their corresponding columns by iterative method(using groupby, transform and apply) .

    • Jason Brownlee October 29, 2018 at 5:56 am #

      Sorry, I don’t understand. Perhaps you can elaborate your question?

      • Manik Chand October 30, 2018 at 3:19 am #

        actually i want to fill missing value in each column. Value is the mean of corresponding column. Is there any iterative method?

  24. Sumod December 27, 2018 at 2:47 pm #

    After replacing zeroes,Can I save it as a new data set?

  25. Amit February 7, 2019 at 6:06 pm #

    import numpy as np
    import pandas as pd

    mydata = pd.read_csv(‘diabetes.csv’,header=None)

    0 1 2 3 4 5 6 7 8
    0 Pregnancies Glucose BloodPressure SkinThickness Insulin BMI DiabetesPedigreeFunction Age Outcome
    1 6 148 72 35 0 33.6 0.627 50 1
    2 1 85 66 29 0 26.6 0.351 31 0
    3 8 183 64 0 0 23.3 0.672 32 1
    4 1 89 66 23 94 28.1 0.167 21 0
    5 0 137 40 35 168 43.1 2.288 33 1

    print((mydata[0] == 0).sum()) — for any column it always shows 0
    0 >>>>>>>…. any thing wrong here ?

    whereas i have 0’s in dataset

    0 Pregnancies
    1 6
    2 1
    3 8
    4 1
    5 0>>>>>>>>>
    6 5
    7 3
    8 10
    9 2
    10 8
    11 4
    12 10
    13 10
    14 1
    15 5
    16 7
    17 0 >>>>>>

    • Jason Brownlee February 8, 2019 at 7:45 am #

      Perhaps post your code and issue to stackoverflow?

  26. Krishna March 24, 2019 at 3:59 am #

    Hi sir,
    For my data after executing following instructions still I get same error
    dataset= dataset.replace(0, np.NaN)
    dataset= dataset.replace(0, np.Inf)
    F1 F2 F3 F4
    count 1200.000000 1200.000000 1200.000000 1200.000000
    mean 0.653527 0.649447 1.751579 inf
    std 0.196748 0.194933 0.279228 NaN
    min 0.179076 0.179076 0.731698 0.499815
    25% 0.507860 0.506533 1.573212 1.694007
    50% 0.652066 0.630657 1.763520 1.925291
    75% 0.787908 0.762665 1.934603 2.216663
    max 1.339335 1.371362 2.650390 inf

    How can I get out from this problem.

    • Jason Brownlee March 24, 2019 at 7:07 am #

      Sorry to hear that, perhaps try posting your code and question to stackoverflow?

  27. mubasher March 28, 2019 at 6:38 pm #

    how can we impute the categorical data in python

    • Jason Brownlee March 29, 2019 at 8:29 am #

      You can use an integer encoding (label encoding), a one hot encoding or even a word embedding.

  28. Rotenda Takalani April 16, 2019 at 9:42 pm #

    Hi Jason

    I am new to Python and I was working through the example you gave. For some reason, When I run the piece of code to count the zeros, the code returns results that indicate that there are no zeros in any of those columns.

    Can you please assist?

  29. chu June 4, 2019 at 7:20 pm #

    Hi Jason,

    Great post. Thanks so much.

    Say I have a dataset without headers to identify the columns, how can I handle inconsistent data, for example, age having a value 2500 without knowing this column captures age, any thoughts?

Leave a Reply