How To Load Machine Learning Data in Python

Last Updated on August 21, 2019

You must be able to load your data before you can start your machine learning project.

The most common format for machine learning data is CSV files. There are a number of ways to load a CSV file in Python.

In this post you will discover the different ways that you can use to load your machine learning data in Python.

Kick-start your project with my new book Machine Learning Mastery With Python, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

  • Update March/2017: Change loading from binary (‘rb’) to ASCII (‘rt).
  • Update March/2018: Added alternate link to download the dataset as the original appears to have been taken down.
  • Update March/2018: Updated NumPy load from URL example to work wth Python 3.
How To Load Machine Learning Data in Python

How To Load Machine Learning Data in Python
Photo by Ann Larie Valentine, some rights reserved.

Considerations When Loading CSV Data

There are a number of considerations when loading your machine learning data from CSV files.

For reference, you can learn a lot about the expectations for CSV files by reviewing the CSV request for comment titled Common Format and MIME Type for Comma-Separated Values (CSV) Files.

CSV File Header

Does your data have a file header?

If so this can help in automatically assigning names to each column of data. If not, you may need to name your attributes manually.

Either way, you should explicitly specify whether or not your CSV file had a file header when loading your data.


Does your data have comments?

Comments in a CSV file are indicated by a hash (“#”) at the start of a line.

If you have comments in your file, depending on the method used to load your data, you may need to indicate whether or not to expect comments and the character to expect to signify a comment line.


The standard delimiter that separates values in fields is the comma (“,”) character.

Your file could use a different delimiter like tab (“\t”) in which case you must specify it explicitly.


Sometimes field values can have spaces. In these CSV files the values are often quoted.

The default quote character is the double quotation marks “\””. Other characters can be used, and you must specify the quote character used in your file.

Need help with Machine Learning in Python?

Take my free 2-week email course and discover data prep, algorithms and more (with code).

Click to sign-up now and also get a free PDF Ebook version of the course.

Machine Learning Data Loading Recipes

Each recipe is standalone.

This means that you can copy and paste it into your project and use it immediately.

If you have any questions about these recipes or suggested improvements, please leave a comment and I will do my best to answer.

Load CSV with Python Standard Library

The Python API provides the module CSV and the function reader() that can be used to load CSV files.

Once loaded, you convert the CSV data to a NumPy array and use it for machine learning.

For example, you can download the Pima Indians dataset into your local directory (download from here).

All fields are numeric and there is no header line. Running the recipe below will load the CSV file and convert it to a NumPy array.

The example loads an object that can iterate over each row of the data and can easily be converted into a NumPy array. Running the example prints the shape of the array.

For more information on the csv.reader() function, see CSV File Reading and Writing in the Python API documentation.

Load CSV File With NumPy

You can load your CSV data using NumPy and the numpy.loadtxt() function.

This function assumes no header row and all data has the same format. The example below assumes that the file is in your current working directory.

Running the example will load the file as a numpy.ndarray and print the shape of the data:

This example can be modified to load the same dataset directly from a URL as follows:

Note: This example assumes you are using Python 3.

Again, running the example produces the same resulting shape of the data.

For more information on the numpy.loadtxt() function see the API documentation (version 1.10 of numpy).

Load CSV File With Pandas

You can load your CSV data using Pandas and the pandas.read_csv() function.

This function is very flexible and is perhaps my recommended approach for loading your machine learning data. The function returns a pandas.DataFrame that you can immediately start summarizing and plotting.

The example below assumes that the ‘‘ file is in the current working directory.

Note that in this example we explicitly specify the names of each attribute to the DataFrame. Running the example displays the shape of the data:

We can also modify this example to load CSV data directly from a URL.

Again, running the example downloads the CSV file, parses it and displays the shape of the loaded DataFrame.

To learn more about the pandas.read_csv() function you can refer to the API documentation.


In this post you discovered how to load your machine learning data in Python.

You learned three specific techniques that you can use:

  • Load CSV with Python Standard Library.
  • Load CSV File With NumPy.
  • Load CSV File With Pandas.

Your action step for this post is to type or copy-and-paste each recipe and get familiar with the different ways that you can load machine learning data in Python.

Do you have any questions about loading machine learning data in Python or about this post? Ask your question in the comments and I will do my best to answer it.

Discover Fast Machine Learning in Python!

Master Machine Learning With Python

Develop Your Own Models in Minutes

...with just a few lines of scikit-learn code

Learn how in my new Ebook:
Machine Learning Mastery With Python

Covers self-study tutorials and end-to-end projects like:
Loading data, visualization, modeling, tuning, and much more...

Finally Bring Machine Learning To
Your Own Projects

Skip the Academics. Just Results.

See What's Inside

111 Responses to How To Load Machine Learning Data in Python

  1. Avatar
    ML704 January 17, 2017 at 7:17 pm #

    What is meant here in section Load CSV with Python Standard Library. You can download the Pima Indians dataset into your local directory.
    Where is my local directory?
    I tried several ways, but it did not work

    • Avatar
      Jason Brownlee January 18, 2017 at 10:13 am #

      It means to download the CSV file to the directory where you are writing Python code. Your project’s current working directory.

      • Avatar
        ML704 January 18, 2017 at 2:56 pm #

        Thank you, I got it now!

      • Avatar
        V ABISHEK HEYER KRUPALIN October 21, 2020 at 4:17 pm #

        thanks budddy

        • Avatar
          Jason Brownlee October 22, 2020 at 6:36 am #

          You’re welcome.

          • Avatar
            anon mouse January 10, 2022 at 6:05 am #


          • Avatar
            James Carmichael January 10, 2022 at 11:07 am #

            You are very welcome Anon!

    • Avatar
      David Angwenyi June 7, 2023 at 10:46 pm #

      For those using Anaconda, you can launch Jupiter notebook and upload data 0n the notebook, that being your working directory.

      • Avatar
        James Carmichael June 8, 2023 at 7:53 am #

        Thank you David for your recommendation!

  2. Avatar
    ruby July 17, 2017 at 2:19 pm #

    how can load video dataset in python?? without tensorflow, keras, …

  3. Avatar
    constantine July 30, 2017 at 4:23 am #


    I want to keep from a CSV file only two columns and use these numbers, as x-y points, for a k-means implementation that I am doing.

    What I do now to generate my points is this:
    ” points = np.vstack(((np.random.randn(150, 2) * 0.75 + np.array([1, 0])),
    (np.random.randn(50, 2) * 0.25 + np.array([-0.5, 0.5])),
    (np.random.randn(50, 2) * 0.5 + np.array([-0.5, -0.5])))) “,
    but I want to apply my code on actual data.

    Any help?

    • Avatar
      Jason Brownlee July 30, 2017 at 7:52 am #

      Sorry, I don’t have any kmeans tutorials in Python. I may not be the best person to give you advice.

      • Avatar
        constantine July 30, 2017 at 7:51 pm #

        I don’t want anything about k-means, I have the code -computations and all- sorted out. I just want some help with the CSV files.

  4. Avatar
    Steve August 3, 2017 at 11:54 am #

    Thank you for explaining how to load data in detail.

  5. Avatar
    Fawad August 8, 2017 at 6:20 pm #

    Thanks you very much…really helpful…

  6. Avatar
    komal September 5, 2017 at 7:18 pm #

    how to load text attribute ? I got error saying could not convert string to float: b’Iris-setosa’

    • Avatar
      Jason Brownlee September 7, 2017 at 12:43 pm #

      You will need to load the data using Pandas then convert it to numbers.

      I give examples of this.

  7. Avatar
    R October 10, 2017 at 3:21 am #

    I was just wondering what the best practices are for converting something in a Relational Database model to an optimal ML format for fields that could be redundant. Ideally the export would be in CSV, but I know it won’t be as simple as an export every time. Hopefully simple example to illustrate my question: Say I have a table where I attribute things to an animal. The structure could be set up similarly to this:
    ID, Animal, Color,Continent
    With the goal of being able to say “If the color is black and white and lives in Africa, it’s probably a zebra.” …so each line represents the animal with a single color associated with it, and other fields as well. Would this type of format be a best practice to feed into the model as is? Or, would it make more sense to concatenate the colors into one line with a delimiter? In other words, it may not always be a 1:1 relationship, and in cases where the dataset is like that, what’s the best way of formatting?
    Thanks for your time.

  8. Avatar
    Hemalatha S November 17, 2017 at 6:52 pm #

    can you tell me how to select features from a csv file

  9. Avatar
    Disha Umarwani November 28, 2017 at 12:41 pm #

    I am trying to load a line separated data.
    majors:computer science


    Any advice on this?

    • Avatar
      Jason Brownlee November 29, 2017 at 8:13 am #

      Ouch, looks like you might need to write some custom code to load each “line” or entity.

  10. Avatar
    Hemalatha S December 1, 2017 at 2:17 am #

    can you tell me how to load a csv file and apply feature selection methods?? can you post code for grey wolf optimizer algorithm??

  11. Avatar
    fxdingscxr January 17, 2018 at 4:42 pm #

    I have loaded the data into numpy array. What is the next thing that i should do to train my model?

  12. Avatar
    Ajinkya January 30, 2018 at 6:29 pm #

    I want to use KDD cup 99 dataset for the intrusion detection project. The dataset consist of String & numerical data. So should I convert entire dataset into numeric data or should I use it as it is?

  13. Avatar
    Bipin February 2, 2018 at 5:11 pm #

    Hey Jason,
    I have a dataset in csv which has header and all the columns have different datatype,
    which one would be better to use in this scenario: loadtxt() or genfromtxt().
    Also, is there any major performance difference in these 2 methods?

    • Avatar
      Jason Brownlee February 3, 2018 at 8:34 am #

      Use whatever you can, consider benchmarking the approaches with your data if speed is an issue.

  14. Avatar
    ML Beginer February 15, 2018 at 3:41 pm #

    I got a ValueError: could not convert string to float
    while reading this data :

    Can you please reply where I am doing wrong?

    • Avatar
      ML Beginer February 15, 2018 at 3:45 pm #

    • Avatar
      Jason Brownlee February 16, 2018 at 8:31 am #

      You might have some “?” values. Convert them to 0 or nan first.

  15. Avatar
    ro May 8, 2018 at 4:25 am #

    filename = ‘C:\Users\user\Desktop\’
    raw_data = open(filename, ‘rt’)
    names = [‘pixle1’, ‘pixle2’, ‘pixle3’, ‘pixle4’, ‘pixle5’, ‘pixle6’, ‘pixle7’, ‘pixle8’, ‘pixle9’, ‘pixle10’, ‘pixle11’, ‘pixle12’, ‘pixle13’, ‘pixle14’, ‘pixle15’, ‘pixle16’, ‘pixle17’, ‘pixle18’, ‘pixle19’, ‘pixle20’, ‘pixle21’, ‘pixle22’, ‘pixle23’, ‘pixle24’, ‘pixle25’, ‘pixle26’, ‘pixle27’, ‘pixle28’, ‘pixle29’, ‘pixle30’, ‘class’]
    data = numpy.loadtxt(raw_data, names= names)

  16. Avatar
    AJS June 1, 2018 at 1:22 pm #

    I have multiple csv files of varying sizes that I want to use for training my neural network. I have around 1000 files ranging from about 15000 to 65000 rows of data. After I preprocess some of this data, one csv may be around 65000 rows by 20 columns array. My computer starts running out of memory very quickly on just 1 of the 65000 by 20 arrays, so I cannot combine all the 1000 files into one large csv file. Is there a way using keras to load one of the csv files, have the model learn on that data, then load the next file, have the file learn on that, and so on? Is there a better way to learn on so much data?

  17. Avatar
    Hemant June 17, 2018 at 2:32 pm #

    I have multiple 200 CSV files and labels files that contains 200 rows as output. I want to train, but unable to load the dataset

    • Avatar
      Jason Brownlee June 18, 2018 at 6:39 am #

      You may have to write come custom code to load each CSV in turn. E.g. in a loop over the files in the directory.

  18. Avatar
    Aman July 12, 2018 at 4:10 am #

    I got the error:

    Traceback (most recent call last):
    File “”, line 8, in
    data= numpy.array(x).astype(float)
    ValueError: setting an array element with a sequence.


  19. Avatar
    Kikio January 17, 2019 at 1:19 pm #

    I have a dataset which contains numbers like this: 3,6e+12, 2.5e-3…
    when reading this dataset as a CSV file, I get the error: “Value error: cannot convert string to float”

    Any solution please??

    • Avatar
      Jason Brownlee January 17, 2019 at 1:47 pm #

      The numbers are in scientific notation and will be read correctly.

      Perhaps there are other non-number fields in the file?

      • Avatar
        Kikio January 17, 2019 at 11:54 pm #

        No, there aren’t, and the error says :” cannot covert string to float in 3.6e+12″
        thank you

        • Avatar
          Jason Brownlee January 18, 2019 at 5:40 am #

          That is surprising, perhaps try a different approach to loading, e.g. numpy or pandas?

          Perhaps try posting to stackoverflow?

          • Avatar
            Kikio January 19, 2019 at 11:14 am #

            I’ll try ,

  20. Avatar
    Sandeep Nithyanandan January 23, 2019 at 6:00 pm #

    Suppose i have 3 csv files , each having a particular attribute in it. So a single row in the 3 csv file correspond to a particular feature instance. So during the loading time can i load all the csv file together and convert each row into numpy array,

    • Avatar
      Jason Brownlee January 24, 2019 at 6:41 am #

      I recommend loading all data into memory then perhaps concatenate the numpy arrays (e.g. hstack).

  21. Avatar
    Sara January 26, 2019 at 7:11 am #

    If I have a data set with .data file extention how can I deal with it in python?

    please help

    • Avatar
      Jason Brownlee January 27, 2019 at 7:36 am #

      Perhaps use a text editor to open it and confirm it is in CVS format, then open it in Python as though it were a CSV file.

  22. Avatar
    francistien January 27, 2019 at 9:05 am #

    I copy your codes as follows:

    # Load CSV using NumPy
    # You can load your CSV data using NumPy and the numpy.loadtxt() function.

    import numpy
    filename = ‘pima-indians-diabetes.csv’
    raw_data = open(filename, ‘rt’)
    data = numpy.loadtxt(raw_data, delimiter=”,”)


    However, I got an error message

    ValueError Traceback (most recent call last)
    5 filename = ‘pima-indians-diabetes.csv’
    6 raw_data = open(filename, ‘rt’)
    —-> 7 data = numpy.loadtxt(raw_data, delimiter=”,”)
    8 print(data.shape)

    ~\Anaconda3\lib\site-packages\numpy\lib\ in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack, ndmin, encoding)
    1099 # converting the data
    1100 X = None
    -> 1101 for x in read_data(_loadtxt_chunksize):
    1102 if X is None:
    1103 X = np.array(x, dtype)

    ~\Anaconda3\lib\site-packages\numpy\lib\ in read_data(chunk_size)
    1027 # Convert each value according to its column and store
    -> 1028 items = [conv(val) for (conv, val) in zip(converters, vals)]
    1030 # Then pack it according to the dtype’s nesting

    ~\Anaconda3\lib\site-packages\numpy\lib\ in (.0)
    1027 # Convert each value according to its column and store
    -> 1028 items = [conv(val) for (conv, val) in zip(converters, vals)]
    1030 # Then pack it according to the dtype’s nesting

    ~\Anaconda3\lib\site-packages\numpy\lib\ in floatconv(x)
    744 if ‘0x’ in x:
    745 return float.fromhex(x)
    –> 746 return float(x)
    748 typ = dtype.type

    ValueError: could not convert string to float: ‘Pregnancies’

    I do not know what is wrong.

  23. Avatar
    vinayak mr March 31, 2019 at 11:29 pm #

    how to load the dataset from the working directory to colab?

  24. Avatar
    Jackson April 3, 2019 at 5:52 am #

    When I click the “update: download from here” to download the CSV file, it takes me to a white page with number on the left side which looks to be the data. How do I get / download this data into a CSV file? Thanks!

  25. Avatar
    Oscar April 8, 2019 at 4:43 am #

    Hi Jason,

    I hope you can help me with the following preprocessed dataset.txt file. How can I load this dataset in python? It contains a total of 54,256 rows and 28 columns. Can I use pandas?

    [0.08148002361739815, 3.446134970078908e-05, 4.747197881944017e-05, 0.0034219001610305954, 0.047596616392169624, 0.11278174138979659, 0.0011501307441196414, 1.0, 0.09648950774661698, 0.09152382450070766, 0.0032736389720705384, 0.02231715511892242, 0.0, -1.0, 0.0, -1.0, -1.0, -1.0, 0.0, -1.0, -1.0, -1.0, 0.0, 0.0, 0.0, -1.0, 1.0, -1.0]

    [0.0816768352686479, 2.929466010613462e-05, 1.2086789450560964e-06, 0.6246987951807229, 0.04743433880824845, 0.11350265074251698, 0.0011614423285977043, 1.0, 0.0965330892767645, 0.0914339631118999, 0.003190342698832632, 0.022268885790504313, 0.0, -1.0, 0.0, -1.0, -1.0, -1.0, 0.0, -1.0, -1.0, -1.0, 0.0, 0.0, 0.0, -1.0, 1.0, -1.0]

    [0.08226727022239716, 2.987144231823633e-05, 2.2329338947249727e-06, 0.047448165869218496, 0.04753095407349041, 0.11459941368369171, 0.0011702815567795678, 1.0, 0.0969906953433135, 0.09170354727832318, 0.003358412434012629, 0.022329898179060795, 0.0, -1.0, 0.0, -1.0, -1.0, -1.0, 0.0, -1.0, -1.0, -1.0, 0.0, 0.0, 0.0, -1.0, 1.0, -1.0]

    • Avatar
      Jason Brownlee April 8, 2019 at 5:59 am #

      You can load it as a dataframe or a numpy array directly.

      What problem are you having exactly?

      • Avatar
        Oscar April 8, 2019 at 6:54 am #

        When I try to load it as a numpy array it returns the list again

        I am using the following code after loading the dataset.txt file into memory:

        import numpy as np

        dataset = load_doc(‘dataset.txt’)

        x= np.asarray(dataset)

        print (x)

        • Avatar
          Jason Brownlee April 8, 2019 at 1:55 pm #



          • Avatar
            Oscar April 9, 2019 at 6:51 am #

            Thank you so much!

            So my last question (hopefully) is that I have the dataset, the labels and a list of 28 titles for the columns. I am trying to load them in python so I can split them and create my training and testing datasets. I am not sure what to do with the titles. Do I need to load them as well?

          • Avatar
            Jason Brownlee April 9, 2019 at 2:36 pm #

            You can use the column heading as the first line in the CSV file and load them automatically with pandas.

            Alternately, you can specify them as the columns in python, if needed.

            Or discard them completely.

  26. Avatar
    Shakil Ahmed May 4, 2019 at 9:39 pm #

    i am new .
    please help me to convert image dataset to csv.

  27. Avatar
    sahil May 23, 2019 at 2:49 am #

    how can i load data from parser

    from parser import load_data #dataloading

    • Avatar
      Jason Brownlee May 23, 2019 at 6:07 am #

      I don’t understand, sorry. Perhaps try posting to stackoverflow?

  28. Avatar
    Akshay Varshney June 23, 2019 at 11:27 pm #

    Hi, Jason, the dataset has been removed from the above link and I want to check that because the whole of your book is based on that dataset only, so please provide us the dataset as it would become easy for us to understand concepts from your book, please provide the dataset.
    Thank You

  29. Avatar
    S M Abdullah Al Shuaeb August 31, 2019 at 4:30 am #

    sir, pls help me

    i just want ,
    how to classify categorical image by SVM and KNN alogrithm using python

  30. Avatar
    Araz Sami September 6, 2019 at 6:00 am #


    Thank you so much for all the great Tutorials. I would like to use a multivariate time series dataset and at first I need to make a similar format as of load_basic_motion data in Python. I have several text files each representing one feature and each file has time series data for each observation. Do you have any suggestions for preparing the data in the required format?


  31. Avatar
    Sudhanshu varun September 7, 2019 at 5:00 pm #

    i successfully loaded my csv file dataset. Its basically a letter dataset and now i want to train my python with this loaded dataset so that i can use this to recognise words later can you help me with is ?
    thank you

  32. Avatar
    JJ October 6, 2019 at 7:07 pm #

    Hi Jason,

    One question here, may I know how can I load my non-csv data (a normal file instead) on spyder pyhton without converting to csv file dataset?

    • Avatar
      Jason Brownlee October 7, 2019 at 8:28 am #

      Yes, you can customize the call to read_csv() function for your dataset.

  33. Avatar
    Nauman October 10, 2019 at 7:35 am #

    X = list(map(lambda x: np.array(x), X))

    X = list(map(lambda x: x.reshape(1, x.shape[0], x.shape[1]), X))
    y = np.expand_dims(y, axis=-1)

    I used Tcn model .when i run i got this error .Index out of Range please please help me how to solve this error ..i also search from stackoverflow but not found

  34. Avatar
    Ipsita Dalai November 30, 2019 at 5:35 pm #

    Thanks for this nice article.I want to know if we have a digit classification problem and the last column contain the class.Then how to load and print the digits ignoring the last column.
    I tried it and it is showing .

    ValueError: cannot reshape array of size 257 into shape (16,16)

  35. Avatar
    Ipsita Dalai December 2, 2019 at 5:32 pm #

    Hi.I got my work done by keeping the data in the csv in numpy arrays and then slicing the array.However your tutorials are very nice and helpful.Thanks.

  36. Avatar
    Alam Noor December 4, 2019 at 3:30 am #

    Dear Jason,
    How I can load .rek dataset in python? please comment if possible. Thanks

  37. Avatar
    hima December 26, 2019 at 3:33 pm #

    how to load image dataset in python code

  38. Avatar
    nazm May 21, 2020 at 12:06 am #

    hi jason, i am a fresher with no experience. how can i learn data science. can you suggest me a roadmap? that will be helpful for me.

  39. Avatar
    Aanya October 4, 2020 at 12:11 am #

    hey jason,

    i actually wanted to use some specific columns in a csv file for loading the data into a machine learning model. can you help me out.

  40. Avatar
    Dan October 7, 2020 at 4:50 pm #

    Hi!! is it possible to cluster the similar rows of a csv file ( 2 columns) together using nlp. If yes could you please guide me with a post to help with the code.

    • Avatar
      Jason Brownlee October 8, 2020 at 8:28 am #

      Yes, sorry, I don’t have an example of clustering for text data.

  41. Avatar
    Dr. Dhaval Maheta October 4, 2022 at 8:18 pm #

    If there are 9 variables in the dataset,

    names = [‘preg’, ‘plas’, ‘pres’, ‘skin’, ‘test’, ‘mass’, ‘pedi’, ‘age’, ‘class’]

    then while selecting X array it should be
    Y =array[:,9]

    Can you explain why you have used?
    X = array[:,0:8]
    Y = array[:,8]

    • Avatar
      James Carmichael October 5, 2022 at 7:37 am #

      Hi…The tutorial is for illustrative purposes. Have you reviewed the content of X and Y variables after execution of the original code? Please let us know your thoughts.

Leave a Reply