If you are using the Python stack for studying and applying machine learning, then the library that you will want to use for data analysis and data manipulation is Pandas.
This post gives you a quick introduction to the Pandas library and point you in the right direction for getting started.
Kick-start your project with my new book Machine Learning Mastery With Python, including step-by-step tutorials and the Python source code files for all examples.
Let’s get started.
Data Analysis In Python
The Python SciPy stack is a popular for scientific computing in general. It provides powerful libraries for handing gridded data (like NumPy) and plotting (like matplotlib). Until recently, a piece that had been missing from the suite was a good library for handling data.
Data, typically does not come in a form that is ready to used. A very large part of working on a data-driven problem like machine learning is data analysis and data munging.
- Data Analysis: This is using the tools like statistics and data visualization to better understand the problem by understanding the data.
- Data Munging: This is the process of transforming raw data into a form so that it is appropriate for your job, like data analysis or machine learning.
Traditionally, you had to cobble together your own tool-chain of scripts in Python to perform these tasks.
These days, if you search for data analysis in Python you can’t avoid learning about Pandas. It has quickly become the go-to library for data handling in Python.
What is Pandas?
Pandas is a Python library for data analysis and data manipulation. It adds the missing piece to the SciPy framework for handling data.
Pandas was create by Wes McKinney in 2008 primarily for quantitative financial work. As such it has a strong foundation in handling time series data and charting.
You use Pandas to load data into Python and perform your data analysis tasks. It is perfect for working with tabular data like data from a relational database or data from a spreadsheet.
Wes describes the vision of Pandas as to crate: the most powerful and flexible open source data analysis and manipulation tool available in any language.
An admirable mission that makes you want to support his cause, if only to make your own data analysis tasks easier.
Pandas is a pleasure to use.
In my experience it is simple, elegant and intuitive. Having come from R, the idioms and operations are familiar and relevant.
Pandas is built on top of standard libraries in the SciPy stack. It uses NumPy for fast array handling, and provides convenient wrappers around some statistical operations from StatsModels and charting from Matplotlib.
There is a strong focus on time series given the libraries inception in the financial domain. It also has a strong focus on data frames for handling standard gridded data. Data handling is a core requirement of a library of this kind and speed has been made a priority. It is fast and provides data structures and operations like indexing and handling of sparsity.
Some important features to note include”
- Manipulation: moving columns, slicing, reshaping, merging, joining, filtering, and others.
- Time-series Handling: operations on date/times, resampling, moving windows and auto-alignment of datasets.
- Missing Data Handling: auto-exclude, drop, replace, interpolate missing values
- Group-by Operations: SQL like group by.
- Hierarchical Indexing: data structure level, powerful for efficiently organizing data by columns.
- Summary Statistics: Fast and powerful summary statistics of data.
- Visualization: Simplified access to plots on data structures, such as histograms, box plots, general plots and a scatter matrix.
Pandas is available under a permissive license (Simplified BSD) and can be easily installed along with the the rest of SciPy.
This has been a quick introduction to the Pandas library and there is more to learn. Install the library, grab a dataset and start to try things out. There is no better way to get started.
A great place to start is the list of tutorials which includes links to cookbooks, lessons, and various notable IPython notebooks around the web.
Finally, for me, I live in the API documentation.
I find papers can give a good overview of an open source library, particularly in the Python and R ecosystems. Take a look at the following papers for a structured overview of what Pandas is all about.
- Data Structures for Statistical Computing in Python
- pandas: a Foundational Python Library for Data Analysis and Statistics
There are a lot of great videos on YouTube of people demonstrating Pandas on their own data and at conferences.
A great starting point is Wes’ own 10-minute tour of pandas. Take a look. It’s a little time-series data heavy, but it’s a great and quick overview. You can also checkout his IPython notebook for this tour.
Finally, Wes is the author of the definitive book on data analysis in Python. If you want to get serious, practice, but also consider grabbing the book. It’s called: Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython.