How to Identify Outliers in your Data

Bojan Miletic asked a question about outlier detection in datasets when working with machine learning algorithms. This post is in answer to his question.

If you have a question about machine learning, sign-up to the newsletter and reply to an email or use the contact form and ask, I will answer your question and may even turn it into a blog post.

Outliers

Many machine learning algorithms are sensitive to the range and distribution of attribute values in the input data. Outliers in input data can skew and mislead the training process of machine learning algorithms resulting in longer training times, less accurate models and ultimately poorer results.

Outlier

Outlier
Photo by Robert S. Donovan, some rights reserved

Even before predictive models are prepared on training data, outliers can result in misleading representations and in turn misleading interpretations of collected data. Outliers can skew the summary distribution of attribute values in descriptive statistics like mean and standard deviation and in plots such as histograms and scatterplots, compressing the body of the data.

Finally, outliers can represent examples of data instances that are relevant to the problem such as anomalies in the case of fraud detection and computer security.

Outlier Modeling

Outliers are extreme values that fall a long way outside of the other observations. For example, in a normal distribution, outliers may be values on the tails of the distribution.

The process of identifying outliers has many names in data mining and machine learning such as outlier mining, outlier modeling and novelty detection and anomaly detection.

In his book Outlier Analysis (affiliate link), Aggarwal provides a useful taxonomy of outlier detection methods, as follows:

  • Extreme Value Analysis: Determine the statistical tails of the underlying distribution of the data. For example, statistical methods like the z-scores on univariate data.
  • Probabilistic and Statistical Models: Determine unlikely instances from a probabilistic model of the data. For example, gaussian mixture models optimized using expectation-maximization.
  • Linear Models: Projection methods that model the data into lower dimensions using linear correlations. For example, principle component analysis and data with large residual errors may be outliers.
  • Proximity-based Models: Data instances that are isolated from the mass of the data as determined by cluster, density or nearest neighbor analysis.
  • Information Theoretic Models: Outliers are detected as data instances that increase the complexity (minimum code length) of the dataset.
  • High-Dimensional Outlier Detection: Methods that search subspaces for outliers give the breakdown of distance based measures in higher dimensions (curse of dimensionality).

Aggarwal comments that the interpretability of an outlier model is critically important. Context or rationale is required around decisions why a specific data instance is or is not an outlier.

In his contributing chapter to Data Mining and Knowledge Discovery Handbook (affiliate link), Irad Ben-Gal proposes a taxonomy of outlier models as univariate or multivariate and parametric and nonparametric. This is a useful way to structure methods based on what is known about the data. For example:

  • Are you considered with outliers in one or more than one attributes (univariate or multivariate methods)?
  • Can you assume a statistical distribution from which the observations were sampled or not (parametric or nonparametric)?

Get Started

There are many methods and much research put into outlier detection. Start by making some assumptions and design experiments where you can clearly observe the effects of the those assumptions against some performance or accuracy measure.

I recommend working through a stepped process from extreme value analysis, proximity methods and projection methods.

Extreme Value Analysis

You do not need to know advanced statistical methods to look for, analyze and filter out outliers from your data. Start out simple with extreme value analysis.

  • Focus on univariate methods
  • Visualize the data using scatterplots, histograms and box and whisker plots and look for extreme values
  • Assume a distribution (Gaussian) and look for values more than 2 or 3 standard deviations from the mean or 1.5 times from the first or third quartile
  • Filter out outliers candidate from training dataset and assess your models performance

Proximity Methods

Once you have explore simpler extreme value methods, consider moving onto proximity-based methods.

  • Use clustering methods to identify the natural clusters in the data (such as the k-means algorithm)
  • Identify and mark the cluster centroids
  • Identify data instances that are a fixed distance or percentage distance from cluster centroids
  • Filter out outliers candidate from training dataset and assess your models performance

Projection Methods

Projection methods are relatively simple to apply and quickly highlight extraneous values.

  • Use projection methods to summarize your data to two dimensions (such as PCA, SOM or Sammon’s mapping)
  • Visualize the mapping and identify outliers by hand
  • Use proximity measures from projected values or codebook vectors to identify outliers
  • Filter out outliers candidate from training dataset and assess your models performance

Methods Robust to Outliers

An alternative strategy is to move to models that are robust to outliers. There are robust forms of regression that minimize the median least square errors rather than mean (so-called robust regression), but are more computationally intensive. There are also methods like decision trees that are robust to outliers.

You could spot check some methods that are robust to outliers. If there are significant model accuracy benefits then there may be an opportunity to model and filter out outliers from your training data.

Resources

There are a lot of webpages that discuss outlier detection, but I recommend reading through a good book on the subject, something more authoritative. Even looking through introductory books on machine learning and data mining won’t be that useful to you. For a classical treatment of outliers by statisticians, check out:

For a modern treatment of outliers by data mining community, see:

3 Responses to How to Identify Outliers in your Data

  1. Sandeep Karkhanis February 7, 2015 at 12:44 am #

    great blog, I have few of your mini guides and really love them.
    For a newbie in ML and python your books just cut the crap and help me get started…

    few questions,

    Q1
    Would you consider writing a mini-book actually showing implementation of ANY or ALL of the ways you described below?

    Q2
    imagine if you have ‘n’ numeric predictors, numeric target and each of them have Na’s / Nan’s in the range of 40-60% values…and lots of outliers
    So what approach would you take,
    2.1. Impute the Nan’s first
    2.2. then use your outlier function to remove outliers
    or the other way around?

    I tried using the scikit imputer in step 2.1 above but didn’t work ..any suggestions?

    http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Imputer.html#sklearn.preprocessing.Imputer

    • Jason Brownlee February 7, 2015 at 6:33 am #

      Q1: Sure.
      Q2: That is a not a lot of data and it may be hard to know the structure of your data. “Many” and “outliers” do not go together. But yes, your approach sounds reasonable.

      Try imputing with a mean, median or knn by hand as a starting point.

  2. atul August 20, 2015 at 7:34 pm #

    Hi Jason,

    I have a month-wise data where same months can have multiple entries. The issue is there are outliers only in some months and not all but the data is in millions. Plus there is no way of selectively removing the outliers.
    Which approach do you suggest?

    E.g. A user born on 1984, buys 10 items of difference cumulative prices in June 2015, which again gets add up in next month, say July 2015. So he will have 10 entries for June, where the recent entry should have maximum amount. Issue is the data is manually entered by someone so values are pretty random. I want to select the most logical value in a month for that subscriber. We can straightway remove the outliers to get a proper trend.

Leave a Reply