Archive | Statistical Methods

How to Calculate McNemar's Test for Two Machine Learning Classifiers

How to Calculate McNemar’s Test to Compare Two Machine Learning Classifiers

The choice of a statistical hypothesis test is a challenging open problem for interpreting machine learning results. In his widely cited 1998 paper, Thomas Dietterich recommended the McNemar’s test in those cases where it is expensive or impractical to train multiple copies of classifier models. This describes the current situation with deep learning models that […]

Continue Reading 20
The Role of Randomization to Address Confounding Variables in Machine Learning

The Role of Randomization to Address Confounding Variables in Machine Learning

A large part of applied machine learning is about running controlled experiments to discover what algorithm or algorithm configuration to use on a predictive modeling problem. A challenge is that there are aspects of the problem and the algorithm called confounding variables that cannot be controlled (held constant) and must be controlled-for. An example is […]

Continue Reading 6
All of Statistics for Machine Learning

All of Statistics for Machine Learning

A foundation in statistics is required to be effective as a machine learning practitioner. The book “All of Statistics” was written specifically to provide a foundation in probability and statistics for computer science undergraduates that may have an interest in data mining and machine learning. As such, it is often recommended as a book to […]

Continue Reading 10
A Gentle Introduction to Effect Size Measures in Python

A Gentle Introduction to Effect Size Measures in Python

Statistical hypothesis tests report on the likelihood of the observed results given an assumption, such as no association between variables or no difference between groups. Hypothesis tests do not comment on the size of the effect if the association or difference is statistically significant. This highlights the need for standard ways of calculating and reporting […]

Continue Reading 4
Scatter Plot of Associated Variables Drawn From a Uniform Distribution

How to Calculate Nonparametric Rank Correlation in Python

Correlation is a measure of the association between two variables. It is easy to calculate and interpret when both variables have a well understood Gaussian distribution. When we do not know the distribution of the variables, we must use nonparametric rank correlation methods. In this tutorial, you will discover rank correlation methods for quantifying the […]

Continue Reading 23