[New Book] Click to get The Beginner's Guide to Data Science!
Use the offer code 20offearlybird to get 20% off. Hurry, sale ends soon!

How To Estimate A Baseline Performance For Your Machine Learning Models in Weka

It is really important to have a performance baseline on your machine learning problem.

It will give you a point of reference to which you can compare all other models that you construct.

In this post you will discover how to develop a baseline of performance for a machine learning problem using Weka.

After reading this post you will know:

  • The importance in establishing a baseline of performance for your machine learning problem.
  • How to calculate a baseline performance using the Zero Rule method on a regression problem.
  • How to calculate a baseline performance using the Zero Rule method on a classification problem.

Kick-start your project with my new book Machine Learning Mastery With Weka, including step-by-step tutorials and clear screenshots for all examples.

Let’s get started.

How To Estimate A Baseline Performance For Your Machine Learning Models in Weka

How To Estimate A Baseline Performance For Your Machine Learning Models in Weka
Photo by Peter Stevens, some rights reserved.

Importance of Baseline Results

You cannot know which algorithm will perform the best for your problem before hand so you must try a suite of algorithms and see what works best, then double down on it.

As such, it is critically important to develop a baseline of performance when working on a machine learning problem.

A baseline provides a point of reference from which to compare other machine learning algorithms.

You can get an idea of both the absolute performance increases you can achieve over the baseline as well as lift ratios that show you relatively how much better you are doing.

Without a baseline you do not know how well you are doing on your problem. You have no point of reference to consider whether or not you have or are continuing to add value. The baseline defines the hurdle that all other machine learning algorithms must cross to demonstrate “skill” on the problem.

Need more help with Weka for Machine Learning?

Take my free 14-day email course and discover how to use the platform step-by-step.

Click to sign-up and also get a free PDF Ebook version of the course.

Zero Rule For Baseline Performance

The baseline for both classification and regression problems is called the Zero Rule algorithm. Also called ZeroR or 0-R.

Let’s take a closer look at how the Zero Rule algorithm can be used on classification and regression problems with some examples.

Baseline Performance For Regression Problems

For a regression predictive modeling problem where a numeric value is predicted, the Zero Rule algorithm predicts the mean of the training dataset.

For example, let’s demonstrate the Zero Rule algorithm on the Boston House Price prediction problem. You can download the ARFF for the Boston House Price prediction dataset from the Weka datasets webpage. It is located in the datasets-numeric.jar package in the file housing.arff.

  1. Start the Weka GUI Chooser.
  2. Click the “Explorer” button to open the Weka Explorer interface.
  3. Load the Boston house price dataset housing.arff file.
  4. Click the “Classify” tab to open the classification tab.
  5. Select the ZeroR algorithm (it should be selected by default).
  6. Select the “Cross-validation” Test options (it should be selected by default).
  7. Click the “Start” button to evaluate the algorithm on the dataset.
Weka Baseline Performance For a Regression Problem

Weka Baseline Performance For a Regression Problem

The ZeroR algorithm predicts the mean Boston House price value of 22.5 (in thousands of dollars) and achieves a RMSE of 9.21.

For any machine learning algorithm to demonstrate that it has skill on this problem, it must achieve an RMSE better than this value.

Baseline Performance for Classification Problems

For a classification predictive modeling problem where a categorical value is predicted, the Zero Rule algorithm predicts the class value that has the most observations in the training dataset.

For example, let’s demonstrate the Zero Rule algorithm on the Pima Indians onset of diabetes problem. This dataset should be located in your data/ directory of your Weka installation. If not, you can download the default Weka installation from the Weka Download webpage targeted for “Other platforms” with a .zip extension, unzip it and locate the diabetes.arff file.

  1. Start the Weka GUI Chooser.
  2. Click the “Explorer” button to open the Weka Explorer interface.
  3. Load the Pima Indians dataset diabetes.arff file.
  4. Click the “Classify” tab to open the classification tab.
  5. Select the ZeroR algorithm (it should be selected by default).
  6. Select the “Cross-validation” Test options (it should be selected by default).
  7. Click the “Start” button to evaluate the algorithm on the dataset.
Weka Baseline Performance For a Classification Problem

Weka Baseline Performance For a Classification Problem

The ZeroR algorithm predicts the tested_negative value for all instances as it is the majority class, and achieves an accuracy of 65.1%.

For any machine learning algorithm to demonstrate that it has skill on this problem, it must achieve an accuracy better than this value.

Summary

In this post you have discovered how to calculate a baseline performance for your machine learning problems using Weka.

Specifically, you learned:

  • The importance of calculating a baseline of performance on your problem.
  • How to calculate a baseline performance for a regression problem using the Zero Rule algorithm.
  • How to calculate a baseline performance for a classification problem using the Zero Rule algorithm.

Do you have any questions about calculating a baseline of performance or about this post? Ask your questions in the comments and I will do my best to answer them.

Discover Machine Learning Without The Code!

Master Machine Learning With Weka

Develop Your Own Models in Minutes

...with just a few a few clicks

Discover how in my new Ebook:
Machine Learning Mastery With Weka

Covers self-study tutorials and end-to-end projects like:
Loading data, visualization, build models, tuning, and much more...

Finally Bring The Machine Learning To Your Own Projects

Skip the Academics. Just Results.

See What's Inside

18 Responses to How To Estimate A Baseline Performance For Your Machine Learning Models in Weka

  1. Avatar
    V October 13, 2016 at 1:40 am #

    does it always have to be ZeroR as the classifier? what about other point of reference like NaiveBayes?
    and which Test Options needs to be choose for baseline? does it always have to be Cross-Validation?

    • Avatar
      Jason Brownlee October 13, 2016 at 8:36 am #

      Great questions. I like to use ZeroR, but you can baseline off whatever you like.

      I would advise using the same test harness/test options as you use to evaluate all methods on your problem.

  2. Avatar
    Dr. Fadil November 15, 2017 at 5:26 pm #

    thank you so much Dr. Jason Brownlee
    my qustion is can I use ZeroR algorithm in my resarch to predicts bankruptcy?
    what is the benefit over other algorithms?
    thank u

    • Avatar
      Jason Brownlee November 16, 2017 at 10:25 am #

      ZeroR is a baseline method to which all other methods can be compared.

  3. Avatar
    NotoriousAPP April 26, 2018 at 12:37 am #

    I’m confused. If I’m using your guidance in this post here: https://machinelearningmastery.com/dont-use-random-guessing-as-your-baseline-classifier/

    ….then my baseline performance or ZeroR would be 54.6% using the following calculations:
    P(class is No) * P(you guess No) + P(class is Yes) * P(you guess Yes) = =0.651*0.651+0.349*0.349 = 0.546.

    What am I missing here?

    -Thanks-

    • Avatar
      Jason Brownlee April 26, 2018 at 6:36 am #

      Zero Rule is not random guessing.

      The equation is for random guessing.

      Zero Rule will guess the mode for classification and the mean for regression.

  4. Avatar
    Shravan June 17, 2018 at 5:37 pm #

    Hi Jason, is there a way to look at the program generated by Weka based on our selections or actions in the weka interface?

    Thanks

    • Avatar
      Jason Brownlee June 18, 2018 at 6:40 am #

      If you are a Java programmer, you can use the API to develop a program.

  5. Avatar
    Shravan June 17, 2018 at 5:46 pm #

    Hi Jason, another question, I see these responses from running a dataset against ZeroR algorithm, but not sure against what exact data it is coming up with these predictions:

    ZeroR predicts class value: yes

    ZeroR predicts class value: FALSE
    — for what data class value is false?

    ZeroR predicts class value: sunny
    — when is it sunny?

    ZeroR predicts class value: 22.532806324110698

  6. Avatar
    Natasja November 28, 2018 at 3:27 am #

    Hi. Thanks for a great guide to ZeroR.
    I am having trouple interpreting the results of my ZeroR.
    I have run a ZeroR on my dataset for regression, and I get -0 in correlation coefficient, what does that mean? Is that a mistake? I get the RMSE to 85029231,6432 – that is also very high, right? What does that mean?

    • Avatar
      Jason Brownlee November 28, 2018 at 7:44 am #

      It suggests the variables are not correlated.

      Perhaps try other algorithms?

  7. Avatar
    San March 21, 2020 at 7:27 am #

    Can I get baseline results using WEKA, for a dataset that has lot of missing values? I already tried this in WEKA, and I got results. I just want to know if I’m making any mistake here or do I have to handle the missing values to get an appropriate baseline performance?

    Thanks
    San

    • Avatar
      Jason Brownlee March 21, 2020 at 8:29 am #

      Yes, impute using mean or median (if needed) and calculate a baseline using ZeroR.

  8. Avatar
    Riyaben Patel March 7, 2021 at 4:38 am #

    I tried running Weka Classifiers and all the classifiers yield same results as the baseline classifier. In this sense, which classifier is better?

    • Avatar
      Jason Brownlee March 7, 2021 at 5:15 am #

      It may be that the model requires further tuning of its configuration.
      It may be that you you need to try an alternate model.
      It may be that your problem is not predictable.

  9. Avatar
    Brandi March 3, 2023 at 2:15 pm #

    When you run the ZeroR model and it correctly classifies 68%, then you run the J48 for a classification tree and you only have 67% accuracy, what would you do next?

Leave a Reply