Last Updated on September 6, 2020
AutoML refers to techniques for automatically discovering the best-performing model for a given dataset.
When applied to neural networks, this involves both discovering the model architecture and the hyperparameters used to train the model, generally referred to as neural architecture search.
AutoKeras is an open-source library for performing AutoML for deep learning models. The search is performed using so-called Keras models via the TensorFlow tf.keras API.
It provides a simple and effective approach for automatically finding top-performing models for a wide range of predictive modeling tasks, including tabular or so-called structured classification and regression datasets.
In this tutorial, you will discover how to use AutoKeras to find good neural network models for classification and regression tasks.
After completing this tutorial, you will know:
- AutoKeras is an implementation of AutoML for deep learning that uses neural architecture search.
- How to use AutoKeras to find a top-performing model for a binary classification dataset.
- How to use AutoKeras to find a top-performing model for a regression dataset.
Let’s get started.
- Update Sep/2020: Updated AutoKeras version and installation instructions.
How to Use AutoKeras for Classification and Regression
Photo by kanu101, some rights reserved.
Tutorial Overview
This tutorial is divided into three parts; they are:
- AutoKeras for Deep Learning
- AutoKeras for Classification
- AutoKeras for Regression
AutoKeras for Deep Learning
Automated Machine Learning, or AutoML for short, refers to automatically finding the best combination of data preparation, model, and model hyperparameters for a predictive modeling problem.
The benefit of AutoML is allowing machine learning practitioners to quickly and effectively address predictive modeling tasks with very little input, e.g. fire and forget.
Automated Machine Learning (AutoML) has become a very important research topic with wide applications of machine learning techniques. The goal of AutoML is to enable people with limited machine learning background knowledge to use machine learning models easily.
— Auto-keras: An efficient neural architecture search system, 2019.
AutoKeras is an implementation of AutoML for deep learning models using the Keras API, specifically the tf.keras API provided by TensorFlow 2.
It uses a process of searching through neural network architectures to best address a modeling task, referred to more generally as Neural Architecture Search, or NAS for short.
… we have developed a widely adopted open-source AutoML system based on our proposed method, namely Auto-Keras. It is an open-source AutoML system, which can be downloaded and installed locally.
— Auto-keras: An efficient neural architecture search system, 2019.
In the spirit of Keras, AutoKeras provides an easy-to-use interface for different tasks, such as image classification, structured data classification or regression, and more. The user is only required to specify the location of the data and the number of models to try and is returned a model that achieves the best performance (under the configured constraints) on that dataset.
Note: AutoKeras provides a TensorFlow 2 Keras model (e.g. tf.keras) and not a Standalone Keras model. As such, the library assumes that you have Python 3 and TensorFlow 2.3.0 or higher installed.
At the time of writing, you require a prerequisite library called keras-tuner to be installed manually. You can install this library as follows:
|
1 |
sudo pip install git+https://github.com/keras-team/[email protected] |
If things change again, as they often do with fast-moving open source projects, see the official installation instructions here:
Now we can instal AutoKeras.
To install AutoKeras, you can use Pip, as follows:
|
1 |
sudo pip install autokeras |
You can confirm the installation was successful and check the version number as follows:
|
1 |
sudo pip show autokeras |
You should see output like the following:
|
1 2 3 4 5 6 7 8 9 10 |
Name: autokeras Version: 1.0.8 Summary: AutoML for deep learning Home-page: http://autokeras.com Author: Data Analytics at Texas A&M (DATA) Lab, Keras Team Author-email: [email protected] License: MIT Location: ... Requires: tensorflow, packaging, pandas, scikit-learn Required-by: |
Once installed, you can then apply AutoKeras to find a good or great neural network model for your predictive modeling task.
We will take a look at two common examples where you may want to use AutoKeras, classification and regression on tabular data, so-called structured data.
AutoKeras for Classification
AutoKeras can be used to discover a good or great model for classification tasks on tabular data.
Recall tabular data are those datasets composed of rows and columns, such as a table or data as you would see in a spreadsheet.
In this section, we will develop a model for the Sonar classification dataset for classifying sonar returns as rocks or mines. This dataset consists of 208 rows of data with 60 input features and a target class label of 0 (rock) or 1 (mine).
A naive model can achieve a classification accuracy of about 53.4 percent via repeated 10-fold cross-validation, which provides a lower-bound. A good model can achieve an accuracy of about 88.2 percent, providing an upper-bound.
You can learn more about the dataset here:
No need to download the dataset; we will download it automatically as part of the example.
First, we can download the dataset and split it into a randomly selected train and test set, holding 33 percent for test and using 67 percent for training.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# load the sonar dataset from pandas import read_csv from sklearn.model_selection import train_test_split from sklearn.preprocessing import LabelEncoder # load dataset url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/sonar.csv' dataframe = read_csv(url, header=None) print(dataframe.shape) # split into input and output elements data = dataframe.values X, y = data[:, :-1], data[:, -1] print(X.shape, y.shape) # basic data preparation X = X.astype('float32') y = LabelEncoder().fit_transform(y) # separate into train and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1) print(X_train.shape, X_test.shape, y_train.shape, y_test.shape) |
Running the example first downloads the dataset and summarizes the shape, showing the expected number of rows and columns.
The dataset is then split into input and output elements, then these elements are further split into train and test datasets.
|
1 2 3 |
(208, 61) (208, 60) (208,) (139, 60) (69, 60) (139,) (69,) |
We can use AutoKeras to automatically discover an effective neural network model for this dataset.
This can be achieved by using the StructuredDataClassifier class and specifying the number of models to search. This defines the search to perform.
|
1 2 3 |
... # define the search search = StructuredDataClassifier(max_trials=15) |
We can then execute the search using our loaded dataset.
|
1 2 3 |
... # perform the search search.fit(x=X_train, y=y_train, verbose=0) |
This may take a few minutes and will report the progress of the search.
Next, we can evaluate the model on the test dataset to see how it performs on new data.
|
1 2 3 4 |
... # evaluate the model loss, acc = search.evaluate(X_test, y_test, verbose=0) print('Accuracy: %.3f' % acc) |
We then use the model to make a prediction for a new row of data.
|
1 2 3 4 5 6 |
... # use the model to make a prediction row = [0.0200,0.0371,0.0428,0.0207,0.0954,0.0986,0.1539,0.1601,0.3109,0.2111,0.1609,0.1582,0.2238,0.0645,0.0660,0.2273,0.3100,0.2999,0.5078,0.4797,0.5783,0.5071,0.4328,0.5550,0.6711,0.6415,0.7104,0.8080,0.6791,0.3857,0.1307,0.2604,0.5121,0.7547,0.8537,0.8507,0.6692,0.6097,0.4943,0.2744,0.0510,0.2834,0.2825,0.4256,0.2641,0.1386,0.1051,0.1343,0.0383,0.0324,0.0232,0.0027,0.0065,0.0159,0.0072,0.0167,0.0180,0.0084,0.0090,0.0032] X_new = asarray([row]).astype('float32') yhat = search.predict(X_new) print('Predicted: %.3f' % yhat[0]) |
We can retrieve the final model, which is an instance of a TensorFlow Keras model.
|
1 2 3 |
... # get the best performing model model = search.export_model() |
We can then summarize the structure of the model to see what was selected.
|
1 2 3 |
... # summarize the loaded model model.summary() |
Finally, we can save the model to file for later use, which can be loaded using the TensorFlow load_model() function.
|
1 2 3 |
... # save the best performing model to file model.save('model_sonar.h5') |
Tying this together, the complete example of applying AutoKeras to find an effective neural network model for the Sonar dataset is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
# use autokeras to find a model for the sonar dataset from numpy import asarray from pandas import read_csv from sklearn.model_selection import train_test_split from sklearn.preprocessing import LabelEncoder from autokeras import StructuredDataClassifier # load dataset url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/sonar.csv' dataframe = read_csv(url, header=None) print(dataframe.shape) # split into input and output elements data = dataframe.values X, y = data[:, :-1], data[:, -1] print(X.shape, y.shape) # basic data preparation X = X.astype('float32') y = LabelEncoder().fit_transform(y) # separate into train and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1) print(X_train.shape, X_test.shape, y_train.shape, y_test.shape) # define the search search = StructuredDataClassifier(max_trials=15) # perform the search search.fit(x=X_train, y=y_train, verbose=0) # evaluate the model loss, acc = search.evaluate(X_test, y_test, verbose=0) print('Accuracy: %.3f' % acc) # use the model to make a prediction row = [0.0200,0.0371,0.0428,0.0207,0.0954,0.0986,0.1539,0.1601,0.3109,0.2111,0.1609,0.1582,0.2238,0.0645,0.0660,0.2273,0.3100,0.2999,0.5078,0.4797,0.5783,0.5071,0.4328,0.5550,0.6711,0.6415,0.7104,0.8080,0.6791,0.3857,0.1307,0.2604,0.5121,0.7547,0.8537,0.8507,0.6692,0.6097,0.4943,0.2744,0.0510,0.2834,0.2825,0.4256,0.2641,0.1386,0.1051,0.1343,0.0383,0.0324,0.0232,0.0027,0.0065,0.0159,0.0072,0.0167,0.0180,0.0084,0.0090,0.0032] X_new = asarray([row]).astype('float32') yhat = search.predict(X_new) print('Predicted: %.3f' % yhat[0]) # get the best performing model model = search.export_model() # summarize the loaded model model.summary() # save the best performing model to file model.save('model_sonar.h5') |
Running the example will report a lot of debug information about the progress of the search.
The models and results are all saved in a folder called “structured_data_classifier” in your current working directory.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
... [Trial complete] [Trial summary] |-Trial ID: e8265ad768619fc3b69a85b026f70db6 |-Score: 0.9259259104728699 |-Best step: 0 > Hyperparameters: |-classification_head_1/dropout_rate: 0 |-optimizer: adam |-structured_data_block_1/dense_block_1/dropout_rate: 0.0 |-structured_data_block_1/dense_block_1/num_layers: 2 |-structured_data_block_1/dense_block_1/units_0: 32 |-structured_data_block_1/dense_block_1/units_1: 16 |-structured_data_block_1/dense_block_1/units_2: 512 |-structured_data_block_1/dense_block_1/use_batchnorm: False |-structured_data_block_1/dense_block_2/dropout_rate: 0.25 |-structured_data_block_1/dense_block_2/num_layers: 3 |-structured_data_block_1/dense_block_2/units_0: 32 |-structured_data_block_1/dense_block_2/units_1: 16 |-structured_data_block_1/dense_block_2/units_2: 16 |-structured_data_block_1/dense_block_2/use_batchnorm: False |
The best-performing model is then evaluated on the hold-out test dataset.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
In this case, we can see that the model achieved a classification accuracy of about 82.6 percent.
|
1 |
Accuracy: 0.826 |
Next, the architecture of the best-performing model is reported.
We can see a model with two hidden layers with dropout and ReLU activation.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 60)] 0 _________________________________________________________________ categorical_encoding (Catego (None, 60) 0 _________________________________________________________________ dense (Dense) (None, 256) 15616 _________________________________________________________________ re_lu (ReLU) (None, 256) 0 _________________________________________________________________ dropout (Dropout) (None, 256) 0 _________________________________________________________________ dense_1 (Dense) (None, 512) 131584 _________________________________________________________________ re_lu_1 (ReLU) (None, 512) 0 _________________________________________________________________ dropout_1 (Dropout) (None, 512) 0 _________________________________________________________________ dense_2 (Dense) (None, 1) 513 _________________________________________________________________ classification_head_1 (Sigmo (None, 1) 0 ================================================================= Total params: 147,713 Trainable params: 147,713 Non-trainable params: 0 _________________________________________________________________ |
AutoKeras for Regression
AutoKeras can also be used for regression tasks, that is, predictive modeling problems where a numeric value is predicted.
We will use the auto insurance dataset that involves predicting the total payment from claims given the total number of claims. The dataset has 63 rows and one input and one output variable.
A naive model can achieve a mean absolute error (MAE) of about 66 using repeated 10-fold cross-validation, providing a lower-bound on expected performance. A good model can achieve a MAE of about 28, providing a performance upper-bound.
You can learn more about this dataset here:
We can load the dataset and split it into input and output elements and then train and test datasets.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# load the sonar dataset from pandas import read_csv from sklearn.model_selection import train_test_split # load dataset url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/auto-insurance.csv' dataframe = read_csv(url, header=None) print(dataframe.shape) # split into input and output elements data = dataframe.values data = data.astype('float32') X, y = data[:, :-1], data[:, -1] print(X.shape, y.shape) # separate into train and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1) print(X_train.shape, X_test.shape, y_train.shape, y_test.shape) |
Running the example loads the dataset, confirming the number of rows and columns, then splits the dataset into train and test sets.
|
1 2 3 |
(63, 2) (63, 1) (63,) (42, 1) (21, 1) (42,) (21,) |
AutoKeras can be applied to a regression task using the StructuredDataRegressor class and configured for the number of models to trial.
|
1 2 3 |
... # define the search search = StructuredDataRegressor(max_trials=15, loss='mean_absolute_error') |
The search can then be run and the best model saved, much like in the classification case.
|
1 2 3 4 5 |
... # define the search search = StructuredDataRegressor(max_trials=15, loss='mean_absolute_error') # perform the search search.fit(x=X_train, y=y_train, verbose=0) |
We can then use the best-performing model and evaluate it on the hold out dataset, make a prediction on new data, and summarize its structure.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
... # evaluate the model mae, _ = search.evaluate(X_test, y_test, verbose=0) print('MAE: %.3f' % mae) # use the model to make a prediction X_new = asarray([[108]]).astype('float32') yhat = search.predict(X_new) print('Predicted: %.3f' % yhat[0]) # get the best performing model model = search.export_model() # summarize the loaded model model.summary() # save the best performing model to file model.save('model_insurance.h5') |
Tying this together, the complete example of using AutoKeras to discover an effective neural network model for the auto insurance dataset is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
# use autokeras to find a model for the insurance dataset from numpy import asarray from pandas import read_csv from sklearn.model_selection import train_test_split from autokeras import StructuredDataRegressor # load dataset url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/auto-insurance.csv' dataframe = read_csv(url, header=None) print(dataframe.shape) # split into input and output elements data = dataframe.values data = data.astype('float32') X, y = data[:, :-1], data[:, -1] print(X.shape, y.shape) # separate into train and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1) print(X_train.shape, X_test.shape, y_train.shape, y_test.shape) # define the search search = StructuredDataRegressor(max_trials=15, loss='mean_absolute_error') # perform the search search.fit(x=X_train, y=y_train, verbose=0) # evaluate the model mae, _ = search.evaluate(X_test, y_test, verbose=0) print('MAE: %.3f' % mae) # use the model to make a prediction X_new = asarray([[108]]).astype('float32') yhat = search.predict(X_new) print('Predicted: %.3f' % yhat[0]) # get the best performing model model = search.export_model() # summarize the loaded model model.summary() # save the best performing model to file model.save('model_insurance.h5') |
Running the example will report a lot of debug information about the progress of the search.
The models and results are all saved in a folder called “structured_data_regressor” in your current working directory.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
... [Trial summary] |-Trial ID: ea28b767d13e958c3ace7e54e7cb5a14 |-Score: 108.62509155273438 |-Best step: 0 > Hyperparameters: |-optimizer: adam |-regression_head_1/dropout_rate: 0 |-structured_data_block_1/dense_block_1/dropout_rate: 0.0 |-structured_data_block_1/dense_block_1/num_layers: 2 |-structured_data_block_1/dense_block_1/units_0: 16 |-structured_data_block_1/dense_block_1/units_1: 1024 |-structured_data_block_1/dense_block_1/units_2: 128 |-structured_data_block_1/dense_block_1/use_batchnorm: True |-structured_data_block_1/dense_block_2/dropout_rate: 0.5 |-structured_data_block_1/dense_block_2/num_layers: 2 |-structured_data_block_1/dense_block_2/units_0: 256 |-structured_data_block_1/dense_block_2/units_1: 64 |-structured_data_block_1/dense_block_2/units_2: 1024 |-structured_data_block_1/dense_block_2/use_batchnorm: True |
The best-performing model is then evaluated on the hold-out test dataset.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
In this case, we can see that the model achieved a MAE of about 24.
|
1 |
MAE: 24.916 |
Next, the architecture of the best-performing model is reported.
We can see a model with two hidden layers with ReLU activation.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 1)] 0 _________________________________________________________________ categorical_encoding (Catego (None, 1) 0 _________________________________________________________________ dense (Dense) (None, 64) 128 _________________________________________________________________ re_lu (ReLU) (None, 64) 0 _________________________________________________________________ dense_1 (Dense) (None, 512) 33280 _________________________________________________________________ re_lu_1 (ReLU) (None, 512) 0 _________________________________________________________________ dense_2 (Dense) (None, 128) 65664 _________________________________________________________________ re_lu_2 (ReLU) (None, 128) 0 _________________________________________________________________ regression_head_1 (Dense) (None, 1) 129 ================================================================= Total params: 99,201 Trainable params: 99,201 Non-trainable params: 0 _________________________________________________________________ |
Further Reading
This section provides more resources on the topic if you are looking to go deeper.
- Automated machine learning, Wikipedia.
- Neural architecture search, Wikipedia.
- AutoKeras Homepage.
- AutoKeras GitHub Project.
- Auto-keras: An efficient neural architecture search system, 2019.
- Results for Standard Classification and Regression Machine Learning Datasets
Summary
In this tutorial, you discovered how to use AutoKeras to find good neural network models for classification and regression tasks.
Specifically, you learned:
- AutoKeras is an implementation of AutoML for deep learning that uses neural architecture search.
- How to use AutoKeras to find a top-performing model for a binary classification dataset.
- How to use AutoKeras to find a top-performing model for a regression dataset.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Does AutoKeras/AutoML tell you which Neural Network Algorithm it chose. The above example doesn’t tell you which algorithm it found to give the best result.
Yes, it reports the model architecture. We show this in the above tutorial.
Dear jason.
Can we use autokeras in a 1d cnn for time series classification?
Perhaps try it and see?
Do you how to use 1d cnn layer with autokeras ? Is there a parameter to do that ?
By default, autokeras is not using 1d cnn layers to build model right ?
Thank you
1D CNNs are appropriate for sequence prediction problems, not simple classification and regression.
Perhaps you can use autokeras for sequence prediction problems, I’m not sure off the cuff – I recommend checking the documentation.
Is it applying a grid search technology, to apply model one by one and find the best results?
No, it is using an architecture search.
Having a problem installing autokeras with Anaconda?
Sorry to hear that, what problems exactly?
pip install git+https://github.com/keras-team/[email protected]
need to install before installing autokeras
Thanks for sharing.
hello Jason,
Do you have in mind to do an article about autokeras with rnnblock?
https://autokeras.com/block/#rnnblock
Thanks for the suggestion!
Jason,
You might want to update your autokeras installation example above. The current autokeras 1.0.8 requires keras-tuner 1.0.2rc1 to successfully install.
pip install git+https://github.com/keras-team/[email protected]
pip install autokeras
This is directly from https://autokeras.com/install/.
Thanks Jeff.
Thanks, Jason and Jeff, I also had trouble installing using the official Autokeras tutorial. Your code helped.
Hi Jason,
I have an error with the AutoKeras for Classification scripts (the sonar dataset example).
It looks like an issue with tensorfow. I have:
NotFoundError: Failed to create a NewWriteableFile: .\structured_data_classifier\trial_4d05b06e9169e134eb46fa62f3778824\checkpoints\ep… : The system cannot find the path specified.
; No such process [Op:SaveV2]
How could I fix this?
Thank you for you help,
Best,
Martin
Sorry to hear that, perhaps these tips will help:
https://machinelearningmastery.com/faq/single-faq/why-does-the-code-in-the-tutorial-not-work-for-me
It was not a big deal. Just need to specify the directory for storing the search outputs.
search = StructuredDataClassifier(max_trials=15,directory=my_directory)
Thank you for your great tuto.
Kind regards
Nice work!
I also have a proble with autokeras.
the installation looks fine and I got exactly the same info as your “sudo pip show autokeras”
I’ve tryed also to uninstall and reinstall with above suggestion ([email protected]), but this is not working.
my error message is
ImportError: cannot import name ‘StructuredDataClassifier’ from partially initialized module ‘autokeras’ (most likely due to a circular import) ….
The error appears on line 5 of th escript : “from autokeras import StructuredDataClassifier
”
Any Idea on how to move foward on such thing.
My config Windows 10. Python 3.8.3. and all librairies such as tensorflow, keras, theano etc are working fine on other scripts.
Thanks Gilles
Sorry to hear that.
Ensure you’re not using an IDE or a notebook, and instead that you’re running from the command line:
https://machinelearningmastery.com/faq/single-faq/how-do-i-run-a-script-from-the-command-line
Dear Dr Jason
* I have installed the packages by installing by source. OK
* Particularly followed the advice https://autokeras.com/install/ on installation and used a git bash command installation.
* I followed the advice of the commenter to direct my output to a particular directory
BACKGROUND INFORMATION – SKIP PLEASE TO BOTTOM OF THIS COMMENT FOR QUESTIONS. THANK YOU.
RESULT – both programs took a ‘long’ time to complete. I don’t know if it is the stochastic nature of the algorithm returning different results.
First program – comparing my results with your results
My accuracy = 0.783, your accuracy 0.826
I ran this program twice and got a lower accuracy than your model
No of params: mine 3162, yours 147313
Second program
I also ran the second program and compared the results of my running and your running.
MAE – very similar between my running and your running, 24.663 cf 24.496
BUT the # params were significantly different
Conclusions:
* First program, I ran a few times and my accuracy was about 0.77 compared to your 0.8+
* In the first program, the # params was significanly different in magnitude.
* Second program very similar result, BUT # params significantly different.
In addition:
* I ran the second program on a command line instead of IDE. I observed that there was CUDA info. Yes my GPU is NVidia, I did not have a CUDA software and python wrappers installed – my machine is nearly 10 years old.
* Further to that, I only installed Tensorflow without GPU.
**Questions:**
* First program, why I frequency achieved an accuracy of 0.77+ compared to your 0.8+ accuracy.
* Why I got significantly different # params.
* Second program produced similar result to yours. BUT # params significantly different.
* Would having CUDA software and Tensorflow for CUDA make the program run faster?
Thank you,
Anthony of Sydney
We can expect a difference in models and performance:
https://machinelearningmastery.com/faq/single-faq/why-do-i-get-different-results-each-time-i-run-the-code
As for the specific differences you see, I don’t know.
Yes, running on a GPU would speed up the run.
Dear Dr Jason,
Thank you,
Anthony of Sydney
Dear Dr Jason,
I did some further experimentation on the first program involving the StructuredDataClassifier.
In the first example,you did a prediction with variable ‘row’.
I wanted to see what happens when predicting the values using X_test.
In sum, whether yhat is a float or integer, the correlation between y_test and yhat (whether decimal or float) is not that strong at 0.62. This is despite the accuracy of the model being 0.811.
Yet when we predict using X_train:
The correlation between the training Y and the fitted yhat5 is 0.879
Even though the length of y_test is 0.33* len(y), why is the correlation higher at 0.88 with the test y_train and not so with the test y_test with correlation 062?
Thank you,
Anthony of Sydney
Nice work.
On 09/19/20, Anthony The Koala noted that his accuracy was 0.783 (with 3162 params).
I believe that the following line would help to feed the structured data classifier:
search.fit(x=X_train, y=y_train, epochs=10) (see my output below).
Cheers,
Slava
My output:
Trial 10 Complete [00h 00m 02s]
val_accuracy: 1.0
Best val_accuracy So Far: 1.0
Total elapsed time: 00h 00m 29s
INFO:tensorflow:Oracle triggered exit
Epoch 1/10
5/5 [==============================] – 1s 4ms/step – loss: 0.6745 – accuracy: 0.5618
Epoch 2/10
5/5 [==============================] – 0s 4ms/step – loss: 0.6105 – accuracy: 0.7298
Epoch 3/10
5/5 [==============================] – 0s 4ms/step – loss: 0.5654 – accuracy: 0.7744
Epoch 4/10
5/5 [==============================] – 0s 5ms/step – loss: 0.5285 – accuracy: 0.8148
Epoch 5/10
5/5 [==============================] – 0s 5ms/step – loss: 0.4966 – accuracy: 0.8148
Epoch 6/10
5/5 [==============================] – 0s 3ms/step – loss: 0.4677 – accuracy: 0.8044
Epoch 7/10
5/5 [==============================] – 0s 3ms/step – loss: 0.4411 – accuracy: 0.8296
Epoch 8/10
5/5 [==============================] – 0s 3ms/step – loss: 0.4159 – accuracy: 0.8405
Epoch 9/10
5/5 [==============================] – 0s 4ms/step – loss: 0.3914 – accuracy: 0.8611
Epoch 10/10
5/5 [==============================] – 0s 3ms/step – loss: 0.3673 – accuracy: 0.8798
INFO:tensorflow:Assets written to: ./structured_data_classifier/best_model/assets
Nice!
Great API, many thanks for drawing our attention to this.
Unfortunately, it seems that the authors have excluded the possibility of using sample weights for the training, which are otherwise smoothly integrated into Keras. I tried to do it by passing weight matrices via **kwargs and also by feeding in weighted data using the TF Datasets objects, but none of them worked. A closer look at the Autokeras source code revealed that it was purposely written without sample weighting in mind. I hope the authors will consider adding this functionality in future releases. Or am I missing something here?
Sorry to hear that.
Perhaps try posting the problem as an issue on the github project to make the authors aware?
Hi Dr. Brownlee,
I am trying to perform regression on a dataframe using Autokeras. I wonder if I need to scale the train and test dataset after splitting them?
Thanks,
Keyvan
Perhaps try it and compare the results.
Thank you for your kind explanation about autokeras. I have a question about model which is searched by autokeras. I found best performance model with 300 tries via autokeras. do I have to train this model again with my dataset? I don’t know this model can be used directly to treat my problem. how can I get further steps for my problem with this model? I’m new one in this area, so I might have basic questions. I look forward to hearing from you. Thank you!
You’re welcome.
Yes, fit the chosen model on your available data and use it start making predictions.
This would be a good place to start to fit your final model and make predictions:
https://machinelearningmastery.com/how-to-make-classification-and-regression-predictions-for-deep-learning-models-in-keras/
Hi Jason,
I would like to know how can I get network configuration from Hdf5 file here as Autokeras thrown an architecture with custom layer so is it possible to manuallly recreate architecture which autokeras proposes as best architecture. If it used by built_in layers then we can get it from get_config() function but here it’s custom layer. Do you have any idea about that?
Thanks for nice article.
I’m not sure off hand, perhaps experiment and discover how to solve it.
Hi Jason,
AutoKeras not working for predict_proba? Is there any work around
Thanks
Call predict() as it will return probabilities directly.
Hi Jason, Actually AutoKeras returning as
[0.],
[0.],
[0.],
[1.],
[1.],
[1.],
[0.]
Its not returning probabilities for predict() function
Thanks
Interesting.
Yes, Actually if we do prediction after exporting the model it returns probabiility
model1.fit(x,y)
model1.predict -> returns class
model_export = model1..export_model()
model_export.predict -> return probability
Thanks
Nice!
Dear Jason, thanks for the tutorial.
Could you please, help, why I get NaN values for my loss?
I tried literally everythin I googled, from checking the zero values in my data, normalizing data, changing activation, changing type of loss, but still, no result.
Thank you for reply.
Scale the data, and confirm that the data is indeed scaled prior to fitting the model.
Also confirm you have no nan in the raw data.
Dear Jason, I have been a dormant beneficiary from many of your posts.
I have couple of doubts, could you please help me on that?
1. is it required to do normalization of features before using AutoKeras or AutoKeras takes care of it? If i do normalization myself before feeding it into AutoKeras, would it have any impacts on the results?
2. I have a usecase where i have to predict 2 target variables (both are categorical), with your article, i was able to predict one variable. How do i predict two categorical variables from same input data simultaneously using StructuredDataClassifier or any other means of AutoKeras?
Thank in Advance
Thanks.
Yes, that is a good idea.
Perhaps try one model per variable?
Perhaps try a multi-output model?
Perhaps try a model with one submodel head per output.
Dear Jason, I was working on some project with Autokeras and generated some good weights. Then, something caused my Ubuntu system to crash and now, the exact same trained weights are not giving the same results. It’s just far worse, any suggestions on that?
Thank you very much for all your work, I have learned a lot from you
That is surprising, the same weights should give the same predictions.
Perhaps there is a big in your test or a faulty assumption?
I have saved every file and code before the crash, so every line of code and the weights are the same. I also have the copies of the prediction results for each file, none of the current results are same with the ones that I have saved. I looked up on the internet and some people say that different Python versions or some operating system related inside-working stuff can cause that but there was no solution or an exact explanation. By the way, I don’t know if it helps but I have saved my model as tf model. Anyway, if you don’t have any idea you don’t have to reply this comment, I don’t want to take your time, thank you for all your work again!
Thanks for sharing the details.
Hi!
Thanks for the article, but do you know how to save the best trained model (estimator) from a bayesian search?
Best regards
The models are all saved for you as part of the search.
This is discussed in the tutorial.
Hi,
Can we specifically design stacked autoencoder using this approach.Can you suggest some way of finding best Autoencoder for given dateset (hyperparameters,hidden layer neurons,number of hidden layers and all).
Perhaps, you may need to experiment.
Is there an equivalent History callback for AutoKeras AutoModels?
I don’t know sorry.
Hi, Can you summarize history for loss in a plot ..i tried the below it didnt work
# summarize history for loss
plt.title(‘Keras model loss’)
plt.plot(clf[‘loss’])
plt.plot(clf[‘val_loss’])
plt.ylabel(‘loss’)
plt.xlabel(‘epoch’)
plt.legend([‘training’, ‘validation’], loc=’upper right’)
plt.show()
please help??
See this example:
https://machinelearningmastery.com/display-deep-learning-model-training-history-in-keras/
Dear Jason, Thank you for your post, it helped me a lot.
But when I try to save the model in h5 format, it raised error
NotImplementedError: Save or restore weights that is not an instance of
tf.Variableis not supported in h5, usesave_format='tf'instead.However I try to save_format=’tf, but it will save a model as a tf folder instead of a file.
And I don’t know how to load the model as tf folder. It as same as h5 file?
No, not the same format. But why would you save a tf.Variable? They are not supposed to be saved.
Hi Jason,
First of all, thanks for your wonderful tutorials!!!
I was just curious to know, why AutoKeras doesn’t support object detection task (I’ve checked it in the current version 1.0.16). It doesn’t have a HEAD implemented for that either? And if one needs to go in that direction, what ar some things we need to implement in AutoKeras to make it work for Object Detection? (something like customized layers, heads, blocks etc.).
Thanks in advance.
Hi Pallavi…You may find the following of interest:
https://machinelearningmastery.com/how-to-train-an-object-detection-model-with-keras/
Great article as always, thank you.
I am trying to understand what trainable and non-trainable params mean in this context? I have a total of 19532 params but appearently 11 of them are non-trainable. What does this might mean?
non-trainable parameters of a model are those that you will not be updating and optimized during training, and that have to be defined a priori, or passed as inputs. The example of such parameters are: the number of hidden layers. nodes on each hidden layer.
Hello,
Thank you for the valuable information. I am curious about “non-trainable parameters”. I applied the code above to my example and it shows that I have 11 non-trainable parameters. What does this mean?
Hi guys, I have a strange issue with autokeras for regression:
Feeding my training data a second time to the model via predict, I receive very bad results. This is even notable with your example here. I just added for plotting issues:
predictions = search.predict(X)
miny = float(y.min())
maxy = float(y.max())
minp = float(min(predictions))
maxp = float(max(predictions))
plt.figure(figsize=(15,15))
plt.scatter(y, predictions, c=’crimson’,s=5)
p1 = max(maxp, maxy)
p2 = min(minp, miny)
plt.plot([p1, 0], [p1, 0], ‘b-‘)
plt.xlabel(‘True Values’, fontsize=15)
plt.ylabel(‘Predictions’, fontsize=15)
plt.axis(‘equal’)
plt.show()
This results somehow in predicted values between 0 and 1, whereas the true values are somewhat between 0 and 400. Am I missing a standardizing process?
can this be applied to multi-input and multi-output NN ?
Hi Arch…The following may be of interest to you:
https://machinelearningmastery.com/multi-output-regression-models-with-python/