Working on a machine learning project means we need to experiment. Having a way to configure your script easily will help you move faster. In Python, we have a way to adapt the code from a command line. In this tutorial, we are going to see how we can leverage the command line arguments to a Python script to help you work better in your machine learning project.

After finishing this tutorial, you will learn

• Why we would like to control a Python script in a command line
• How we can work in a command line efficiently

Kick-start your project with my new book Python for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.

Command line arguments for your Python script. Photo by insung yoon. Some rights reserved

Overview

This tutorial is in three parts; they are:

• Running a Python script in command line
• Working on the command line
• Alternative to command line arguments

Running a Python Script in Command Line

There are many ways to run a Python script. Someone may run it as part of a Jupyter notebook. Someone may run it in an IDE. But in all platforms, it is always possible to run a Python script in command line. In Windows, you have the command prompt or PowerShell (or, even better, the Windows Terminal). In macOS or Linux, you have the Terminal or xterm. Running a Python script in command line is powerful because you can pass in additional parameters to the script.

The following script allows us to pass in values from the command line into Python:

We save these few lines into a file and run it in command line with an argument:

Then, you will see it takes our argument, converts it into an integer, adds one to it, and prints. The list sys.argv contains the name of our script and all the arguments (all strings), which in the above case, is ["commandline.py", "15"].

When you run a command line with a more complicated set of arguments, it takes some effort to process the list sys.argv. Therefore, Python provided the library argparse to help. This assumes GNU-style, which can be explained using the following example:

The optional arguments are introduced by “-” or “--“, where a single hyphen will carry a single character “short option” (such as -a, -B, and -v above), and two hyphens are for multiple characters “long options” (such as --exclude and --ignore-existing above). The optional arguments may have additional parameters, such as in -B 1024 or --exclude="*.pyc"; the 1024 and "*.pyc" are parameters to -B and --exclude, respectively. Additionally, we may also have compulsory arguments, which we just put into the command line. The part 192.168.0.3:/tmp/ and ./ above are examples. The order of compulsory arguments is important. For example, the rsync command above will copy files from 192.168.0.3:/tmp/ to ./ instead of the other way round.

The following replicates the above example in Python using argparse:

If you run the above script, you will see:

This means you didn’t provide the compulsory arguments for src and dest. Perhaps the best reason to use argparse is to get a help screen for free if you provided -h or --help as the argument, like the following:

While the script did nothing real, if you provided the arguments as required, you will see this:

The parser object created by ArgumentParser() has a parse_args() method that reads sys.argv and returns a namespace object. This is an object that carries attributes, and we can read them using args.ignore_existing, for example. But usually, it is easier to handle if it is a Python dictionary. Hence we can convert it into one using vars(args).

Usually, for all optional arguments, we provide the long option and sometimes also the short option. Then we can access the value provided from the command line using the long option as the key (with the hyphen replaced with an underscore or the single-character short option as the key if we don’t have a long version). The “positional arguments” are not optional, and their names are provided in the add_argument() function.

There are multiple types of arguments. For the optional arguments, sometimes we use them as Boolean flags, but sometimes we expect them to bring in some data. In the above, we use action="store_true" to make that option set to False by default and toggle to True if it is specified. For the other option, such as -B above, by default, it expects additional data to go following it.

We can further require an argument to be a specific type. For example, in the -B option above, we can make it expect integer data by adding type like the following:

And if we provided the wrong type, argparse will help terminate our program with an informative error message:

Working on the Command Line

Empowering your Python script with command line arguments can bring it to a new level of reusability. First, let’s look at a simple example of fitting an ARIMA model to a GDP time series. World Bank collects historical GDP data from many countries. We can make use of the pandas_datareader package to read the data. If you haven’t installed it yet, you can use pip (or conda if you installed Anaconda) to install the package:

The code for the GDP data that we use is NY.GDP.MKTP.CN; we can get the data of a country in the form of a pandas DataFrame by:

Then we can tidy up the DataFrame a bit using the tools provided by pandas:

Fitting an ARIMA model and using the model for predictions is not difficult. In the following, we fit it using the first 40 data points and forecast for the next 3. Then compare the forecast with the actual in terms of relative error:

Putting it all together, and after a little polishing, the following is the complete code:

This script prints the following output:

The above code is short, but we made it flexible enough by holding some parameters in variables. We can change the above code to use argparse so that we can change some parameters from the command line, as follows:

If we run the code above in a command line, we can see it can now accept arguments:

In the last command above, we pass in -c NO to apply the same model to the GDP data of Norway (NO) instead of Sweden (SE). Hence, without the risk of messing up the code, we reused our code for a different dataset.

The power of introducing a command line argument is that we can easily test our code with varying parameters. For example, we want to see if the ARIMA(1,1,1) model is a good model for predicting GDP, and we want to verify with a different time window of the Nordic countries:

• Denmark (DK)
• Finland (FI)
• Iceland (IS)
• Norway (NO)
• Sweden (SE)

We want to check for the window of 40 years but with different starting points (since 1960, 1965, 1970, 1975). Depending on the OS, you can build a for loop in Linux and mac using the bash shell syntax:

Or, as the shell syntax permits, we can put everything in one line:

Or even better, give some information at each iteration of the loop, and we get our script run multiple times:

If you’re using Windows, you can use the following syntax in command prompt:

or the following in PowerShell:

Both should produce the same result.

While we can put a similar loop inside our Python script, sometimes it is easier if we can do it at the command line. It could be more convenient when we are exploring different options. Moreover, by taking the loop outside of the Python code, we can be assured that every time we run the script, it is independent because we will not share any variables between iterations.

Alternative to command line arguments

Using command line arguments is not the only way to pass in data to your Python script. At least, there are several other ways too:

• using environment variables
• using config files

Environment variables are features from your OS to keep a small amount of data in memory. We can read environment variables in Python using the following syntax:

For example, in Linux, the above two-line script will work with the shell as follows:

In Windows, the syntax inside the command prompt is similar:

You may also add or edit environment variables in Windows using the dialog in the Control Panel:

So we may keep the parameters to the script in some environment variables and let the script adapt its behavior, like setting up command line arguments.

In case we have a lot of options to set, it is better to save the options to a file rather than overwhelming the command line. Depending on the format we chose, we can use the configparser or json module from Python to read the Windows INI format or JSON format, respectively. We may also use the third-party library PyYAML to read the YAML format.

For the above example running the ARIMA model on GDP data, we can modify the code to use a YAML config file:

The YAML config file is named as config.yaml, and its content is as follows:

Then we can run the above code and obtain the same result as before. The JSON counterpart is very similar, where we use the load() function from the json module:

And the JSON config file, config.json, would be:

You may learn more about the syntax of JSON and YAML for your project. But the idea here is that we can separate the data and algorithm for better reusability of our code.

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Libraries

• argparse module, https://docs.python.org/3/library/argparse.html
• Pandas Data Reader, https://pandas-datareader.readthedocs.io/en/latest/
• ARIMA in statsmodels, https://www.statsmodels.org/devel/generated/statsmodels.tsa.arima.model.ARIMA.html
• configparser module, https://docs.python.org/3/library/configparser.html
• json module, https://docs.python.org/3/library/json.html
• PyYAML, https://pyyaml.org/wiki/PyYAMLDocumentation

Articles

• Working with JSON, https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Objects/JSON
• YAML on Wikipedia, https://en.wikipedia.org/wiki/YAML

Books

• Python Cookbook, third edition, by David Beazley and Brian K. Jones, https://www.amazon.com/dp/1449340377/

Summary

In this tutorial, you’ve seen how we can use the command line for more efficient control of our Python script. Specifically, you learned:

• How we can pass in parameters to your Python script using the argparse module
• How we can efficiently control the argparse-enabled Python script in a terminal under different OS
• We can also use environment variables or config files to pass in parameters to a Python script

Get a Handle on Python for Machine Learning!

Be More Confident to Code in Python

...from learning the practical Python tricks

Discover how in my new Ebook:
Python for Machine Learning

It provides self-study tutorials with hundreds of working code to equip you with skills including:
debugging, profiling, duck typing, decorators, deployment, and much more...

Showing You the Python Toolbox at a High Level for
Your Projects

See What's Inside