More special features in Python

Python is an awesome programming language! It is one of the most popular languages for developing AI and machine learning applications. With a very easy to learn syntax, Python has some special features that distinguish it from other languages. In this tutorial, we’ll talk about some special attributes of the Python programming language.

After completing this tutorial, you will know:

• Constructs for list and dictionary comprehension
• How to use zip and enumerate functions
• What are function contexts and decorators
• What is the purpose of generators in Python

Let’s get started.

Python Special Features
Photo by M Mani, some rights reserved.

Tutorial Overview

This tutorial is divided into 4 parts; they are:

1. List and dictionary comprehension
2. Zip and enumerate functions
3. Function contexts and decorators
4. Generators in Python with example of Keras generator

Import Section

The libraries used in this tutorial are imported in the code below.

List Comprehension

List comprehension provides a short, simple syntax for creating new lists from existing ones. For example, suppose we require a new list, where each new item is the old item multiplied by 3. One method is to use a for loop as shown below:

The shorter method using list comprehension requires only a single line of code:

You can even create a new list based on a special criterion. For example, if we want only even numbers added to the new list.

It is also possible to have an else associated with the above. For example, we can leave all even numbers intact and replace the odd numbers by zero:

List comprehension can also used to replace nested loops. For example the following:

can be done as follows, with two “for” inside the list comprehension:

Syntax

Syntax for list comprehension is given by:

newlist = [expression for item in iterable if condition == True]

Or

newList = [expression if condition == True else expression for item in iterable]

Dictionary Comprehension

Dictionary comprehension is similar to list comprehension, except now we have (key, value) pairs. Here is an example; we’ll modify each value of the dictionary by concatenating the string ‘number ‘ to each value:

Again, conditionals are also possible. We can choose to add (key, value) pairs based on a criterion in the new dictionary.

Enumerators and Zip in Python

In Python an iterable is defined as any data structure that can return all its items, one at a time. This way you can use a for loop for further processing of all items one by one. Python has two additional constructs that make for loops easier to use, i.e., enumerate() and zip().

Enumerate

In traditional programming languages, you need a loop variable to iterate through different values of a container. In Python this is simplified by giving you access to a loop variable along with one value of the iterable object. The enumerate(x) function returns two iterables. One iterable varies from 0 to len(x)-1. The other is an iterable with value equal to items of x. An example is shown below:

By default, enumerate starts at 0 but we can start at some other number if we specified it. This is useful in some situation, for example:

Zip

Zip allows you to create an iterable object of tuples. Zip takes as argument multiple containers $(m_1, m_2, \ldots, m_n)$, and creates the i-th tuple by pairing one item from each container. The i-th tuple is then $(m_{1i}, m_{2i}, \ldots, m_{ni})$. If the passed objects have different lengths, then the total number of tuples formed have a length equal to the minimum length of passed objects.

Below are examples of using both zip() and enumerate().

Function Context

Python allows nested functions, where you can define an inner function within an outer function. There are some awesome features related to nested functions in Python.

• The outer function can return a handle to the inner function
• The inner function retains all its environment and variables local to it and in its enclosing function even if the outer function ends its execution.

An example is given below with explanation in comments.

Decorators in Python

Decorators are a powerful feature of Python. You can use decorators to customize the working of a class or a function. Think of them as a function applied to another function. Use the function name with @ symbol to define the decorator function on the decorated function. The decorator takes a function as argument, giving a lot of flexibility.

Consider the following function square_decorator() that takes a function as an argument, and also returns a function.

• The inner nested function square_it()takes an argument arg.
• The square_it()function applies the function to arg and squares the result.
• We can pass a function such as sin to square_decorator(), which in turn would return $\sin^2(x)$.
• You can also write your own customized function and use the square_decorator() function on it using the special @symbol as shown below. The function plus_one(x) returns x+1. This function is decorated by the square_decorator() and hence, we get $(x+1)^2$.

Generators in Python

Generators in Python allow you to generate sequences. Instead of writing a return statement, a generator returns multiple values via multiple yield statements. The first call to the function returns the first value from yield. The second call returns the second value from yield and so on.

The generator function can be invoked via next().Every time next() is called the next yield value is returned. An example of generating the Fibonacci sequence up till a given number x is shown below.

Example of Data Generator in Keras

One use of generator is the data generator in Keras. The reason it is useful is that we do not want to keep all data in memory but want to create it on the fly when the training loop needs it. Remember in Keras, a neural network model is trained in batches, so a generator are to emit batches of data. The function below is from our previous post “Using CNN for financial time series prediction“:

The function above is to pick a random row of a pandas dataframe as a starting point and clip next several rows as one time interval sample. This process is repeated several times to collect many time intervals into one batch. When we collected enough number of interval samples, at the second last line in the above function, the batch is dispatched using the yield command. As you may already noticed, generator functions do not have return statement. In this example, the function even will run forever. This is useful and necessary because it allows our Keras training process to run as many epoch as we want.

If we do not use generator, we will need to convert the dataframe into all possible time intervals and keep them in the memory for the training loop. This will be a lot of repeating data (because the time intervals are overlapping) and takes up a lot of memory.

Because it is useful, Keras has some generator function predefined in the library. Below is an example of ImageDataGenerator(). We have loaded the cifar10 dataset of 32×32 images in x_train. The data is connected to the generator via flow() method. The next() function returns the next batch of data. In the example below, there are 4 calls to next(). In each case 8 images are returned as the batch size is 8.

Below is the entire code that also displays all images after every call to next().

This section provides more resources on the topic if you are looking to go deeper.

Summary

In this tutorial, you discovered special features of Python

Specifically, you learned:

• The purpose of list and dictionary comprehension
• How to use zip and enumerate
• Nested functions, function contexts and decorators
• Generators in Python and the ImageDataGenerator in Python

7 Responses to More special features in Python

1. Gerard Elifuraha Mtalo, PhD December 18, 2021 at 8:11 pm #

Dear Jason Brownlee (PhD) and About Mehreen Saeed,

Thank you for the useful knowledge that you continue to share globally without reservation. I am a keen follower of your many tutorials.

I have observed the following behavior in the use of the set() function and the zip() function.

>>> shapes=zip(name,sides,colors)
>>> shapes

>>> b=set(shapes)
>>> b
{(‘Pentagon’, 5, ‘blue’), (‘Triangle’, 3, ‘red’), (‘Hexagon’, 6, ‘yellow’), (‘Square’, 4, ‘green’)}
>>> print(b)
{(‘Pentagon’, 5, ‘blue’), (‘Triangle’, 3, ‘red’), (‘Hexagon’, 6, ‘yellow’), (‘Square’, 4, ‘green’)}
>>> shapes

>>> b=set(shapes)
>>> shapes

>>> print(b)
set()
>>> b
set()

Can any one tell me why b becomes an empty set when I issue the assignment b=set(shapes) the second time?

NOTE that the object shapes is still defined internally as

• Adrian Tam December 19, 2021 at 1:57 pm #

In Python 2 your code should produce what you want. But in Python 3, output of zip is a generator, which can only be read once. Hence if you run b=set(shapes) the second time, it will be empty.

2. Gerard Elifuraha Mtalo, PhD December 18, 2021 at 8:22 pm #

Dear Jason Brownlee and About Mehreen,

I noticed that the internal representation of the variable ‘shapes’ which was ” filtered out in the comment that I sent earlier.

I also needed to inform you that I run the code on Python 3.7.7 shell installed on Windows 10.

Gerard Elifuraha Mtalo

3. Anthony The Koala January 9, 2022 at 6:24 pm #

I had a go at the section “Function Context”

In the code you had two functions circle.

Output:

I blocked the first circle function. What was the purpose of the first circle function? It was never called?

Thank you,
Anthony of Sydney

• Anthony The Koala January 10, 2022 at 2:07 am #

To clarify, even if the first version of function “def circle(r)” the first function would never get invokedc.

Overloading does not work in the same way as say another language:

Output:

Conclusions:
* Obviously python does not allow overloading as demonstrated by the runtime error
* But it does not answer why one can have two functions. Note that the first function does not get invoked at all

Output showing that only the second function is called. The first function is never called/invoked.

Thank you
Anthony of Sydney

• James Carmichael January 10, 2022 at 11:13 am #

Thank you for the feedback and interest Anthony!

• Anthony The Koala January 10, 2022 at 11:45 am #

How does having two functions and only the 2nd function is invoked.

Blocking out the 1st function by commenting it out does no harm.

I don’t understand the point of this section of the tutorial.

Thank you,
Anthony of Sydney