[New Book] Click to get The Beginner's Guide to Data Science!
Use the offer code 20offearlybird to get 20% off. Hurry, sale ends soon!

More Special Features in Python

Python is an awesome programming language! It is one of the most popular languages for developing AI and machine learning applications. With a very easy-to-learn syntax, Python has some special features that distinguish it from other languages. In this tutorial, we’ll talk about some unique attributes of the Python programming language.

After completing this tutorial, you will know:

  • Constructs for list and dictionary comprehension
  • How to use the zip and enumerate functions
  • What are function contexts and decorators
  • What is the purpose of generators in Python

Kick-start your project with my new book Python for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

Python Special Features
Photo by M Mani, some rights reserved.

Tutorial Overview

This tutorial is divided into four parts; they are:

  1. List and dictionary comprehension
  2. Zip and enumerate functions
  3. Function contexts and decorators
  4. Generators in Python with an example of Keras generator

Import Section

The libraries used in this tutorial are imported in the code below.

List Comprehension

List comprehension provides a short, simple syntax for creating new lists from existing ones. For example, suppose we require a new list, where each new item is the old item multiplied by 3. One method is to use a for loop as shown below:

The shorter method using list comprehension requires only a single line of code:

You can even create a new list based on a special criterion. For example, if we want only even numbers added to the new list:

It is also possible to have an else associated with the above. For example, we can leave all even numbers intact and replace the odd numbers with zero:

List comprehension can also used to replace nested loops. For example:

can be done as follows, with two “for” inside the list comprehension:

Syntax

Syntax for list comprehension is given by:

newlist = [expression for item in iterable if condition == True]

or

newList = [expression if condition == True else expression for item in iterable]

Want to Get Started With Python for Machine Learning?

Take my free 7-day email crash course now (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

Dictionary Comprehension

Dictionary comprehension is similar to list comprehension, except now we have (key, value) pairs. Here is an example; we’ll modify each value of the dictionary by concatenating the string ‘number ‘ to each value:

Again, conditionals are also possible. We can choose to add (key, value) pairs based on a criterion in the new dictionary.

Enumerators and Zip in Python

In Python, an iterable is defined as any data structure that can return all its items, one at a time. This way, you can use a for loop to further process all items one by one. Python has two additional constructs that make for loops easier to use, i.e., enumerate() and zip().

Enumerate

In traditional programming languages, you need a loop variable to iterate through different values of a container. In Python, this is simplified by giving you access to a loop variable along with one value of the iterable object. The enumerate(x) function returns two iterables. One iterable varies from 0 to len(x)-1. The other is an iterable with a value equal to items of x. An example is shown below:

By default, enumerate starts at 0, but we can start at some other number if we specify it. This is useful in some situations, for example:

Zip

Zip allows you to create an iterable object of tuples. Zip takes as an argument multiple containers $(m_1, m_2, \ldots, m_n)$ and creates the i-th tuple by pairing one item from each container. The i-th tuple is then $(m_{1i}, m_{2i}, \ldots, m_{ni})$. If the passed objects have different lengths, then the total number of tuples formed has a length equal to the minimum length of passed objects.

Below are examples of using both zip() and enumerate().

Function Context

Python allows nested functions, where you can define an inner function within an outer function. There are some awesome features related to nested functions in Python.

  • The outer function can return a handle to the inner function.
  • The inner function retains all its environment and variables local to it and in its enclosing function even if the outer function ends its execution.

An example is given below, with an explanation in the comments.

Decorators in Python

Decorators are a powerful feature of Python. You can use decorators to customize the working of a class or a function. Think of them as a function applied to another function. Use the function name with the @ symbol to define the decorator function on the decorated function. The decorator takes a function as an argument, giving a lot of flexibility.

Consider the following function square_decorator() that takes a function as an argument and also returns a function.

  • The inner nested function square_it()takes an argument arg.
  • The square_it()function applies the function to arg and squares the result.
  • We can pass a function such as sin to square_decorator(), which in turn would return $\sin^2(x)$.
  • You can also write your own customized function and use the square_decorator() function on it using the special @ symbol as shown below. The function plus_one(x) returns x+1. This function is decorated by the square_decorator(), and hence we get $(x+1)^2$.

Generators in Python

Generators in Python allow you to generate sequences. Instead of writing a return statement, a generator returns multiple values via multiple yield statements. The first call to the function returns the first value from yield. The second call returns the second value from yield and so on.

The generator function can be invoked via next().Every time next() is called, the next yield value is returned. An example of generating the Fibonacci sequence up to a given number x is shown below.

Example of Data Generator in Keras

One use of a generator is the data generator in Keras. It is useful because we do not want to keep all data in memory but want to create it on the fly when the training loop needs it. Remember, in Keras, a neural network model is trained in batches, so a generator is to emit batches of data. The function below is from our previous post, “Using CNN for financial time series prediction“:

The function above is to pick a random row of a pandas dataframe as a starting point and clip the next several rows as a one-time interval sample. This process is repeated several times to collect many time intervals into one batch. When we collect enough interval samples, at the second to the last line in the above function, the batch is dispatched using the yield command. You may have already noticed that generator functions do not have a return statement. In this example, the function will run forever. This is useful and necessary because it allows our Keras training process to run as many epochs as we want.

If we do not use a generator, we will need to convert the dataframe into all possible time intervals and keep them in memory for the training loop. This will be a lot of repeating data (because the time intervals are overlapping) and take up a lot of memory.

Because it is useful, Keras has some generator function predefined in the library. Below is an example of the ImageDataGenerator(). We have loaded the cifar10 dataset of 32×32 images in x_train. The data is connected to the generator via the flow() method. The next() function returns the next batch of data. In the example below, there are 4 calls to next(). In each case, 8 images are returned as the batch size is 8.

Below is the entire code that also displays all images after every call to next().

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Python Documentation

Books

API Reference

Summary

In this tutorial, you discovered some special features of Python.

Specifically, you learned:

  • The purpose of list and dictionary comprehension
  • How to use zip and enumerate
  • Nested functions, function contexts, and decorators
  • Generators in Python and the ImageDataGenerator in Python

Do you have any questions about the Python features discussed in this post? Ask your questions in the comments below, and I will do my best to answer.

 

Get a Handle on Python for Machine Learning!

Python For Machine Learning

Be More Confident to Code in Python

...from learning the practical Python tricks

Discover how in my new Ebook:
Python for Machine Learning

It provides self-study tutorials with hundreds of working code to equip you with skills including:
debugging, profiling, duck typing, decorators, deployment, and much more...

Showing You the Python Toolbox at a High Level for
Your Projects


See What's Inside

, , , , , , , , ,

8 Responses to More Special Features in Python

  1. Avatar
    Gerard Elifuraha Mtalo, PhD December 18, 2021 at 8:11 pm #

    Dear Jason Brownlee (PhD) and About Mehreen Saeed,

    Thank you for the useful knowledge that you continue to share globally without reservation. I am a keen follower of your many tutorials.

    I have observed the following behavior in the use of the set() function and the zip() function.

    >>> shapes=zip(name,sides,colors)
    >>> shapes

    >>> b=set(shapes)
    >>> b
    {(‘Pentagon’, 5, ‘blue’), (‘Triangle’, 3, ‘red’), (‘Hexagon’, 6, ‘yellow’), (‘Square’, 4, ‘green’)}
    >>> print(b)
    {(‘Pentagon’, 5, ‘blue’), (‘Triangle’, 3, ‘red’), (‘Hexagon’, 6, ‘yellow’), (‘Square’, 4, ‘green’)}
    >>> shapes

    >>> b=set(shapes)
    >>> shapes

    >>> print(b)
    set()
    >>> b
    set()

    Can any one tell me why b becomes an empty set when I issue the assignment b=set(shapes) the second time?

    NOTE that the object shapes is still defined internally as

    • Avatar
      Adrian Tam December 19, 2021 at 1:57 pm #

      In Python 2 your code should produce what you want. But in Python 3, output of zip is a generator, which can only be read once. Hence if you run b=set(shapes) the second time, it will be empty.

  2. Avatar
    Gerard Elifuraha Mtalo, PhD December 18, 2021 at 8:22 pm #

    Dear Jason Brownlee and About Mehreen,

    I noticed that the internal representation of the variable ‘shapes’ which was ” filtered out in the comment that I sent earlier.

    I also needed to inform you that I run the code on Python 3.7.7 shell installed on Windows 10.

    Gerard Elifuraha Mtalo

  3. Avatar
    Anthony The Koala January 9, 2022 at 6:24 pm #

    Dear authorised answerer,

    I had a go at the section “Function Context”

    In the code you had two functions circle.

    Output:

    Questions please:
    I blocked the first circle function. What was the purpose of the first circle function? It was never called?

    Was the purpose of both the circle functions for function overloading?

    Thank you,
    Anthony of Sydney

    • Avatar
      Anthony The Koala January 10, 2022 at 2:07 am #

      To clarify, even if the first version of function “def circle(r)” the first function would never get invokedc.

      Overloading does not work in the same way as say another language:

      Output:

      Conclusions:
      * Obviously python does not allow overloading as demonstrated by the runtime error
      * But it does not answer why one can have two functions. Note that the first function does not get invoked at all

      Output showing that only the second function is called. The first function is never called/invoked.

      Thank you
      Anthony of Sydney

      • Avatar
        James Carmichael January 10, 2022 at 11:13 am #

        Thank you for the feedback and interest Anthony!

        • Avatar
          Anthony The Koala January 10, 2022 at 11:45 am #

          How does having two functions and only the 2nd function is invoked.

          Blocking out the 1st function by commenting it out does no harm.

          I don’t understand the point of this section of the tutorial.

          Thank you,
          Anthony of Sydney

          • Avatar
            Adrian Tam April 16, 2022 at 5:28 am #

            To understand the behavior, you need to remember that everything is an object in Python. This is the major difference to many other languages. When you define a function circle() and define it again, the later definition prevails as the name circle is reassigned to the new function object. Similarly, if you can return an integer from a function, you can also return a function since both integer and function are objects from Python’s point of view. They are not much difference.

            Therefore, in your code, it is not the 2nd function invoked. It is the latest definition of circle getting invoked.

Leave a Reply