Manipulating Tensors in PyTorch

Last Updated on January 23, 2023

PyTorch is a deep learning library. Just like some other deep learning libraries, it applies operations on numerical arrays called **tensors**. In the simplest terms, tensors are just multidimensional arrays. When we are dealing with the tensors, there are some operations that are used very often. In PyTorch, there are some functions defined specifically for dealing with tensors.

In the following, we will have a brief overview of what PyTorch provides on tensors and how we can use them. After finishing this tutorial, you will know:

  • How to create and operate on PyTorch tensors
  • PyTorch’s tensor syntax is similar to NumPy
  • The common functions you can use from PyTorch to manipulate a tensor

Let’s get started.

Manipulating Tensors in PyTorch. Photo by Big Dodzy. Some rights reserved.

Overview

This tutorial is in three parts; they are:

  • Creating Tensors
  • Checking a Tensor
  • Manipulating Tensors
  • Tensor Functions

Creating Tensors

If you’re familiar with NumPy, you should recall that there are multiple ways of creating an array. Same is true in PyTorch for creating tensors. The simplest way to create a specific constant matrix like the following:

$$
\begin{bmatrix}
1 & 2 & 3 \\
4 & 5 & 6
\end{bmatrix}
$$

is the following:

It prints:

The dtype argument is to specify the data type of the values in the tensor. It is optional. You can also provide the values from a NumPy array and convert it to PyTorch tensor.

Usually, you would create a tensor for some specific purpose. For example, you want to have 10 values evenly distributed between -1 and 1, you can use the linspace() function:

It prints:

However, if you would like to have a tensor of random values (which is very useful in testing your functions), you can make one like the following:

It prints, for example:

This resulting tensor is of dimension $3\times 4$ and each value is uniformly distributed between 0 and 1. If you would like to have the values normally distributed, just change the function to randn():

If you want to have the random values to be integer, e.g., between 3 to 10, you can use the randint() function:

This will give, for example:

The values are in the range $3 \le x < 10$. By default the lowerbound is zero, so if you want the values to be $0 \le x < 10$, you can use:

The other commonly used tensors are the zero tensor and tensors with all values the same. To create a zero tensor (e.g., of dimension $2\times 3\times 4$), you can do:

It prints:

and to create a tensor of all values are 5, we can do

It prints:

But if you want all values are one, there is a simpler function:

Finally, if you want an identity matrix, you can get it with diag() or eye():

It prints:

Checking a Tensor

Once you have a tensor and you want to know more about it, you can simply print it to the screen using print(). But if the tensor is too big, it is easier to show its dimension by checking its shape:

It prints:

The shape of a tensor can be accessed using the shape property or the size() function. If you want to see how many dimensions you have (i.e., $2\times 3\times 4$ is 3 and $3\times 4$ is 2), you can read the ndim property:

This will give you “3”. If you use len() to check a tensor, it will only give you the size of the first dimension, e.g.,

It prints

Another property that we would like to learn about a tensor is its data type. Usually we use floating points in deep learning but sometimes the tensors should be in integers (e.g., in image as pixel values). To check the data type, you can read the dtype property:

It prints

If you would like to change the data type, you can recreate the tensor with a new type:

The above prints:

Manipulating Tensors

One common operation on tensors in deep learning is to change the tensor shape. For example, you may want to convert a 2D tensor into 1D or add a dummy dimension to a tensor. You may also want to extract a sub-tensor from a larger tensor.

For example, we create a tensor like the following:

Which if we get:

We can take a slice using the same syntax as in NumPy:

which will be:

or

which will be

You can also make use of the same slicing syntax to add a new dimension. For example,

You will see

Here you use None to insert a new dimension at a specific place. This is useful if, for example, need to convert an image into a batch of only one image. If you’re familiar with NumPy, you may recall there is a function expand_dims() for this purpose but PyTorch doesn’t provide it. The similar function is unsqueeze(), which is demonstrated below:

which prints:

One powerful nature of NumPy slicing syntax is boolean indexing. This is also supported with PyTorch tensors. For example:

You may see:

The above selects the columns that all elements are greater than -1. We can also manipulate the tensor by selecting specific columns:

which is

To convert a 2D tensor into 1D, we can use

The result will be

You may also use reshape() function to achieve the same:

which the result should be same as that of ravel(). But usually, reshape() function is for more complicated target shapes:

This will print:

One common case of reshaping tensors is to do matrix transpose. For a 2D matrix, it is easily done in the same way as NumPy:

which prints:

But the transpose() function in PyTorch requires you to specify which axes to swap explicitly:

This result is same as above. If you have multiple tensors, you can combine them by stacking (vstack() for vertically and hstack() for horizontally). For example:

This may print:

The concatenate function is similar:

You will get the same tensor:

The reverse is to split, e.g.,

It prints

This function tells how many tensors to split into, rather than what size is each tensor. The latter is indeed more useful in deep learning (e.g., to split a tensor of a large dataset into many tensors of small batches). The equivalent function would be:

This should give you the same result as before. Which split(c, 3, dim=0) means to split on dimension 0 such that each resulting tensor will be of size 3.

Tensor Functions

PyTorch tensors can be treated as arrays. So you can often use it the similar way as NumPy arrays. For example, you have the functions of common mathematical functions:

This prints:

Note that, if a function is undefined (e.g., square root of negative numbers), nan will be resulted but no exception will be raised. In PyTorch, we have a function to check if values of a tensor is nan:

You will get:

Indeed, besides these defined functions, the Python operators can be applied to the tensors too:

You get:

But among the operators, the matrix multiplications are very important in deep learning. We can do so with:

This prints

These two are the same. Indeed, the @ operator from Python can also be used for vector dot-product, e.g.

It prints:

If you treat the values in a tensor as samples, you may also want to find some statistics about it. Some are provided in PyTorch, too:

It prints:

But for linear algebra functions, you should find it in PyTorch’s linalg submodule. For example:

You will see:

And specifically for convolution neural networks, padding a tensor is done with the following:

This prints:

This example of pad() function is to create (1,1) padding on dimension 0 and (0,2) on dimension 1. In other words, for each dimension 0 (rows), we add one dummy value (0) at the beginning and at the end. For each dimension 1 (columns), we add zero dummy values at the beginning but two dummy values at the end.

Finally, since PyTorch tensors can be considered as arrays, you can use them directly with other tools such as matplotlib. Below is an example of plotting a surface using PyTorch tensors:

The meshgrid produced the xx tensor as:

and the plot created is:

Summary

In this tutorial, you discovered how to manipulate PyTorch tensors. Specifically you learned:

  • What is a tensor
  • How to create various kinds of tensors in PyTorch
  • How to reshape, slice, and manipulate tensors in PyTorch
  • The common functions that can be applied to PyTorch tensors

No comments yet.

Leave a Reply