There is no denying that calculus is a difficult subject. However, if you learn the fundamentals, you will not only be able to grasp the more complex concepts but also find them fascinating. To understand machine learning algorithms, you need to understand concepts such as gradient of a function, Hessians of a matrix, and optimization, etc. The concept of limits and continuity serves as a foundation for all these topics.
In this post, you will discover how to evaluate the limit of a function, and how to determine if a function is continuous or not.
After reading this post, you will be able to:
Determine if a function f(x) has a limit as x approaches a certain value
Evaluate the limit of a function f(x) as x approaches a
Determine if a function is continuous at a point or in an interval
Let’s get started.
Tutorial Overview
This tutorial is divided into two parts
Limits
Determine if the limit of a function exists for a certain point
Compute the limit of a function for a certain point
Formal definition of a limit
Examples of limits
Left and right hand limits
Continuity
Definition of continuity
Determine if a function is continuous at a point or within an interval
Examples of continuous functions
A Gentle Introduction to Limits and Continuity Photo by Mehreen Saeed, some rights reserved.
A Simple Example
Let’s start by looking at a simple function f(x) given by:
f(x) = 1+x
What happens to f(x) near -1?
We can see that f(x) gets closer and closer to 0 as x gets closer and closer -1, from either side of x=-1. At x=-1, the function is exactly zero. We say that f(x) has a limit equal to 0, when x approaches -1.
Extend the Example
Extending the problem. Let’s define g(x):
g(x) = (1-x^2)/(1+x)
We can simplify the expression for g(x) as:
g(x) = (1-x)(1+x)/(1+x)
If the denominator is not zero then g(x) can be simplified as:
g(x) = 1-x, if x ≠ -1
However, at (x = -1), the denominator is zero and we cannot divide by zero. So it looks like there is a hole in the function at x=-1. Despite the presence of this hole, g(x) gets closer and closer to 2 as x gets closer and closer -1, as shown in the figure:
This is the basic idea of a limit. If g(x) is defined in an open interval that does not include -1, and g(x) gets closer and closer to 2, as x approaches -1, we write this as:
lim(x→-1) g(x) = 2
In general, for any function f(x), if f(x) gets closer and closer to a value L, as x gets closer and closer to k, we define the limit of f(x) as x approaches k, as L. This is written as:
Left and Right Hand Limits
For the function g(x), it doesn’t matter whether we increase x to get closer to -1 (approach -1 from left) or decrease x to get closer to -1 (approach -1 from right), g(x) still gets closer and closer to 2. This is shown in the figure below:
This gives rise to the notion of one-sided limits. The left hand limit is defined on an interval to the left of -1, which does not include -1, e.g., (-1.003, -1). As we approach -1 from the left, g(x) gets closer to 2.
Similarly, the right hand limit is defined on an open interval to the right of -1 and does not include -1, e.g., (-1, 0.997). As we approach -1 from the right, the right hand limit of g(x) is 2. Both the left and right hand limits are written as follows:
We say that f(x) has a limit L as x approaches k, if both its left and right hand limits are equal. Therefore, this is another way of testing whether a function has a limit at a specific point, i..e,
Formal Definition of a Limit
In mathematics, we need to have an exact definition of everything. To define a limit formally, we’ll use the the notion of the Greek letter ????. The mathematics community agrees to use ???? for arbitrarily small positive numbers, which means we can make ???? as small as we like and it can be as close to zero as we like, provided ????>0 (so ???? cannot be zero).
The limit of f(x) is L as x approaches k, if for every ????>0, there is a positive number ????>0, such that:
if 0<|????−????|<???? then |????(????)−????|<????
The definition is quite straightforward. x-k is the difference of x from k and |x-k| is the distance of x from k that ignores the sign of the difference. Similarly, |f(x)-L| is the distance of f(x) from L. Hence, the definition says that when the distance of x from k approaches an arbitrary small value, the distance of f(x) from L also approaches a very small value. The figure below is a good illustration of the above definition:
Examples of Limits
The figure below illustrates a few examples, which are also explained below:
1.1 Example with Absolute Value
f_1(x) = |x|
The limit of f_1(x) exists at all values of x, e.g., lim(x→0) f_1(x) = 0.
1.2 Example with a Polynomial
f_2(x) = x^2 + 3x + 1
The limit of f_2(x) exists for all values of x, e..g, lim(x→1) f_2(x) = 1+3+1 = 5.
1.3 Example with Infinity
f_3(x) = 1/x, if x>0
f_3(x) = 0, if x<=0
For the above as x becomes larger and larger, the value of f_3(x) gets smaller and smaller, approaching zero. Hence, lim(x→∞) f_3(x) = 0.
Example of Functions that Don’t have a Limit
From the definition of the limit, we can see that the following functions do not have a limit:
2.1 The Unit Step Function
The unit step function H(x) is given by:
H(x) = 0, if x<0
H(x) = 1, otherwise
As we get closer and closer to 0 from the left, the function remains a zero. However, as soon as we reach x=0, H(x) jumps to 1, and hence H(x) does not have a limit as x approaches zero. This function has a left hand limit equal to zero and a right hand limit equal to 1.
The left and right hand limits do not agree, as x→0, hence H(x) does not have a limit as x approaches 0. Here, we used the equality of left and right hand limits as a test to check if a function has a limit at a particular point.
2.2 The Reciprocal Function
Consider h_1(x):
h_1(x) = 1/(x-1)
As we approach x=1 from the left side, the function tends to have large negative values. As we approach x=1, from the right, h_1(x) increases to large positive values. So when x is close to 1, the values of h_1(x) do not stay close to a fixed real value. Hence, the limit does not exist for x→1.
2.3 The Ceil Function
Consider the ceil function that rounds a real number with a non-zero fractional part to the next integer value. Hence, lim(x→1) ceil(x) does not exist. In fact ceil(x) does not have a limit at any integer value.
All the above examples are shown in the figure below:
Continuity
If you have understood the notion of a limit, then it is easy to understand continuity. A function f(x) is continuous at a point a, if the following three conditions are met:
f(a) should exist
f(x) has a limit as x approaches a
The limit of f(x) as x->a is equal to f(a)
If all of the above hold true, then the function is continuous at the point a. Some examples follow:
Examples of Continuity
The concept of continuity is closely related to limits. If the function is defined at a point, has no jumps at that point, and has a limit at that point, then it is continuous at that point. The figure below shows some examples, which are explained below:
3.1 The Square Function
The following function f_4(x) is continuous for all values of x:
f_4(x) = x^2
3.2 The Rational Function
Our previously used function g(x):
g(x) = (1-x^2)/(1+x)
g(x) is continuous everywhere except at x=-1.
We can modify g(x) as g*(x):
g*(x) = (1-x^2)/(1+x), if x ≠ -1
g*(x) = 2, otherwise
Now we have a function that is continuous for all values of x.
3.3 The Reciprocal Function
Going back to our previous example of f_3(x):
f_3(x) = 1/x, if x>0
f_3(x) = 0, if x<=0
f_3(x) is continuous everywhere, except at x=0 as the value of f_3(x) has a big jump at x=0. Hence, there is a discontinuity at x=0.
Further Reading
This section provides more resources on the topic if you are looking to go deeper. Math is all about practice, and below is a list of resources that will provide more exercises and examples on this topic.
Please see the introduction: “To understand machine learning algorithms, you need to understand concepts such as gradient of a function, Hessians of a matrix, and optimization, etc. The concept of limits and continuity serves as a foundation for all these topics.”
Congratulations for the post!
I was wondering obout how those concepts are related to machine learning algorithms in practice? . Can you help me?
Best regards!
Please see the introduction: “To understand machine learning algorithms, you need to understand concepts such as gradient of a function, Hessians of a matrix, and optimization, etc. The concept of limits and continuity serves as a foundation for all these topics.”