[New Book] Click to get The Beginner's Guide to Data Science!
Use the offer code 20offearlybird to get 20% off. Hurry, sale ends soon!

How to Use and Remove Trend Information from Time Series Data in Python

Our time series dataset may contain a trend.

A trend is a continued increase or decrease in the series over time. There can be benefit in identifying, modeling, and even removing trend information from your time series dataset.

In this tutorial, you will discover how to model and remove trend information from time series data in Python.

After completing this tutorial, you will know:

  • The importance and types of trends that may exist in time series and how to identify them.
  • How to use a simple differencing method to remove a trend.
  • How to model a linear trend and remove it from a sales time series dataset.

Kick-start your project with my new book Time Series Forecasting With Python, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

  • Updated Apr/2019: Updated the link to dataset.
How to Use and Remove Trend Information from Time Series Data in Python

How to Use and Remove Trend Information from Time Series Data in Python
Photo by john78727, some rights reserved.

Trends in Time Series

A trend is a long-term increase or decrease in the level of the time series.

In general, a systematic change in a time series that does not appear to be periodic is known as a trend.

— Page 5, Introductory Time Series with R

Identifying and understanding trend information can aid in improving model performance; below are a few reasons:

  • Faster Modeling: Perhaps the knowledge of a trend or lack of a trend can suggest methods and make model selection and evaluation more efficient.
  • Simpler Problem: Perhaps we can correct or remove the trend to simplify modeling and improve model performance.
  • More Data: Perhaps we can use trend information, directly or as a summary, to provide additional information to the model and improve model performance.

Types of Trends

There are all kinds of trends.

Two general classes that we may think about are:

  • Deterministic Trends: These are trends that consistently increase or decrease.
  • Stochastic Trends: These are trends that increase and decrease inconsistently.

In general, deterministic trends are easier to identify and remove, but the methods discussed in this tutorial can still be useful for stochastic trends.

We can think about trends in terms of their scope of observations.

  • Global Trends: These are trends that apply to the whole time series.
  • Local Trends: These are trends that apply to parts or subsequences of a time series.

Generally, global trends are easier to identify and address.

Identifying a Trend

You can plot time series data to see if a trend is obvious or not.

The difficulty is that in practice, identifying a trend in a time series can be a subjective process. As such, extracting or removing it from the time series can be just as subjective.

Create line plots of your data and inspect the plots for obvious trends.

Add linear and nonlinear trend lines to your plots and see if a trend is obvious.

Removing a Trend

A time series with a trend is called non-stationary.

An identified trend can be modeled. Once modeled, it can be removed from the time series dataset. This is called detrending the time series.

If a dataset does not have a trend or we successfully remove the trend, the dataset is said to be trend stationary.

Using Time Series Trends in Machine Learning

From a machine learning perspective, a trend in your data represents two opportunities:

  1. Remove Information: To remove systematic information that distorts the relationship between input and output variables.
  2. Add Information: To add systematic information to improve the relationship between input and output variables.

Specifically, a trend can be removed from your time series data (and data in the future) as a data preparation and cleaning exercise. This is common when using statistical methods for time series forecasting, but does not always improve results when using machine learning models.

Alternately, a trend can be added, either directly or as a summary, as a new input variable to the supervised learning problem to predict the output variable.

One or both approaches may be relevant for your time series forecasting problem and may be worth investigating.

Next, let’s take a look at a dataset that has a trend.

Stop learning Time Series Forecasting the slow way!

Take my free 7-day email course and discover how to get started (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

Shampoo Sales Dataset

This dataset describes the monthly number of sales of shampoo over a 3 year period.

The units are a sales count and there are 36 observations. The original dataset is credited to Makridakis, Wheelwright, and Hyndman (1998).

Below is a sample of the first 5 rows of data, including the header row.

Below is a plot of the entire dataset, where you can learn more and download the dataset.

The dataset shows an increasing trend.

Shampoo Sales Dataset

Shampoo Sales Dataset

Load the Shampoo Sales Dataset

Download the dataset and place it in the current working directory with the filename “shampoo-sales.csv“.

The dataset can be loaded with a custom date parsing routine as follows:

Running the example loads the dataset and creates a plot.

Shampoo Sales Dataset Plot

Shampoo Sales Dataset Plot

Detrend by Differencing

Perhaps the simplest method to detrend a time series is by differencing.

Specifically, a new series is constructed where the value at the current time step is calculated as the difference between the original observation and the observation at the previous time step.

This has the effect of removing a trend from a time series dataset.

We can create a new difference dataset in Python by implementing this directly. A new list of observations can be created.

Below is an example that creates the difference detrended version of the Shampoo Sales dataset.

Running the example creates the new detrended dataset and then plots the time series.

Because no difference value can be created for the first observation (there is nothing for it to be subtracted from), the new dataset contains one less record. We can see that indeed the trend does appear to have been removed.

Shampoo Sales Dataset Difference Detrended

Shampoo Sales Dataset Difference Detrended

This approach works well for data with a linear trend. If the trend is quadratic (the change in the trend also increases or decreases), then a difference of the already-differenced dataset can be taken, a second level of differencing. This process can be further repeated if needed.

Because differencing only requires the observation at the previous time step, it can easily be applied to unseen out-of-sample data to either preprocess or provide an additional input for supervised learning.

Next, we will look at fitting a model to describe the trend.

Detrend by Model Fitting

A trend is often easily visualized as a line through the observations.

Linear trends can be summarized by a linear model, and nonlinear trends may be best summarized using a polynomial or other curve-fitting method.

Because of the subjective and domain-specific nature of identifying trends, this approach can help to identify whether a trend is present. Even fitting a linear model to a trend that is clearly super-linear or exponential can be helpful.

In addition to being used as a trend identification tool, these fit models can also be used to detrend a time series.

For example, a linear model can be fit on the time index to predict the observation. This dataset would look as follows:

The predictions from this model will form a straight line that can be taken as the trend line for the dataset. These predictions can also be subtracted from the original time series to provide a detrended version of the dataset.

The residuals from the fit of the model are a detrended form of the dataset. Polynomial curve fitting and other nonlinear models can also be used.

We can implement this in Python by training a scikit-learn LinearRegression model on the data.

Running the example first fits the linear model to the integer-indexed observations and plots the trend line (green) over the original dataset (blue).

Shampoo Sales Dataset Plot With Trend

Shampoo Sales Dataset Plot With Trend

Next, the trend is subtracted from the original dataset and the resulting detrended dataset is plotted.

Shampoo Sales Dataset Model Detrended

Shampoo Sales Dataset Model Detrended

Again, we can see that this approach has effectively detrended the dataset. There may be a parabola in the residuals, suggesting that perhaps a polynomial fit may have done a better job.

Because the trend model takes only the integer index of the observation as input, it can be used on new data to either detrend or provide a new input variable for the model.

Further Reading

Below are some additional resources on trend estimation and detrending in time series.

Summary

In this tutorial, you discovered trends in time series data and how to remove them with Python.

Specifically, you learned:

  • About the importance of trend information in time series and how you may be able to use it in machine learning.
  • How to use differencing to remove a trend from time series data.
  • How to model a linear trend and remove it from time series data.

Do you have any questions about detrending or about this tutorial?
Ask your questions in the comments below and I will do my best to answer.

Want to Develop Time Series Forecasts with Python?

Introduction to Time Series Forecasting With Python

Develop Your Own Forecasts in Minutes

...with just a few lines of python code

Discover how in my new Ebook:
Introduction to Time Series Forecasting With Python

It covers self-study tutorials and end-to-end projects on topics like: Loading data, visualization, modeling, algorithm tuning, and much more...

Finally Bring Time Series Forecasting to
Your Own Projects

Skip the Academics. Just Results.

See What's Inside

63 Responses to How to Use and Remove Trend Information from Time Series Data in Python

  1. Avatar
    Carmen Mardiros January 27, 2017 at 11:40 pm #

    Really useful Jason. One question. Would the de-trended values be used as additional features or instead of the original features. Also, is it common to de-trend the target variable as well? If yes, do the de-trended values become the new target variable? If yes, then what is the process for obtaining full predictions from ML models (for sklearn at least, I’m guessing a custom transformer needs to store the removed trend and add it back in using inverse_transform).

    Thank you!

    • Avatar
      Jason Brownlee January 28, 2017 at 7:49 am #

      Hi Carmen,

      The discussion in this post is for de-trending a univariate time series. If you wish to detrend a univariate time series, you would treat it as a suite of univariate time series and de-trend one at a time.

      The whole series may be de-trended and is used as input for a learning algorithm like ARIMA. See this post:
      https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/

      Alternatively, the detrended series may be used as an additional input for a machine learning algorithm.

  2. Avatar
    Giovanni May 24, 2017 at 7:28 pm #

    Very useful, thank you! I’m wondering if it is possible to give back the original trend to the detrended time series…do you know how I can do it? Thanks

    • Avatar
      Jason Brownlee June 2, 2017 at 11:32 am #

      Yes, if the trend was removed with subtraction, it can be added with addition.

    • Avatar
      Gabriele February 3, 2020 at 9:22 pm #

      I guess you can just start with the initial value of the original timeseries, and then recreate the original timeseries by adding each delta obtained with the differentiation.

  3. Avatar
    Marianico July 21, 2017 at 10:52 pm #

    Would detrending technique improve ARIMA forecasting?

    • Avatar
      Jason Brownlee July 22, 2017 at 8:35 am #

      It can. Try it and see. ARIMA has it built in with the q var.

      • Avatar
        Aman September 23, 2022 at 2:45 pm #

        You mean the var ‘d’ ?

        • Avatar
          James Carmichael September 24, 2022 at 6:46 am #

          Hi Aman…Please clarify your question so that we may better assist you.

  4. Avatar
    Ahmed August 11, 2017 at 12:55 pm #

    Great tutorial! really thanks for such useful information. My question is, Is it possible to have linear and non-linear trend added to the same data set? If yes, Shall I remove the linear part first, then the non-linear part (which is a straight line fitted to the data and then subtract it) and then remove the non-linear part (fit a low-order polynomial to the signal and then subtract it)? Or just removing the non-linear part would be sufficient ??

    • Avatar
      Jason Brownlee August 12, 2017 at 6:46 am #

      Interesting question. Generally, I would recommend trying both and see which model results in better forecasts.

  5. Avatar
    Weber August 12, 2017 at 9:50 pm #

    Suppose the time series shows downward trend most of the times then subtracting from previous time stamp may result in a negative value. Perhaps simple moving averages can be used in this case..?

    • Avatar
      Jason Brownlee August 13, 2017 at 9:53 am #

      Perhaps. But what is wrong with negative values – you’re working in a transformed space. You can always reverse the transform on the prediction to get back to the original scale.

      • Avatar
        Weber August 13, 2017 at 7:00 pm #

        Thanks for your reply. This technique would fail for Poisson Auto regression model. Do you have any suggestions to detrending in event count models?

  6. Avatar
    Mamta August 21, 2017 at 4:35 am #

    Dear Jason,
    I followed your approach to detrend the time series data before fitting a neural network model on the training set.

    I need to forecast hourly values for next day.

    When I have to use my trained model to predict on unseen data, how do I deal with the trend ?

    Do I have to predict the trend along with the the usual observations, as my model is trained on detrended data? Should this predicted trend be added to the predicted output of the trained model ?

    Thanks
    Mamta

  7. Avatar
    Tushar February 8, 2018 at 11:14 pm #

    Dear Jeson

    Sir, I have a time series data obtained from integration of another data. The nature of data is exponentially decaying in amplitude with time. Would you please suggest me the process to obtain (identify) linear or non-linear trend (Equation of poly nomial) present in the derived time series data (i.e. obtained from integration of another data)?

    Thanks

    • Avatar
      Jason Brownlee February 9, 2018 at 9:09 am #

      I don’t have an example. Perhaps you can use a power transform on the obs to stabilize the changing variance, then try removing the cycle?

      Or perhaps you can use an scipy function call to model the exponential trend and remove it manually?

      Perhaps explore a few approaches and see what looks good.

  8. Avatar
    Faiza March 16, 2018 at 12:58 am #

    Dear Jason

    Sir plzz give me any example i have a data of time series in 4 quarters.how i solve this “assuming the multiplicative model and using the movig average method,obtain the detrended time seies”??????

    • Avatar
      Jason Brownlee March 16, 2018 at 6:20 am #

      Sorry, I cannot write code for you. Perhaps hire a contractor?

  9. Avatar
    Rahul May 21, 2018 at 7:28 pm #

    Hi Jason,

    Really useful tutorial. Just wanted to inform you that Detrending Notes, GEOS 585A, Applied Time Series Analysis [PDF] in Further Reading section is not working. If possible, could you fix it please?

    Regards,
    Rahul Sathyajit

    • Avatar
      Jason Brownlee May 22, 2018 at 6:27 am #

      Thanks, I’ve added a link to the page to learn more.

  10. Avatar
    hamid August 16, 2018 at 8:47 pm #

    Jason,
    Thank you for your incredible post.
    So, in one of your LSTM posts, you used multivariate weather input features (x1,x2,x3,x4) to predict another weather parameter (Y). Should I also use detrending techniques for the input features which also have trends ( for example, suppose x1 and x4 have trend) ? If yes, I don’t feel confident about my results, as I’ve changed my input data to a new one. If no, maybe my model cannot predict well. What am I missing?

    • Avatar
      Jason Brownlee August 17, 2018 at 6:27 am #

      Ideally yes. LSTMs are sensitive to trends and seasonality.

  11. Avatar
    HAMID August 21, 2018 at 6:31 am #

    Hello Jason

    What is the best way to select the optimal subset(rows) to be trained by LSTM?

    regards
    Salah

  12. Avatar
    Jakub Kajan October 18, 2018 at 3:59 am #

    Hello Jason.
    Thank you very much for great tutorial. I tried to apply diff to data via pandas diff, but with diff data model was not able to converge, I tried different topologies, neurons in layers etc. but with almost same result. But when I don’t modify data, fit is really good. Have you ever meet such issue, or could you know some good reading related to this topic?

    Regards Jakub.

    • Avatar
      Jason Brownlee October 18, 2018 at 6:37 am #

      Yes, MLPs and CNNs seem to perform well even better when the data is non-stationary, at least in a few tests I performed.

  13. Avatar
    Theekshana November 7, 2018 at 7:38 pm #

    Hi Janson,

    Is there any well-known method for identifying the time series trend. As an example given a time series, the method should be able to give a value that says the series increasing, decreasing or stabilized (the value should represent the trend perfectly. some kind of a scale maybe 0-100, 0 for decreasing 100 for increasing at a higher rate, 50 for stable etc).

    From your example, I can see that the fitted line can be used to come up with some value about the trend ( maybe using the gradient ). Is there any sophisticated way to do this?

    Thank you.
    Great tutorial. Big fan of this site…

  14. Avatar
    S February 2, 2019 at 9:52 pm #

    thanks for the tutorial!

    Question – once the trend has been removed (as you have in your last image), one could use that detrended data to make some predictions with ARIMA. How would one interpret the predicted value? Let us assume it goes up by 2 ‘units’ in the next period = could expect an increase of 2 x the scale that was used in the original data set?

    • Avatar
      Jason Brownlee February 3, 2019 at 6:16 am #

      The trend can be added back to any predictions made and the interpretation would then be specific to the domain.

  15. Avatar
    Afreen February 14, 2019 at 9:47 pm #

    Hi Jason,
    Thank you for this tutorial!
    So, I am working on a problem where I need to incorporate time-independent features to make predictions for target variable that is time dependent.
    I detrend the target (time-dependent) and I build a supervised ML model with features as the time-independent features. To the prediction of this model, I add back the trend.
    Would this be the right way to go about it?

    Does any of your other tutorial talk about it? Searched for it but couldn’t find it.

    Thanks a lot for your help!

    • Avatar
      Jason Brownlee February 15, 2019 at 8:05 am #

      Perhaps add the new variables as exog variables for a time step.

      Otherwise, perhaps use an ensemble model that combined predictions from a model on the time series and a model no the static data?

  16. Avatar
    Leen March 16, 2019 at 10:59 pm #

    Hello Jason,

    Thank you for the awesome tutorial!!

    I wanted to ask if we are to detrend and deseasonalize, van’t we just use the seasonal_decompose() function, and extract the trend and seasonality components from it and leave the residual ?? Is the residual considered detrended and deseasonalized ?

    Example
    result = seasonal_decompose(self.series, model=’additive’, freq=frequency)
    residual = result.resid # this is what I am reffering to.

    Thanks in advance

    • Avatar
      Jason Brownlee March 17, 2019 at 6:22 am #

      Perhaps. It may not be reliable for your data. It is better to have more control and understand the structure of each.

      • Avatar
        Leen March 17, 2019 at 6:48 am #

        So which is more reliable ? detrending and deseasonalizing or just extrating residual from seasonal_decompose()

        • Avatar
          Jason Brownlee March 18, 2019 at 6:01 am #

          It depends on your project, in general I recommend detrending and seasonal adjustment prior to modeling or as part of modeling in the case of methods like SARIMA.

  17. Avatar
    deepika March 20, 2019 at 10:15 pm #

    Thanks for this post, Jason!
    I was wondering, if I am performing an LSTM with multiple variables, and I observe a trend only for 2 out of the 5 variables,
    1. do I detrend all 5 variables by differencing?
    2. If one of them is categorical and I have coded it as(1,-1) does this have to be differnced as well?

    Thanks a lot!

    • Avatar
      Jason Brownlee March 21, 2019 at 8:12 am #

      Try modeling the data directly, then with differencing on the series with trends to see if there is an impact.

      For categorical vars, explore integer encode, one hot encode and perhaps embeddings.

  18. Avatar
    Teyang July 8, 2019 at 6:22 pm #

    Hi Jason,

    Great post!

    Can I check with you regarding the model fitting method.

    When you mentioned subtracting the trend from the original data, is it the same as getting the residuals of the linear/quadratic/polynomial model?

    Thanks!

  19. Avatar
    Laura August 23, 2019 at 12:38 am #

    Hi Jason,
    If the goal is to forecast the future N timesteps of a timeseries showing a stochastic trend, then we should fit a model to approximate the trend, remove this approximated trend from the original data and use models such as ARIMA to predict there residuals. However, the final forecast will be Arima forecast + trend forecast.

    The timeseries data I work with is not well approximated by a linear regression, it consists of random patterns with the trend going up and down at different amplitudes (like a random walk). I didn’t find yet a model which can approximated it well enough. What model do you recommend using for this kind of stochastic trend approximation?
    Thanks!

  20. Avatar
    Ha My Nguyen February 29, 2020 at 5:42 pm #

    Hi, I run the code on Python and I dont know why this function kept showing error datetime.strptime. Is that possible to write in another way?

  21. Avatar
    Nirmal June 24, 2020 at 1:09 am #

    Dear Jason
    How can we use non linear regression for timeseries data detrending?
    What is local regression for data detrending?
    How can we use quadratic polynomial trend for detrending?
    Please clarify me this.
    Which software can be used for these all?

    • Avatar
      Jason Brownlee June 24, 2020 at 6:36 am #

      You can use scipy to implement each, perhaps the above example can provide a useful starting point for your experiments.

  22. Avatar
    Laila July 4, 2020 at 5:06 am #

    Hi Jason,

    It seems like a stupid question but I did detrend my dataset according to your instructions and it worked out fine. However, I have trouble interpreting the detrended version mainly because the y-values are now negative and around zero. In my dataset i plot CO2-conc over time.
    How can I have negative values for co2-conc?? I get that the data is transformed but I don’t know what it really means for my measured values.

    I’d be very happy to hear from you.
    Bests
    Laila

    • Avatar
      Jason Brownlee July 4, 2020 at 6:06 am #

      Yes, you must invert the transform on any predictions so that you can interpret the result.

      I believe the example in this post will help with inverting the process:
      https://machinelearningmastery.com/machine-learning-data-transforms-for-time-series-forecasting/

      • Avatar
        Laila July 7, 2020 at 6:53 pm #

        I think I mistakenly used the notion transformed. I have not done any power transformation or similar with my data before I applied the linear model. I took the raw data, fitted the lm and then looked at the residuals plot and the data are partially negative and around 0. For example for the year 2010, I had 22.000 t co2 in my raw data and the value for the same year in my detrended data (residuals of lm) amounts to – 500. My question is if the detrended data as it is now, can be used e.g for a calculation of the expected value in decade 2010-2040.

        Thank you very much for your time and quick reply!

  23. Avatar
    Aji October 6, 2020 at 9:14 pm #

    Hi Jason,
    I have a question on detrending for LSTM models. I removed the trend and seasonality from the input data using seasonal_decompose function. However, when I input this detrended data to LSTM endoder decoder model, training does not happen. The training loss and val_loss do not reduce from the first epoch. On the other hand, when I supply the normal data ( i.,e, without detrending), the model gets trained properly.

    I am wondering if some other tweeks are needed to train on a detrended data using LSTM. Any thoughts / suggestions are appreciated.

    Rgds,
    Aji

    • Avatar
      Jason Brownlee October 7, 2020 at 6:45 am #

      You may need to tune the model architecture and learning hyperparameters to your prepared data.

  24. Avatar
    Devansh Popat November 4, 2020 at 7:24 pm #

    Hi, My question is, How do we use the de-trended data/plot for building a model and forecasting? It seems we are changing the data-set here so our forecast by the model will be some undefined value. How do we get back to the data-set value (Sales/Joins/Lands/Clicks/etc) from the forecasted value?

    • Avatar
      Jason Brownlee November 5, 2020 at 6:32 am #

      You can remove a trend by using a differencing transform – as described in the above tutorial.

  25. Avatar
    M. Mossad March 22, 2022 at 6:42 pm #

    Hello, this is a very informative article, may I ask how do you add a trend though? let’s say you’re simulating periodic data and want to test spectral analysis methods on trended data, how do you add a linear trend to make your simulated data look like the data in this article. Thanks in advance.

  26. Avatar
    Senthilkumar Radhakrishnan August 22, 2022 at 11:03 pm #

    Hi Jason,
    Can we remove trend alone from the time series data, without affecting the seasonal pattern.
    I have tried using differencing as well as removing trend section from seasonal decomposing techniques but they still remove the seasonal bump in the plot.
    What can be done to remove the trend .

    Thanks in advance ,
    really appreciate your work.

  27. Avatar
    neha December 28, 2022 at 8:54 pm #

    HI Jason,

    Really helpful post. I am working on multivariate problem where I am predicting Sales of a item. I want to include trend as a input in this supervised problem. How I can do that? How I can capture trend for Sales and add it as input for my model?

    Thanks in advance,

  28. Avatar
    nehapawar December 28, 2022 at 8:56 pm #

    HI Jason,

    Really helpful post. I am working on multivariate problem where I am predicting Sales of a item. I want to include trend as a input in this supervised problem. How I can do that? How I can capture trend for Sales and add it as input for my model?

    Thanks in advance,

    • Avatar
      James Carmichael December 29, 2022 at 8:46 am #

      Hi nehapawar…For this purpose you could leave the original time-series data with trend information. That is…simply do not remove it.

Leave a Reply