LSTM Time Series Forecasting Tutorial in Python
ฝัง
- เผยแพร่เมื่อ 3 ต.ค. 2021
- Thank you for watching the video! Here is the Colab Notebook: colab.research.google.com/dri...
I offer 1 on 1 tutoring for Data Structures & Algos, and Analytics / ML! Book a free consultation here: calendly.com/greghogg/30min
Learn Python, SQL, & Data Science for free at mlnow.ai/ :)
Subscribe if you enjoyed the video!
Best Courses for Analytics:
---------------------------------------------------------------------------------------------------------
+ IBM Data Science (Python): bit.ly/3Rn00ZA
+ Google Analytics (R): bit.ly/3cPikLQ
+ SQL Basics: bit.ly/3Bd9nFu
Best Courses for Programming:
---------------------------------------------------------------------------------------------------------
+ Data Science in R: bit.ly/3RhvfFp
+ Python for Everybody: bit.ly/3ARQ1Ei
+ Data Structures & Algorithms: bit.ly/3CYR6wR
Best Courses for Machine Learning:
---------------------------------------------------------------------------------------------------------
+ Math Prerequisites: bit.ly/3ASUtTi
+ Machine Learning: bit.ly/3d1QATT
+ Deep Learning: bit.ly/3KPfint
+ ML Ops: bit.ly/3AWRrxE
Best Courses for Statistics:
---------------------------------------------------------------------------------------------------------
+ Introduction to Statistics: bit.ly/3QkEgvM
+ Statistics with Python: bit.ly/3BfwejF
+ Statistics with R: bit.ly/3QkicBJ
Best Courses for Big Data:
---------------------------------------------------------------------------------------------------------
+ Google Cloud Data Engineering: bit.ly/3RjHJw6
+ AWS Data Science: bit.ly/3TKnoBS
+ Big Data Specialization: bit.ly/3ANqSut
More Courses:
---------------------------------------------------------------------------------------------------------
+ Tableau: bit.ly/3q966AN
+ Excel: bit.ly/3RBxind
+ Computer Vision: bit.ly/3esxVS5
+ Natural Language Processing: bit.ly/3edXAgW
+ IBM Dev Ops: bit.ly/3RlVKt2
+ IBM Full Stack Cloud: bit.ly/3x0pOm6
+ Object Oriented Programming (Java): bit.ly/3Bfjn0K
+ TensorFlow Advanced Techniques: bit.ly/3BePQV2
+ TensorFlow Data and Deployment: bit.ly/3BbC5Xb
+ Generative Adversarial Networks / GANs (PyTorch): bit.ly/3RHQiRj
I offer 1 on 1 tutoring for Data Structures & Algos, and Analytics / ML! Book a free consultation here: calendly.com/greghogg/30min
hey link doesn't work. i copied the code exactly and not working please help
Wow the amount of flexibility on the communication is huge, great skill and of course the LSTM skills and python are perfect, can’t say enough thanks Greg.
Glad to hear it!!
Your video is amazing, man. Thank you so much. I was having problems understanding time series forecasting but you just made everything so clear and easy to understand. Again, thank you😁😁
Thank you Greg, you cleared my fear of deep learning.
great video! Thanks for sharing. One comment, hard coding `i+5` in the `df_to_X_f` function will lead to some very unexpected results if the `window_size` is not set to 5. Better would be to use the `window_size` variable here to ensure the slice is always the same as the window. Cheers :)
Oops, good catch yes I definitely meant window size, thanks!
@@GregHogg I'm a complete newbie trying to complete his capstone project in a bootcamp. Knowing that you could make that kind of mistake that even I saw really is going to help me with my imposter syndrome moving forward. Also, 11/10 video
Thank you Greg, perfect explanation and presentation!
Buddy this is the best explanation I have seen on this topic. You are an excellent communicator, if you're not a professor or instructor you should be.
Glad to hear it!
excellent video and covered all my doubts, thanks Greg...
Greg, this has been a great help. Even after 1 year, this remains relevant and super useful. Love it ✅
Glad to hear it! :)
You know @@GregHogg I saw this and right now I'm trying to apply this in different scenarios. I'm failing to get it right and I am hoping to reach you. Do you have an instagram? LinkedIn?
Very well explained! Thanks for sharing
Beautiful tutorial.
I would kindly like to get your attention towards some bad practice when using the Numpy skills.
You can use np.fromfunction( lambda x, y: x+y, shape=(n_windows, window_size) ) to create a matrix of indices and then slicing the df_as_np with the indices matrix to create X. Then you can use np.reshape to convert it to (n_windows, window_size, 1) shape. That avoids any explicit iterations (for loops) and allows numpy's backend to perform parallelism if possible.
Overall very good and clear tutorial!
you sir are a thousands times better than my professor appreciate it
Good explanation overall for the concept. However, some quick suggestions to make the videos a bit more comprehensive is to explain the key details like what is a tensor and why did you design a 3-D matrix for the model, or the part where you flatten the predictions; instead of just saying that the output will have extra dimensions run the code and exhibit it and then apply flatten.
Given this, really like your work to help people like me understand Machine Learning from the very basics. 😊
I really appreciate feedback like this. Very specific on important ideas like the tensor. I will keep this in mind for sure, and I appreciate your kind words!
There was not even a single reason why I should leave without subscribing , leaving this comment and liking the video , what an explanation , what a model 🤗
Very nice of you to say! Thank you greatly for the support :)
Hi! I have a question with respect to the 20:07 part where you define the 'linear' function, in case I'm using a variable that doesn't take negatives like the price of an action which one would you recommend? Thank you :)
Many thanks for this video. Can we use this method for forecasting univariate time series that involve outliers? If so, how to treat the outliers first?
Thanks Greg, for the very informative content on LSTM. It's been great help!
You're very welcome Adithya!
I have a question. How do I line up the Dates with the corresponding actual values and predicted values? It seems like the lengths don't matchup
Thanks for the video! Exactly what I needed
Really glad to hear that!
Hello congrats amazing video!!
What should we do differently to consider more variables as X?
Hi Howell, thank you for sharing your valuable information through this channel. I am one of the new followers of time series. If possible, could you create a series on how to implement Transformers on time series data, covering both univariate and multivariate approaches? Focusing on operations like forecasting, classification, or anomaly detection-just one of these would be greatly appreciated. There are no videos available on TH-cam that have implemented this before. It would be extremely helpful for students and new researchers in the field of time series.
Thank you Greg very much for the course! I have one question: if I want to forecast the temperature at 06:00 based on not one X(parameter) but 10 Xs (parameters) from 00:00-05:00(sliding window of 5) how should I tweak my df_to_X_y function?
This video was really helpful with implementing an LSTM in tensorflow! A lot of sources talk about it as either theoretical or building a toy one from scratch. Nice to actually see a tensorflow workflow used. When you are dealing with n-featured examples in the time series, how would you set up the model layers? For example: lets say you used barometric pressure in addition to temperature so your training matrix now looks like : [[ [x11, x12], [x21, x22], ... [xn1, xn2] ]] where x1 is the first entry in the series, x11 is the first feature of the first entry, and n is the window size. How would you set up the model layers? you would be dealing with a 3D matrix, where the third dimension is another feature matrix of window size n. Would the initial input layer just be a tuple of (5, 2)?
edit:
I just realized you have a whole other video on this so I will watch that lol
Great video! Thanks for the great content!
Hi Greg,
Does your model predict different numbers each time you run it? Could you make it give range instead of one number? Kind of Monte Carlo simulation.
Greg, Thx for great video. I'm a complete noobie, just starting to learn Python. But I'm confused. I've seen a number of videos with output plots for RNNs where, like this one, the prediction line seems to mirror the actual line, often AFTER (to the right of) the actual. Of course, I could take a stubby pencil and look at the thermometer outside my window and write the temp down and say it was my prediction five minutes ago. Voila! One line of code. But the prediction line should be BEFORE the actual line, no? What am I missing? Am I just not seeing it due to the scale on your graph? Rob
Great video! I have a question, do you have to do normalization in time series forecasting, I see it was not presented in the video.Thanks！
This video helped me greatly.
Amazing explanation.
Thank you.
Super glad to hear it :)
Hello, amazing video, may I ask something?
So lets say we were to predict a single value, we don't need to use the time somehow or not? Does the model kinda learns to which time the data sequence is relationed? like: "this values n sequence... must be winter so the prediction is: x!" right???
Hello, and thanks for the fantastic video. If I may ask, what would you recommend if I look for a specific kind of time series forecasting in which I don't want to give the model the real values x_test? I am pretending to use the model's prediction to create an x_test_model and then use the model on this set to achieve a y_test_model that I can compare with the real y_test.
Nice tutorial! I'm struggling to conceptually understand what happens when you call predict. I understand the training portion, but how do I get values for say 30 days out? It seems I need to feed it 30 values that I already know the answer for? What is it actually returning when you call predict?!
For example, if I have a dataset of 100 values with a look back/ window of 5. I want to predict values 100 to 105. I train the model on 95 values and call model.predict on the remaining 5, do I get the predictions for 95-100 or 100-105?
Great VDO mate!!! Keep it up!!
Thanks so much!
Hi, great video! Is it possible to run the data in a sliding window way and update the model as it runs? Thank you
Hi Gregg , when running the predictions :train_results = pd.DataFrame(data={'Train Predictions':train_predictions, 'Actuals':y_train1}) i have noticed that you've used data but no prior reference could you state why?
Do you consider the prediction of first predicted hour in second input to do further prediction?
Great video. Do you know where I can find more information on how to determine the number of layers when performing the model.add() commands. For example, I believe you used 16 for your LSTM.
thank you for the very good video ...but isnt working with np Arrays exhust the computer memory during fitting ? can we find another way to prepare the data before training... i am working on a multivariate multistep LSTM model ... i have 90 features to predict ine target value ... with many zeros in the target and also in the input features... it is very hallanging specially for multi step forcasting... any advice!?
really helpful man, thank u
really awesome video, learned a lot about LSTMs. thank you Greg!
Great to hear!!
@@GregHogg do you have anything to predict future stock price? :)
@@moondevonyt Yes check my recent Videos :)
@@GregHogg youre the man, thanks!
Can I use the R2 value to measure the accuracy of the LSTM model in time series prediction?
Hi there! Found this really helpful, but I've managed to confuse myself about the data windowing and the test set. Since every target in y_test (except the last one) is somewhere in X_test, how are we not assuming our solution? I had a look at your stock price notebook and there's the bit about recursive predictions, but you don't have that here. Any clarification?
Thank you so much. It definitely was much needed.😇😇
You're very welcome and happy it helped!
awesome tutorial bro!!!,any thanks for this video, to confirm, could you say that the 5 data above predict 1 value?
hi Greg, what if my target variable is not normally distributed but positively skewed? what should I do in such case? and also, is it recommended to datetime function as cos and sin of hour, weeks n all to provide it in the x variable?
Great video, really helped a lot!
Glad to hear it :)
Thanks a lot, Greg
great, please how should we prepare our data if we want to predict the temperature in different cities, (redundant dates)??
Probably the best video on youtube for LSTM. Question which I cant find anywhere, if I want to apply this and show what my next forecast would be, would I just use model.predict() and pass in the last 5 intervals to get my next hour prediction? I cannot find a good example anywhere of someone actually using the model to forecast.
That's very nice of you to say, Eric! Yeah, that's what you would do.
Amazing! Congrats for great explanation, my friend. I have a question. If I wanted to forecast future temperature regarding date in the future, how could I do that? Regards, Lucas from Brazil.
How would we do if we did not know the values of x that correspond to y that we want to forecast?
Great Video Gregg!
Thanks Akash!
Would this approach work if you wanted to predict the next say 24 or 96 values, instead of just the next 1?
awesome, great explanation!
Thanks Ken, I really appreciate that!
Hi may i know how to evaluate the model using rmse, mae, mape and mse?
fantastic, thank you very much for sharing
You're very welcome!
Hello, how can u predict to future with this code?
Hi Greg, that was a lovely and informative presentation. but you are using the training data again in prediction. so why did we split the data into testing, training, and validation?
I have tried creating a similar forecasting model but have to normalise the data, whereas you did not in your code. Is there another reason I'm not understanding that allowed your model to work with the real values?
Great video! Question: I don’t understand the difference between validation data and test data. You say that test data has not seen the model before, but so did the validation set? I thought we only divide into train and test data
Thank you.
What shall be modified in the code in case the X is not a single value (like temperature), but many features vector and y is a single value?
I will cover this in the future. But essentially, we'd probably want the single items in each list (the third dimension) to be more values. Then you'd have to change the input shape of the model accordingly
Thank you captain.
Thank you so much, I followed the tutorial and got my first LSTM network working. I am however having trouble with generating out of sample predictions, I feel like I have tried everything and just can’t crack it. I am using the last batch of the test set to predict the first out of sample data point, but it ends in an error. I feel like it’s a dimension thing. Any help, anyone?
Why do you use rmse over r2 score? I would like to know thanks!
Can you giv e an example of multivariate forecasting?Also if i have temperature datas in two different files and i want to train the LSTM on both the datasets, how will i include that ? Example: first dataset is say temperature on antartica (-10 to 5 degree celsius) and other file has temperature of south africa(30-50 degree celsius). I cant just simply append both data frames. So do i need to train it twice on both datasets?
Hello Greg, first of all, thank you very much for the very lovely and exciting video. I do realize that in the test part, you are not updating the window with the new predictions. So mainly, you are doing predictions for every five timesteps, but then in the window, you are using the actual test values to predict the next value. So my question is did you try in this way as well, and if so, how was your results? Thanks again!
I have the same question!!
Idk why but I got an error "No file or directory found at model1/" when trying to save the best model
Thank you so much, it's very helpful
You're very welcome and I'm super glad to hear that :)
How to implement data filling (missing values) using LSTM?
Great video Greg. Thank you for sharing.
I have a question. Is it better to perform data normalization/standardization before we start the process? Thank you
Thank you. As long as you've preprocessed inputs in some form, you probably can't go too wrong
This also seems only helpful if you want to predict a singular value in the future. What if you wanted a very large number of predictions?
hey I need a help, I got an error saying ( UnimplementedError: Graph execution error:), what should I do?
great video. Thank you very much
Very welcome!
Hey, can you also forecast time series data like that but using random forest??
This video at 13.26 shows a time series function, but it is single-step forecasting only... can you show how to modify it for multistep. I have tried to add multiple labels in the loop but it is not working with LSTM. Thanks, Greg for this video also.
How should I predict feature? Let's say for next 1 week time period? I know val_train is made from historical data with windows size, but how should I make this kind of input for next n days? Thanks.
great video, thank you
You're very welcome!
Which data you as been taken in this video can you please show us or update a dataset your
Hello, thank you so much for this video, after doing
train_predictions = model1.predict(X_train1).flatten()
train_results = pd.DataFrame(data={'Train Predictions':train_predictions, 'Actuals':y_train1})
train_results
i got this error:
ValueError: All arrays must be of the same length
What should I do ?
does the lstm train only on 1 input ? i see this ((70086, 5, 1), for X.shape
Great video. I do time series forecasting using ARIMA and Prophet models. I wanted to have a go at a different approach such as LSTM, and this video is very helpful. I do have a couple of questions for you, and all of the readers here. What would I need to do if I wanted to forecast temperature for the next 60 days (or x number of days) with time stamp as our index? Thank you in advance!
Thanks Jake!! I'll defer you to the TensorFlow tutorials for this one, it's where I based my lesson off. www.tensorflow.org/tutorials/structured_data/time_series
how would the df_to_X_y function change if instead of only having a date and temperature column you also have a location column because now X would have the format (n_samples, window_size, 2)
Merci 😊
which direction are you feeding the data in the lstm?
Great! What should I do to improve the accuracy on my trained LSTM model?
Dropout, regularization, adjust the model in any way you see fit!
wow ,you're a GOD!!
Thank you haha
tnx.nice job
Nice Video ! Just think about mention François Chollet and Keras for their tutorial
Bro you sound like Robert Greene and you are just as awesome as he is or even more awesome !
Sir how can we take all input features that is all columns
hey nice video. ive got a question pls answer quick if possible.
I have a list of time series data for 100 stocks. I want combine them into a single dataframe that has open, close, high etc for each stock. so i'd have multiple rows of the same date.
My question is, is it possible for me to then create a model to predict change in stock price based on all that data from the single combined dataframe? if so, how?
This might help: th-cam.com/video/kGdbPnMCdOg/w-d-xo.html and if you're predicting differences, well, you'd have to make that column.
Thanks a bunch! Do you have any advice on how to create confidence intervals for predictions?
I'd like to know that too =)
awesome tutorial bro!!!
Thank you!!
I want to code Boosting LSTM, what should I do? Is it much different from LSTM? Thank you very much.
I realized it was the previous statement to create a problem:
model1.add(InputLayer((5, 1)))
I cancelled one parenthesis believing it was redundant and I wrote model1.add(InputLayer(5, 1))
but this causes a crash to the following line.
At the moment it is not exactly clear to me the reason. I have to study about.
In any case I can go on now.
Sorry for the inconvenience.
Great stuff Greg! Thanks to this vid I finally got the input dimensions of the LSTM right. One quick question: why is the extra `Dense(8, "relu")` layer necessary, and not just the final Dense(1)?
Thank you, and great to hear! I forget the final model but most likely this is for extra complexity
@@GregHogg Hey so, is it okay to put fully connected layer after LSTM Layer? Thank you
Great video with good explanation!
But if I want to forecast the values for the next month how should I do it?
I would my stock prediction video. And thank you :)
thanks for video
hello
i'm a beginner of machine learning
is it important that the index on the dataframe must use a timestamp before being put into the train model?
is there any effect if the index still uses auto increment?
sorry for bad english
It doesn't do anything once it reaches the NumPy stage. It just allows the use of pandas functions to manipulate the dataframe in pandas
@@GregHogg Thank you so much for your attention
I have 1 more question
how to determine a good number of hidden layers and good activation function ?
I have tried the lstm model with different hidden layer configurations and activation functions several times with my data, but the training results are always not good
Is it necessary to use a MinMaxScaler here?
What is the difference between test data and validation data? They seem to be used in the exact same way in your notebook.
Validation is going to be repeatedly checked to pick the best model. Test is saved at the end for a final evaluation of the chosen model