For anyone wondering, this model means nothing as many previous videos have done, it’s simply a lag prediction, day 1 is 100: day 2 is 120 and model predicts 99 for day 2 for example, on day 3 the model will predict a number near day 2 ie 120, thus generating a lag graph. Now how to actually predict it works ? Let it predict percent change difference not just price therefore isn’t as affected by previous day price and the lag effect is gone and you can actually see how many correct predictions happen.
@codeTrading hey i know you mentioned it in the description and you know this. Just thought I’d clarify for those who were wowed about it :D I did my thesis trading crypto with lSTM actually and got it to work (it did well but not amazing), but yeah I’d say there are better models than LSTM for this.
@@yaas actually the reason I did this video is because it looks so good at first and might trick some of us. It's good you could get it to work for me it looked more challenging than other algos.
Yes it took months and tens of hyperoptimizations just to get about 65% accuracy ( When entered in a trading bot though profits where significant year B&H: -180% model profit 160%, in multiple coins. So yea overall not too shabby.
Be careful, you did a very common mistake: the scaling of data must be done after train/test splitting and not before, otherwise it can introduce data leakage.
Hey Jorge thanks for pointing this out indeed it can affect the scaler leaking the max and min of train data to the test data, but even with this it is still not a good technical indicator in spite the error which should play in favor of the prediction.
During analysis train test split should be done with the way you mentioned, but during live deployment you are not splitting based on the params you use and you need as much as recent data possible. This makes significant improvement in my model. I haven’t lost a single penny over the past 6 months doing that. Plus another problem is that LSTM is not used to predict price, hence why the results are not amazing in the video, I have a live bot running on this and my use case is completely different from the video + I use as much as data possible, although this is an interesting video for beginners
@@AI_BotBuilder hi sir i am working on a bot for trading and im using algo strategies and i am trying to implement machine learing to it. if you could help me with the process i would really apriciat.(even hints are appreciated)
Mate, you did an amasing job, you are better instructor than 99% of university professors, they only give you books to read and 0 coding practice. Additionally, I believe that the main reason of this video is to show how to program and how actually machine learning works, therefore, it depends on the person what indicatora use and what timeframes use. You did 90% of the job. Congratulations and thanks for sharing
Thank u so much 4 ur efforts most of predictions or all them kind of shifting actual data by certain value it should be like this> use real data up to now then predict what after that
Thank you, very much brother. I have been searching for 3 days for how to make strategies for NN classification. I have got a lot of ideas from your video. seriously thank you very much brother.
along with this.. can you also include how market depth affect prediction. . and financial news of a company affects the price prediction at that time.
Excelente explicación, me gusto mucho como fuiste detallando cada parte, estoy revisando todo tus videos y los veo varias veces para estar seguro que todo comprendí y no perdiera algo.
Amazing work! Although I have a few questions, sorry for the noob question, as I’m still learning CNN, just a normal pandas and numpy user. When using these models to predict, are there any cases of accidental data snooping? What are the best practices to avoid it? Also when the neural network predicts one of the next days closing price and moves to the next one, does it use the predicted closing price from previous day or the actual data’s adjusted close for computation? I’m sorry if it’s a stupid question, I’m just curious to understand this better. Thankyou!
Hi thank you for your support, actually your questions are not as noob as you think, the model uses the last real close value to predict the next one and this is very tricky I am making a video about it for the next week because people shouldn't be tricked by a nicely looking curve... More to come hopefully it will answer your questions.
first thank you a lot for such a great video. if I am not wrong the results you shared at the end of the video are the scaled one. what do you think will be the best approach to rescale them and will scaling the data increase the predation gaps?
Hi scaling will not change much in this case, the problem is more serious than this, so errors from scaling are invisible here. But just to be more precise scaling is playing to the predictions favor here because of data leakage when splitting into train and test chunks.
I am completely NEW and just want to try out the sample in the video and ran into 2 problems below. I typed in all the codes, and guessed what toward the end, the screen got cut off. so help! model.fit(x = X_train, y = y_train, batch_size=15, epochs=30, shuffle=True, validati???????) ALSO the last line caused problem as well. lstm_input = Input(shape=(days,8), name='lstm_input') inputs = LSTM(150, name='first_layer')(lstm_input) inputs = Dense(1, name ='dense_layer')(inputs) output = Activation('linear', name='output')(inputs) adam = optimizers.Adam() model.compile(optimizer=adam, loss='mse')
Super!!!! I didn't understand one point. When u define data[TargetClass] and set 1 if data ... Greater than 0 or 0 for i in a range. What it means to set it to 1 or 0?
Hi thank you, 1 or 0 are 2 classes these are called labels in machine learning so that the model learns from these labels when. Price is going up and when it's going down.
helllo, thanks a lot for your video, it was wonderful to watch. I have a question, in the end, your predicted prices are not scaled back to normal, it is possible for you to telle me how i can scale everything back to what it originally waws ? thank you !
Hi thank you, honestly I wouldn't use neural networks for this, as the results are not that accurate, I would for a classic less computation and more reliable.
instead of predicting the next day price outcome, can we make the model to predict a trend? I'm not sure about the ideal timeframe but maybe between 2 weeks to 2 months ahead? So it's more like a medium term price predictor... If it's possible I would be very interested to see the next part the series to dig deeper into this. As always great video!
Hi thank you for sharing your ideas, yes it's possible to try and predict let's an average market price over the next 5-10 days I think a video is worth it...
Good lucking trying to predict the next day’s close with a simplistic model such as this. It will not work. As for predicting weeks or months ahead? Absolutely no chance.
Great job, I learned a lot, but I have a question , what should I do/ what the code should I use if I want to forecast next 4-5-10 days. as per as I understood from this useful video that you built a model to predict the next closing price and compare it with the real data you already have, but my question is, what if I want to forecast for n days??
Thank you for you support! answering your question you will have to modify the training part so the target values considering 5 days and then train your model on this, followed by predictions which are assumed to predict for 5 days. That's one simple way of doing it.
Hi, when you scaled your entire dataset before splitting it into training and test sets, do you think information from the test set might have leaked into the training set during scaling, which could've maybe artificially boosted performance?
Well to be honest I prefer education, understanding and critical thinking so why not? (Of course if the code would make us rich as well that would be nice but so far my codes can only make 20percent return per year... So not rich yet 🙂)
It seems that the gradient found that y( i ) = y( i - 5 ) reduce the error as much as it can~ But would't act as prediction, it's just a forward shift of the past values. I'm pretty sure that if the models runs in autonomous mode for just 100 close steps it will diverge into a whole new plot. Plus, using minmax scaling to normalize price values than can exceed over 0 or 1 by far, doesn't seems a good idea. I would use price percentual derivative or percent change in pandas.
I see your point but how do you mean percent regarding open high low and close values. ... actually you might be interested in next week's video as a continuation
@@CodeTradingCafe You don't, you calculate all the indicators you want by closing price at first, and then, those indicators that are not index like RSI or Stochastic RSI that are already "normalized", you need to apply the percent change value over EMA, SMA, BB, MACD or any other indicator that still represents some direct price value. This last needs to be customize a little bit to each Indicator, because you have to decide which values are you applying the percent change ( like, Bollinger Bands, which pct change would be better for Top band? The pct change of itself, or relative to the middle band? ) This is just what I've tested by my own, I actually never tried using the price directly which afaik neural networks don't like descriptors like that. And, since we are using time series input, seems to make sense to predict time series aswell, but did you think about classifier? Works even better. In case you feel curious about what looks like the classification I've made, here is an example by different cost functions and different periods. i.postimg.cc/fLcrpjL7/nn-class-sample.png This model was just using Stochastic RSI, annd no recurrent layer needed to this, just one layer basic model, but still need to test it on LSTM layer if makes some improve. I've already tried in CNN, but at more indicators you use, the complex it gets, specially if indicators have different periods.
Thank you for sharing, that's a lot of work 🙂 will keep some ideas in mind for next projects... But to be honest lstm is the most challenging in this field I don't think it would work, simple methods are providing better results.
Hi thank you! it's an open question how to normalize data it seems that most of the times data scientists try both and they pick the better results, however in our case it wasn't worth it since we are far from real predictions.
I think that using the price as a feature can't be the right thing, because we don't have a ceiling or a floor for the price, the maximum price in the train dataset may be 100, but in the real data, we have more than 100.
Thanks for the Video! Im currenly learnig a few things and your videos helped me a lot. I noticed that the model will give me different results with the same dataset and same parameters. Is there any way to avoid this behavior?
Hi, thank you for your message, I am glad these videos are helpful. For the results differences might come from any parameter it's hard to guess but you should get something close anyway.
For me, the green line looks quite identical but also with an offset of one or two days. Its Not really a prediction If I Just repeat the values fromm the days before. Actually, I encountered the same issue but dont know how to deal with it yet.
Hi, thank you, I agree, it's totally a copy of the previous price I have another video where I zoom into this issue and show why it happens th-cam.com/video/lhrCz6t7rmQ/w-d-xo.html I hope it helps
Greetings from HK, thank you for the wonderful tutorial! I am wondering if there is a chance we may adjust the input parameter(s)' "time range" including the return change of the stock itself and other indicators, to make this suitable for stock prediction 1 year later (but not tmr), I am trying to include machine learning elements into equity research report that usually has stock target price set for 1 year later. Would appreciate it a lot if you could let me know how to amend the code, thankss!! 🙏
Hi, thank you for your comment, you can change the target function parameter it takes a number of future candles, however for this I would use the weekly or monthly timeframe, it makes things easier and with less noise for your long term predictions.
It is actually doing nothing but giving maximum importance to the last day, if you watch closely it is the same graph with one day lag, it is not forward looking but backward looking model, I do not recommend this for any prediction whatsover. You can look at the % accuracy of the model in terms of buy and close signal and you will understand what I mean here. You require some forward looking elements to make a prediction which is not close enough but able to help you predict right maximum number of times.
Hi thank you for sharing, actually check this video th-cam.com/video/lhrCz6t7rmQ/w-d-xo.htmlsi=NbrAXUucFpuIXwZQ it's more realistic, agreeing with what you just mentioned.
Hi, thank you, for the non numeric features I have to show it in a video, but one example could be if EMA trend is up or down (categorical feature), or also sentiment analysis of a webpage for example.
In general you would need to encode non-numeric features. Basically translating them into numerical features with associations between values and categories of data. There are quite a few ways to do this, like one-hot encoding and also label-encoding. If you have five categories for one feature (for example: a stock could be in the manufacturing sector, technology sector, farm sector, health sector, or retail sector) you could assign each category a value from 1-5 and use that in your feature set. This would be label encoding. There are some good guides on youtube and some more in-depth articles on medium if you google them.
Adj close is more precise considering market changed occurring when sessions are closed, if it's available it's usually better to use, but we're not at that level of precision yet with the LSTM trading approach.
Love this video and your series. Keep it up bro. Question, how to start prediction once you've saved the model? iow, how can I generate a df['predictedprice'] column using model.predict? Without using the 0 - 1 scaled data conversion.
@@CodeTradingCafe Perdón por la pregunta super noob pero la función necesita un rango de características y estoy teniendo problemas para encontrar lo que es en el código como lo que debo poner como el rango de características. Podrías ayudarme por favor 😂
Hi, I will be glad to get some help to understand better your network. as you said you predict the 10th day value based on the back candles days. so I expecting to get 1 value if I will send to the predict function of the model back candles number of data. but I actually get back a back candles predicted values. I will be glad if you can explain it better for me.
Hi, it's hard to debug if I am not running the code, but you are probably getting a single forecast value per point and you can have many points in the test set I mean in the code X_test is a range of values.
thx! I was able to combine some stuff from this video with some stuff from u're bbands scalping... backtested it on unknown data and the AI is able make some amazing signals! it performs better than all the normal strategies I've tested. Like suggested I've let it have to choose only one category: long or not long. However the AI signals alone are useless... but in combination with normals signal it performs really good.
Thank you for your feedback, this is nice to here, I initially started this channel to share ideas where people can tweak and learn and experiment just like you did. Good luck!
Nothing at all, actually in another video I show why it's just an illusion that this works while actually for the moment it doesn't. (the video th-cam.com/video/lhrCz6t7rmQ/w-d-xo.html )
It is, but the results are not up to the expectations, check some other videos they provide better indicators "support and resistance", or "price patterns". Good luck!
@CodeTradingCafe Can I use this model to predict the Index? My goal is to predict changes in NASDAQ with machine learning using of global market indices such FTSE, DAX, Nifty, etc.
Thanks @@CodeTradingCafe. I think you are correct. I have tried with LSTM but is not working as expected for my use case. Let me try with bagged adaboost/xgbtree classification model.
@@CodeTradingCafe Then the train and test samples being mixed together instead of having a never before seen period of testing data 😂 I remember being hyped about it evaluating well, and then I tried it on actual values and the algorithm turned out to be absolute dog water. This prediction problem is the definition of an emotional rollercoaster
it is but since the time between candles is regular and anyway not needed here just the order of the candles is important, we can skip the datetime info.
Forget predicting exact price, why not just make it follow the trend and learn when to buy or sell. Trading is not about predicting price but rather making profits from the ongoing moves or am I wrong?
Hi, thank you. We are comparing the predicted future price to the real future price, so no shift needed here, because in real life if we use the model to predict future value... we can't shift values
Hi, you would use the current week data as test data and see the prediction for the next day rather than the next week, but in this case you need to train the model also to detect next day or next week values, so the way we train the model influences how the model applies predictions.
@@CodeTradingCafe yes but since its a times series, how do I train lstm on future values rather than previous? Im new at this, sorry if I ask stupid questions
Predicting the closing is nearly impossible cuz its affected by multiple factors, moreover a good causality and correlation analysis will help improving the predicting model performance
@@CodeTradingCafe I have tried but it will give ValueError: non-broadcastable output operand with shape (506,1) doesn't match the broadcast shape (506,11)
The shape of the data has to match if you fit the data on a dataframe of 11 features then you can inverse transform on the same data size, try fitting the transform on a single column first if you want to inverse transform a single column
Thank you for your nice video. One comment is that because the the train-set and test-set are splitted AFTER applied the MinMaxScaler, so, to visualize and calculate MSE, the target and predicted value need to be inverted using the same scaler. You can find that your plot (to compare real and predicted close price) shows the price in range $0 - $1 while it should be the original close price which are much higher (> $700). Would be very appreciated if you can share that code to invert the predicted value if you did it lol
@@CodeTradingCafe i did inverse_transform function , but i am getting error like that ValueError: non-broadcastable output operand with shape (503,1) doesn't match the broadcast shape (503,11), pls help how to solve this error
@@Lbvi you have to apply the function on dataframe of the same shape so you are replying only on the target which is one column however you have transformed 11 column dataframe.
@@CodeTradingCafe This was a very helpful video for build my first NN for stocks. I'm also struggling with this inverse. I've been at it for days and cannot get it to match the shape. Any suggestion on how you take the single y_pred and reverse it back to the original 11 column df that the scaler was applied to?
In my opinion, your model needs regularization and cross validation. Seems highly overfitted to me. Some prunning maybe will work fine, bias & variance balance needs to be applied to.
Hi thanks a lot for your input, I think your are right regarding regularisation it's crucial, but I was expecting at least a small signal in the predictions which I haven't been able to get maybe I will revisit taking into account your comment. (If I remember correctly in a later video I did include some regularisation but with no major improvement, however I never did a full cross validation).
@@CodeTradingCafe I think your model needs to decrease a bit in complexity in order to avoid overlearning the training data. A consequence of that could be not being able to predict unseen data due to overlearning on the training set mentioned above. I will probably try it with a much simpler polynomial regression(grade=3 maybe? If you actually plot it, you can see it has some similarities with the sigmoidal function you use in your RNN) and test my results, trying to find the best balance between the accuracy on the training data (I think you can check with .score()) while maintaining a healthy MAE ((mean_absolute_error(y_test,y_pred)). If polynomial regresion doesnt work as expected then try to reduce complexity on your RNN (I don´t know exactly how, I will probably have to study it). Sorry for my poor English. I am not native. I hope this helps. Best,
Hi thank you, overfitting can be avoided by pruning some of the knots and we can also use less variables and layers, but I believe it is much more complex than this, neural networks are good in some cases but they perform very poorly in others (NN work will for CV for example).
Theoretically scaler can be applied to both train and validation because if the model is trained on scaled data it will only work with scaled data. But I agree that to avoid data leakage scaling should be done independently on training and validation sets.
Pycharm, vscode, anaconda, Spyder... Any python interface works. But from your question I guess you haven't used python before, so I would start with some basic lessons, try my Python learning playlist for a quick start. Good luck!
@@CodeTradingCafe fare enough nah I’m studying IT HAHA but u know how the asset that’s were like should we change it in the code I just want a bot that really good and passed funded accounts if that’s possible anyway thanks for replying❤️
All this is fine until a company/news event announces mid day that they have really great/bad news and your algo never trained on this sort of event because you're using historical prices that don't account for this as a feature. This happens all the time with anything that gets traded. Some new event happens and the price discontinuously jumps/falls to some new level that wipes out your account . I've never seen any machine learning prediction model take account of black swan events. Disingenuous.
And if I may ask, since you seem very knowledgeable in this matter, how often big news happen? and why big news should wipe your account if your strategy is locked in a low risk trade management system? please enlighten my disingenuous understanding.
Hi forward testing can be done by saving the trained model and then you feed live features to the model which provides forecast. But if you are not familiar with python and ML it might take longer than expected.
@@CodeTradingCafe I'm using an edit that uses GRU instead of LSTM and i tweaked the hyper parameters a little, but i can't really find a way to have it provide forecast for live stocks. Normally i'm good at this stuff 🥲
@@CodeTradingCafe Also i was wondering if it would be smart to have a sort of "ultimate" forecast algorithm that uses a lot of different methods, neural networks or simple algorithms and use the (maybe weighted) average of the predictions as the forecast 🤣
It might be an approach to have an ensemble method, but it might be time consuming to build such a tool, and sometimes a box of troubles you're opening 😂
Przewidywania są stale opóźnione. Model obrał najlepszą strategię w postaci podążania za trendem, stąd podobieństwo wykresów porównujących przewidywania i wartości prawdziwe - zwyczajnie model przewiduje wartość wejściową z wejściowej, czyli przewiduje, ze cena przyszła będzie taka sama jak obecna. Dlaczego? Niech sobie każdy sam odpowie. A idioci łykają to jak pelikany. Jeszcze jak widać nie było wysypu milionerów.
Hi, because absolute values don't mean anything for the algorithm. Think about it this way if I tell you the price is 1$ could you guess the next price? or if I tell you the price difference is +0.5$ in this case you could guess the price is going up and you are more likely to guess the next price.
why to use these moving averages and other lagging indicators.... It looks you just get the same lagging indicators implemented by neural network... And it's not logical to use min max scaler when future prices can be and will be out of the training range sooner or later...
Hi. I have error at this line lstm_input = Input(shape=(backcandles, 8), name='lstm_input') The error is: TypeError: descriptor '_values' for 'BaseContainer' objects doesn't apply to a 'RepeatedCompositeFieldContainer' object Can you kindly check and advise? Thank you.
just plotting and looking they are close does not mean its accurate enough ....u need metrics ok, even if u do training and testimg..u need to assess ur accuracy
@@CodeTradingCafe no need to check, my fault. The fit function generates batches that can be shuffled and, in theory, should preserve the sequential order of the TS in each batch. Anyway, thanks for the quality content.
For anyone wondering, this model means nothing as many previous videos have done, it’s simply a lag prediction, day 1 is 100: day 2 is 120 and model predicts 99 for day 2 for example, on day 3 the model will predict a number near day 2 ie 120, thus generating a lag graph.
Now how to actually predict it works ? Let it predict percent change difference not just price therefore isn’t as affected by previous day price and the lag effect is gone and you can actually see how many correct predictions happen.
Thank you Yassin, check my next week video, you might be interested in.
@codeTrading hey i know you mentioned it in the description and you know this. Just thought I’d clarify for those who were wowed about it :D I did my thesis trading crypto with lSTM actually and got it to work (it did well but not amazing), but yeah I’d say there are better models than LSTM for this.
@@yaas actually the reason I did this video is because it looks so good at first and might trick some of us. It's good you could get it to work for me it looked more challenging than other algos.
Yes it took months and tens of hyperoptimizations just to get about 65% accuracy ( When entered in a trading bot though profits where significant year B&H: -180% model profit 160%, in multiple coins. So yea overall not too shabby.
I can imagine the amount of work sometimes I am drained after a TH-cam video 😂
Great job dude!! I'm learning a lot watching your videos. You have an excellent way of teaching, using plain, clear and concise English. Thank you!!!
Thank you so much for your support, I started this channel as a learning space, but drifted more towards trading 🙂
Watch the Lazy Programmer's videos if you want to learn why everything in this video is crap and doesn't work.
Be careful, you did a very common mistake: the scaling of data must be done after train/test splitting and not before, otherwise it can introduce data leakage.
Hey Jorge thanks for pointing this out indeed it can affect the scaler leaking the max and min of train data to the test data, but even with this it is still not a good technical indicator in spite the error which should play in favor of the prediction.
During analysis train test split should be done with the way you mentioned, but during live deployment you are not splitting based on the params you use and you need as much as recent data possible. This makes significant improvement in my model. I haven’t lost a single penny over the past 6 months doing that. Plus another problem is that LSTM is not used to predict price, hence why the results are not amazing in the video, I have a live bot running on this and my use case is completely different from the video + I use as much as data possible, although this is an interesting video for beginners
@@AI_BotBuilder hi sir i am working on a bot for trading and im using algo strategies and i am trying to implement machine learing to it. if you could help me with the process i would really apriciat.(even hints are appreciated)
@@AI_BotBuilder Whats your use case
I fell like this is the issue am having with my model , how is this even possible ?
Mate, you did an amasing job, you are better instructor than 99% of university professors, they only give you books to read and 0 coding practice. Additionally, I believe that the main reason of this video is to show how to program and how actually machine learning works, therefore, it depends on the person what indicatora use and what timeframes use. You did 90% of the job. Congratulations and thanks for sharing
Glad this was helpful, thank you so much for your support!
Thank u so much 4 ur efforts
most of predictions or all them kind of shifting actual data by certain value
it should be like this> use real data up to now then predict what after that
Thank you for your comment, more of these to come... Good luck!
Thank you, very much brother. I have been searching for 3 days for how to make strategies for NN classification. I have got a lot of ideas from your video. seriously thank you very much brother.
Glad it helped, good luck!
Wow, you're awesome! There's lots of stock/crypto predictions on TH-cam but hardly any for forex so this is great for learning it
Welcome aboard! Glad this is helpful!
along with this.. can you also include how market depth affect prediction. . and financial news of a company affects the price prediction at that time.
Yes it's possible a bit more complicated though. Requires more time 🙂
Excelente explicación, me gusto mucho como fuiste detallando cada parte, estoy revisando todo tus videos y los veo varias veces para estar seguro que todo comprendí y no perdiera algo.
Gracias amigo 🙂 good luck!
Amazing work! Although I have a few questions, sorry for the noob question, as I’m still learning CNN, just a normal pandas and numpy user. When using these models to predict, are there any cases of accidental data snooping? What are the best practices to avoid it? Also when the neural network predicts one of the next days closing price and moves to the next one, does it use the predicted closing price from previous day or the actual data’s adjusted close for computation? I’m sorry if it’s a stupid question, I’m just curious to understand this better. Thankyou!
Hi thank you for your support, actually your questions are not as noob as you think, the model uses the last real close value to predict the next one and this is very tricky I am making a video about it for the next week because people shouldn't be tricked by a nicely looking curve... More to come hopefully it will answer your questions.
I can not wait until the next video.Thank you so much!
Thank you for your support!
first thank you a lot for such a great video. if I am not wrong the results you shared at the end of the video are the scaled one. what do you think will be the best approach to rescale them and will scaling the data increase the predation gaps?
Hi scaling will not change much in this case, the problem is more serious than this, so errors from scaling are invisible here. But just to be more precise scaling is playing to the predictions favor here because of data leakage when splitting into train and test chunks.
I am completely NEW and just want to try out the sample in the video and ran into 2 problems below.
I typed in all the codes, and guessed what toward the end, the screen got cut off. so help!
model.fit(x = X_train, y = y_train, batch_size=15, epochs=30, shuffle=True, validati???????)
ALSO the last line caused problem as well.
lstm_input = Input(shape=(days,8), name='lstm_input')
inputs = LSTM(150, name='first_layer')(lstm_input)
inputs = Dense(1, name ='dense_layer')(inputs)
output = Activation('linear', name='output')(inputs)
adam = optimizers.Adam()
model.compile(optimizer=adam, loss='mse')
Hi thank you for your comment, you can download the code from the link in the description, you don't need to type all of it. Let me know if it works.
Super!!!!
I didn't understand one point. When u define data[TargetClass] and set 1 if data ... Greater than 0 or 0 for i in a range.
What it means to set it to 1 or 0?
Hi thank you, 1 or 0 are 2 classes these are called labels in machine learning so that the model learns from these labels when. Price is going up and when it's going down.
helllo, thanks a lot for your video, it was wonderful to watch. I have a question, in the end, your predicted prices are not scaled back to normal, it is possible for you to telle me how i can scale everything back to what it originally waws ? thank you !
if you are using a scaler, you can use scaler.inverse_transform(...) function from sklearn. Thank you for your support.
This is absolutely eye opening.
Thank you for your support!
Thank you for this video, would you have any idea, genreally speaking how to approach this if one had multiple stocks to predict?
Hi thank you, honestly I wouldn't use neural networks for this, as the results are not that accurate, I would for a classic less computation and more reliable.
instead of predicting the next day price outcome, can we make the model to predict a trend? I'm not sure about the ideal timeframe but maybe between 2 weeks to 2 months ahead? So it's more like a medium term price predictor...
If it's possible I would be very interested to see the next part the series to dig deeper into this.
As always great video!
Hi thank you for sharing your ideas, yes it's possible to try and predict let's an average market price over the next 5-10 days I think a video is worth it...
Good lucking trying to predict the next day’s close with a simplistic model such as this. It will not work. As for predicting weeks or months ahead? Absolutely no chance.
Great job, I learned a lot, but I have a question , what should I do/ what the code should I use if I want to forecast next 4-5-10 days. as per as I understood from this useful video that you built a model to predict the next closing price and compare it with the real data you already have, but my question is, what if I want to forecast for n days??
Thank you for you support! answering your question you will have to modify the training part so the target values considering 5 days and then train your model on this, followed by predictions which are assumed to predict for 5 days. That's one simple way of doing it.
Excellent work. As I got motivated with your video, I also watched your python training in Udemi. Congrats!!
Thanks a lot for your support I hope the courses were useful ! Good luck in your coding journey!
Interesante..., ¿predecir el futuro sería Maravilloso.....?
Not really accurate though
Nice video! How do we use this to predict the prices for tomorrow??
It's not tested enough, and actually at this stage it would still waste/loose money, check my most recent videos for live trading from python.
Thanks for sharing, its really educational and so helpful as this improves analytical view concerning the market
Thank you for your support 😊, it really helps.
Hi, when you scaled your entire dataset before splitting it into training and test sets, do you think information from the test set might have leaked into the training set during scaling, which could've maybe artificially boosted performance?
Definitely yes, there is a leak of information especially the max and min values, but still with this the performance of RNN is still bad.
Excellent. Could you make a video on Forecasting using Prophet?
Thank you for your support! check this one I used Prophet th-cam.com/video/lmoHsgSKBNA/w-d-xo.html
I tested the model and it is so accurate on testing data..I want to predict the future like 2024 values what should i do??
Hi, it's impossible to predict 2024, a lot can happen in the meantime and market will change eventually.
Do u seriously share the code?! Man u are just unique, this is so nice. Thanks so much, really!!
Well to be honest I prefer education, understanding and critical thinking so why not? (Of course if the code would make us rich as well that would be nice but so far my codes can only make 20percent return per year... So not rich yet 🙂)
@@CodeTradingCafe 20% per year is great, you got banks making 12 only.
True! But greed is the culprit 😂 always looking for more return
It seems that the gradient found that y( i ) = y( i - 5 ) reduce the error as much as it can~
But would't act as prediction, it's just a forward shift of the past values. I'm pretty sure that if the models runs in autonomous mode for just 100 close steps it will diverge into a whole new plot.
Plus, using minmax scaling to normalize price values than can exceed over 0 or 1 by far, doesn't seems a good idea. I would use price percentual derivative or percent change in pandas.
I see your point but how do you mean percent regarding open high low and close values.
... actually you might be interested in next week's video as a continuation
@@CodeTradingCafe You don't, you calculate all the indicators you want by closing price at first, and then, those indicators that are not index like RSI or Stochastic RSI that are already "normalized", you need to apply the percent change value over EMA, SMA, BB, MACD or any other indicator that still represents some direct price value. This last needs to be customize a little bit to each Indicator, because you have to decide which values are you applying the percent change ( like, Bollinger Bands, which pct change would be better for Top band? The pct change of itself, or relative to the middle band? )
This is just what I've tested by my own, I actually never tried using the price directly which afaik neural networks don't like descriptors like that.
And, since we are using time series input, seems to make sense to predict time series aswell, but did you think about classifier? Works even better.
In case you feel curious about what looks like the classification I've made, here is an example by different cost functions and different periods.
i.postimg.cc/fLcrpjL7/nn-class-sample.png
This model was just using Stochastic RSI, annd no recurrent layer needed to this, just one layer basic model, but still need to test it on LSTM layer if makes some improve. I've already tried in CNN, but at more indicators you use, the complex it gets, specially if indicators have different periods.
Thank you for sharing, that's a lot of work 🙂 will keep some ideas in mind for next projects... But to be honest lstm is the most challenging in this field I don't think it would work, simple methods are providing better results.
Aren't you supposed to use the normal distribution instead of min-max?
Great video by the way. I enjoyed it's simplicity
Hi thank you! it's an open question how to normalize data it seems that most of the times data scientists try both and they pick the better results, however in our case it wasn't worth it since we are far from real predictions.
finished watching
thank you for your support!
I think that using the price as a feature can't be the right thing, because we don't have a ceiling or a floor for the price, the maximum price in the train dataset may be 100, but in the real data, we have more than 100.
you are right, absolute values are not useful in this case, we need relative variables like slopes or so.
Thanks for the Video!
Im currenly learnig a few things and your videos helped me a lot.
I noticed that the model will give me different results with the same dataset and same parameters.
Is there any way to avoid this behavior?
Hi, thank you for your message, I am glad these videos are helpful. For the results differences might come from any parameter it's hard to guess but you should get something close anyway.
Nice work , I will learns this video for forex option trading.
Best of luck, and thank you for your support!
It won’t work. Sorry.
For me, the green line looks quite identical but also with an offset of one or two days. Its Not really a prediction If I Just repeat the values fromm the days before.
Actually, I encountered the same issue but dont know how to deal with it yet.
Hi, thank you, I agree, it's totally a copy of the previous price I have another video where I zoom into this issue and show why it happens th-cam.com/video/lhrCz6t7rmQ/w-d-xo.html I hope it helps
Greetings from HK, thank you for the wonderful tutorial!
I am wondering if there is a chance we may adjust the input parameter(s)' "time range" including the return change of the stock itself and other indicators, to make this suitable for stock prediction 1 year later (but not tmr), I am trying to include machine learning elements into equity research report that usually has stock target price set for 1 year later. Would appreciate it a lot if you could let me know how to amend the code, thankss!! 🙏
Hi, thank you for your comment, you can change the target function parameter it takes a number of future candles, however for this I would use the weekly or monthly timeframe, it makes things easier and with less noise for your long term predictions.
@@CodeTradingCafe Thank you very much!!!
It is actually doing nothing but giving maximum importance to the last day, if you watch closely it is the same graph with one day lag, it is not forward looking but backward looking model, I do not recommend this for any prediction whatsover. You can look at the % accuracy of the model in terms of buy and close signal and you will understand what I mean here. You require some forward looking elements to make a prediction which is not close enough but able to help you predict right maximum number of times.
Hi thank you for sharing, actually check this video th-cam.com/video/lhrCz6t7rmQ/w-d-xo.htmlsi=NbrAXUucFpuIXwZQ
it's more realistic, agreeing with what you just mentioned.
Could you also do predictions with other mathematical models like Quantum Finance and Complex Fuzzy Logic
? Thanks.
For fun yes, but don't expect much on the results side, all still experimental.
How would you inverse transform the predicted values to get actual predictions?
there is function in sklearn called inverse_transform() for this. Good luck!
Hi, Nice work!! Could you also elaborate on how to use non-numeric features in a similar price prediction model?
Hi, thank you, for the non numeric features I have to show it in a video, but one example could be if EMA trend is up or down (categorical feature), or also sentiment analysis of a webpage for example.
In general you would need to encode non-numeric features. Basically translating them into numerical features with associations between values and categories of data. There are quite a few ways to do this, like one-hot encoding and also label-encoding. If you have five categories for one feature (for example: a stock could be in the manufacturing sector, technology sector, farm sector, health sector, or retail sector) you could assign each category a value from 1-5 and use that in your feature set. This would be label encoding. There are some good guides on youtube and some more in-depth articles on medium if you google them.
ahg i cant wait for the next video
Lol you're not going to like it unfortunately, spoiler alert the hero dies ... The algorithm looses 😂
Is there any reason why you chose adj close instead of closing price?
Adj close is more precise considering market changed occurring when sessions are closed, if it's available it's usually better to use, but we're not at that level of precision yet with the LSTM trading approach.
can you combine this with time series forecasting algorithms? I done Autoregression with multi layer perceptrons before could we do ARIMA with RNNs?
Hi, it's possible I am not sure about the results though we have to try it
Where do you grab that data? it's just import on python? can I configurate to take live data with this?
Hi, yes you can get the data directly within python using yfinance module.
Love this video and your series. Keep it up bro.
Question, how to start prediction once you've saved the model?
iow, how can I generate a df['predictedprice'] column using model.predict? Without using the 0 - 1 scaled data conversion.
Thank you for your support, to go back to original values you can use the inverse_transform function of the scaler
... but you might want to check this video first th-cam.com/video/lhrCz6t7rmQ/w-d-xo.html
@@CodeTradingCafe Perdón por la pregunta super noob pero la función necesita un rango de características y estoy teniendo problemas para encontrar lo que es en el código como lo que debo poner como el rango de características. Podrías ayudarme por favor 😂
Hi, I will be glad to get some help to understand better your network.
as you said you predict the 10th day value based on the back candles days.
so I expecting to get 1 value if I will send to the predict function of the model back candles number of data.
but I actually get back a back candles predicted values.
I will be glad if you can explain it better for me.
Hi, it's hard to debug if I am not running the code, but you are probably getting a single forecast value per point and you can have many points in the test set I mean in the code X_test is a range of values.
thx! I was able to combine some stuff from this video with some stuff from u're bbands scalping... backtested it on unknown data and the AI is able make some amazing signals! it performs better than all the normal strategies I've tested. Like suggested I've let it have to choose only one category: long or not long. However the AI signals alone are useless... but in combination with normals signal it performs really good.
Thank you for your feedback, this is nice to here, I initially started this channel to share ideas where people can tweak and learn and experiment just like you did. Good luck!
@@CodeTradingCafe does this community has a discord?
Unfortunately not, I wouldn't be able to manage 🙂
how do you do multistep predictions for this example?
Hi what do you mean by multistep?
hey I'm trying to add indicators it says "ATR missing arg 'low' and 'close' any ideas?
Hi, yes the atr function requires the arguments low close etc... try help(ta.atr) it should show you what input args are required.
Wavevlet transform can help to denoise time series
Hi, yes in general it does, I don't know if it would work for this dataset, there's only one way to find out I guess. Thanks for sharing.
What edge is expected from this, if it's expected at all?
Nothing at all, actually in another video I show why it's just an illusion that this works while actually for the moment it doesn't. (the video th-cam.com/video/lhrCz6t7rmQ/w-d-xo.html )
then this algorithm not is a prediction for future price?
It is, but the results are not up to the expectations, check some other videos they provide better indicators "support and resistance", or "price patterns". Good luck!
@CodeTradingCafe Can I use this model to predict the Index? My goal is to predict changes in NASDAQ with machine learning using of global market indices such FTSE, DAX, Nifty, etc.
I am generally very skeptical with neural networks. But I guess you can give it a quick try on historical data and see what it gives.
Thanks @@CodeTradingCafe. I think you are correct. I have tried with LSTM but is not working as expected for my use case. Let me try with bagged adaboost/xgbtree classification model.
Thanks for the feedback, good luck!
what activation function did you use?
Hey, good questions, I don't remember I will have to re-watch the video but I assume RELU
Great video, but how can I re-scale to the original numbers
Thank you, you can use inverse_transform() function for your scaler it brings back unscaled numbers.
I'm just gonna assume the problem is the usage of absolute values (actual prices) instead of relative growth
It is one big pitfall, among others but this one is deadly :)
@@CodeTradingCafe Then the train and test samples being mixed together instead of having a never before seen period of testing data 😂 I remember being hyped about it evaluating well, and then I tried it on actual values and the algorithm turned out to be absolute dog water. This prediction problem is the definition of an emotional rollercoaster
The worst feeling ever is the hype followed by deception 😂
why u r dropping the date column isnt it time series data
it is but since the time between candles is regular and anyway not needed here just the order of the candles is important, we can skip the datetime info.
Forget predicting exact price, why not just make it follow the trend and learn when to buy or sell. Trading is not about predicting price but rather making profits from the ongoing moves or am I wrong?
I am preparing something to predict the trend, I am not saying it's working though 😂
Does this work with a small dataset? Let's say less than 15?
Hi, no it shouldn't work neural networks require relatively larger amount of data
Thanks! Suscribed :)
Welcome 🙂 and thank you for your support
Nice work! Shouldn't we shift the predictions back by one step to better match the real data?
Hi, thank you. We are comparing the predicted future price to the real future price, so no shift needed here, because in real life if we use the model to predict future value... we can't shift values
cannot access the code
could you please send the link again?
Hi, the code link is still good, I just verified.
Hey yes it did.
Thank you so much.
Is lstm really that practical? watched most lstm tutorial video all predicted with trailing prices (lagging)
Hi there is a recent video check it out b it's the continuation of this one. Let me know what you think.
th-cam.com/video/lhrCz6t7rmQ/w-d-xo.html
@@CodeTradingCafe thank for your works on this, looking forward on your next video!
How would you do this without test data? For example if i want to se an prediction a week from now?
Hi, you would use the current week data as test data and see the prediction for the next day rather than the next week, but in this case you need to train the model also to detect next day or next week values, so the way we train the model influences how the model applies predictions.
@@CodeTradingCafe yes but since its a times series, how do I train lstm on future values rather than previous? Im new at this, sorry if I ask stupid questions
You don't train on future values because theoretically you don't have them yet, you train on past and you predict future using the same trained model.
@@CodeTradingCafe yes but this is the thing that im missing, what values do I fill testset with when I dont have access to it?
The current values are filled in the model so it will provide the future values.
Why didn't you used closed price for input
Hi, closing price wouldn't help at all, in Machine Learning you need relative values instead of absolute prices, so the model picks up the trend.
Predicting the closing is nearly impossible cuz its affected by multiple factors, moreover a good causality and correlation analysis will help improving the predicting model performance
Hi it's been a while... I agree completely it's impossible not even with the best data parameters at hand, next video might show it clearly.
so how can i use this model to predict the live market
It's not viable as is check this video first th-cam.com/video/lhrCz6t7rmQ/w-d-xo.html
Dear Sir,
How to retrieve original value from scaled value. I.e. y_pred, y_test
Hi, you can use the function inverse_transform()
@@CodeTradingCafe I have tried but it will give ValueError: non-broadcastable output operand with shape (506,1) doesn't match the broadcast shape (506,11)
@@CodeTradingCafe i have tried inverse_transform()
The shape of the data has to match if you fit the data on a dataframe of 11 features then you can inverse transform on the same data size, try fitting the transform on a single column first if you want to inverse transform a single column
@@CodeTradingCafe thank you so much sir you are awesome
Thank you for your nice video. One comment is that because the the train-set and test-set are splitted AFTER applied the MinMaxScaler, so, to visualize and calculate MSE, the target and predicted value need to be inverted using the same scaler. You can find that your plot (to compare real and predicted close price) shows the price in range $0 - $1 while it should be the original close price which are much higher (> $700). Would be very appreciated if you can share that code to invert the predicted value if you did it lol
You can use inverse_transform function for this... Just use it with the same scaler as you mentioned, good luck!
@@CodeTradingCafe i did inverse_transform function , but i am getting error like that
ValueError: non-broadcastable output operand with shape (503,1) doesn't match the broadcast shape (503,11), pls help how to solve this error
@@Lbvi you have to apply the function on dataframe of the same shape so you are replying only on the target which is one column however you have transformed 11 column dataframe.
@@CodeTradingCafe thanks you
@@CodeTradingCafe This was a very helpful video for build my first NN for stocks. I'm also struggling with this inverse. I've been at it for days and cannot get it to match the shape. Any suggestion on how you take the single y_pred and reverse it back to the original 11 column df that the scaler was applied to?
where will i get dataset for this code
Hi I usually use 2 sources yfinance and dukascopy both provide free data
In my opinion, your model needs regularization and cross validation. Seems highly overfitted to me. Some prunning maybe will work fine, bias & variance balance needs to be applied to.
Hi thanks a lot for your input, I think your are right regarding regularisation it's crucial, but I was expecting at least a small signal in the predictions which I haven't been able to get maybe I will revisit taking into account your comment. (If I remember correctly in a later video I did include some regularisation but with no major improvement, however I never did a full cross validation).
@@CodeTradingCafe I think your model needs to decrease a bit in complexity in order to avoid overlearning the training data. A consequence of that could be not being able to predict unseen data due to overlearning on the training set mentioned above.
I will probably try it with a much simpler polynomial regression(grade=3 maybe? If you actually plot it, you can see it has some similarities with the sigmoidal function you use in your RNN) and test my results, trying to find the best balance between the accuracy on the training data (I think you can check with .score()) while maintaining a healthy MAE ((mean_absolute_error(y_test,y_pred)).
If polynomial regresion doesnt work as expected then try to reduce complexity on your RNN (I don´t know exactly how, I will probably have to study it).
Sorry for my poor English. I am not native. I hope this helps.
Best,
Hi thank you, overfitting can be avoided by pruning some of the knots and we can also use less variables and layers, but I believe it is much more complex than this, neural networks are good in some cases but they perform very poorly in others (NN work will for CV for example).
I have done the same thing regularlization and cross validation and i get similiar results
Thank you for your feedback, yes I think it's kind of expected.
scaler should be applied only on trained otherwise will lead to data leakage
Theoretically scaler can be applied to both train and validation because if the model is trained on scaled data it will only work with scaled data. But I agree that to avoid data leakage scaling should be done independently on training and validation sets.
Hey , how is this even possible ?
We’re do we put the code does it work at pycharm
Pycharm, vscode, anaconda, Spyder... Any python interface works. But from your question I guess you haven't used python before, so I would start with some basic lessons, try my Python learning playlist for a quick start. Good luck!
@@CodeTradingCafe fare enough nah I’m studying IT HAHA but u know how the asset that’s were like should we change it in the code I just want a bot that really good and passed funded accounts if that’s possible anyway thanks for replying❤️
Anyone know how to scale the predicted values back up?
You need to use inverse_transform function: scaler.inverse_transform(predictions_scaled)
@@CodeTradingCafe thank you
Rtx 2070 подходит для lstm ?
Thank you!
All this is fine until a company/news event announces mid day that they have really great/bad news and your algo never trained on this sort of event because you're using historical prices that don't account for this as a feature. This happens all the time with anything that gets traded. Some new event happens and the price discontinuously jumps/falls to some new level that wipes out your account . I've never seen any machine learning prediction model take account of black swan events. Disingenuous.
And if I may ask, since you seem very knowledgeable in this matter, how often big news happen? and why big news should wipe your account if your strategy is locked in a low risk trade management system? please enlighten my disingenuous understanding.
Splendid video, how could someone use that for unknown data?
I don't really want to use it for live trading but i do want to forward test it!
Hi forward testing can be done by saving the trained model and then you feed live features to the model which provides forecast. But if you are not familiar with python and ML it might take longer than expected.
@@CodeTradingCafe Yes i was wondering how i could actually feed it live features, i made my own edit of your model which is pretty decent
@@CodeTradingCafe I'm using an edit that uses GRU instead of LSTM and i tweaked the hyper parameters a little, but i can't really find a way to have it provide forecast for live stocks. Normally i'm good at this stuff 🥲
@@CodeTradingCafe Also i was wondering if it would be smart to have a sort of "ultimate" forecast algorithm that uses a lot of different methods, neural networks or simple algorithms and use the (maybe weighted) average of the predictions as the forecast 🤣
It might be an approach to have an ensemble method, but it might be time consuming to build such a tool, and sometimes a box of troubles you're opening 😂
how to transform back to data got scaled before?
Hi, you can use the function inverse_transform from your scaling model to get back to original values. Good luck!
Przewidywania są stale opóźnione. Model obrał najlepszą strategię w postaci podążania za trendem, stąd podobieństwo wykresów porównujących przewidywania i wartości prawdziwe - zwyczajnie model przewiduje wartość wejściową z wejściowej, czyli przewiduje, ze cena przyszła będzie taka sama jak obecna. Dlaczego? Niech sobie każdy sam odpowie. A idioci łykają to jak pelikany. Jeszcze jak widać nie było wysypu milionerów.
Hi, thank you, actually this is discussed in detail in the next video, showing why it doesn't work: th-cam.com/video/lhrCz6t7rmQ/w-d-xo.html
Why should we used closed - open as target instead of using the close price directly
Hi, because absolute values don't mean anything for the algorithm. Think about it this way if I tell you the price is 1$ could you guess the next price? or if I tell you the price difference is +0.5$ in this case you could guess the price is going up and you are more likely to guess the next price.
@@CodeTradingCafeSo it means you are analysing the trend right?
I think that you are a little off. the last column shouldn’t be included.
Hi, the last column, did you mean the future price value? it has to be there it's our target for the model.
why to use these moving averages and other lagging indicators.... It looks you just get the same lagging indicators implemented by neural network... And it's not logical to use min max scaler when future prices can be and will be out of the training range sooner or later...
I agree for the scaler unless we're just doing a test to see if it works, definitely we can't deploy the same model.
Hi. I have error at this line
lstm_input = Input(shape=(backcandles, 8), name='lstm_input')
The error is:
TypeError: descriptor '_values' for 'BaseContainer' objects doesn't apply to a 'RepeatedCompositeFieldContainer' object
Can you kindly check and advise?
Thank you.
Hi, please make sure you ran the previous cells correctly to prepare the data.
@@CodeTradingCafe Thanks. it works now.
No module named 'pandas_ta'
Hi you need to install it, try in your notebook "!pip install pandas_ta"
@@CodeTradingCafe Thanks
just plotting and looking they are close does not mean its accurate enough ....u need metrics ok, even if u do training and testimg..u need to assess ur accuracy
Hi yes I agree, there is another video showing more realistic analysis check it out th-cam.com/video/lhrCz6t7rmQ/w-d-xo.html
Time series must not be shuffled
Hi thank you, I don't think I shuffled I usually split in order, I will reverify
@@CodeTradingCafe no need to check, my fault. The fit function generates batches that can be shuffled and, in theory, should preserve the sequential order of the TS in each batch. Anyway, thanks for the quality content.
@@kadourkadouri3505 ah ok, thank you for the feedback.
It's not a prediction, it's only a lag.
yes unfortunately!
this is awful. when happen rise or fall, this will predict price
No it doesn't, wait for next video couple of days... It's an excellent way to show what works and why it doesn't
@@CodeTradingCafe my english is not well. tnx i waiting for it
No problem, I hope you will enjoy coding on this channel. Good luck!
Great video. THANK YOU!
Thank you for your support!
for anyone receiving error: module 'keras.optimizers' has no attribute 'Adam'
solution: keras.optimizers.Adam()
Thank you I guess it depends on the libraries version
@@CodeTradingCafe of course no problem