FYI you didn't take into account the initial bad data that SMA12 causes. That is, when setting window_size and frame_bound instead of setting frame_bound start equal to the widow_size you should set it to windows_size + 11(or higher) because the first 11 values of SMA are going to be bad values due to not having enough data to calculate. Otherwise, during training you'll be having 11 bad observations per round. So window_size=12, frame_bound=(12,50) should be window_size=12, frame_bound=(23,50).
I think it's a huge learning curve but trying to get there! Still a bunch I'm learning around it everyday! Rebuilt the entire environment for trading this morning and realised I wasn't iterating through my data *doh* either way, trying to share the stuff I learn along the journey!
Really interesting series of videos! Clarified a lot of things I've been struggling to get my head around! Looking forward to more videos in this series, great work!
You don't need add_signals and CustomEnv if you calculate all your indicators in the Dataframe before create the environment and then pass the data to the environment?
Well done! One thing that would be interesting to see would be the implementation of the model where you mock stream “real-time” data to the model and allow it to potentially return a buy or sell signal so that it can be closer to real world usage. Thanks!
That would actually be awesome, some of the subscribers mentioned hooking it up to the live Binance API and running it using that. I've got three model videos planned in this series: 1. Adding the training callback 2. Hooking up to the Binance API for real time data 3. (Possibly) trying out live trading on a security
Amazing video! I learned a lot. Could you also talk about choice of hyper-parameters such as learning rate, number of neurons, number of layers, etc. and maybe how to use callbacks to get a better model.
@@NicholasRenotte Been enjoying your videos or RL for a while... Did you ever get around to creating the next videos with the callback, hyperparameters, etc? That would really take things to the next level! Great stuff!
This is a great series of videos! What's the right kind of callback to implement a break and save a good model? I can find a way to do it in the documentation based on reward, but should it be done with the explained variance instead? Thanks again 🙏🏻
I saw several people below ask this same question I had, but not answered. I was curious as to why you're using the low value. Wouldn't you want to use the close value, or am I misunderstanding how it works? Thanks, love the videos
Hey Nicholas, huge thanks! I stumbled upon your RL videos while I was searching for some tutorials for RL. I’ve built tons of custom indicators for LSTM models and I’ve gone deep down that rabbit hole. I’m excited to try applying them to RL. I actually have a really good custom loss function for trading that beast mse & Mae. Hit me up if you’re curious about that one. Cheers! And again, great content!
Sick!. In a next tutorial could you use binance's API to automatically get data to python and dinamically plot it. It sounds simple but it can be challenging.
Can you please do a short video on how to get trading signals dynamically based (instead of just backtesting). If you could integrate it to a live exchange like Binance, it would be awesome aswell! Awesome vid!
Thanks so much. Such a great starting point for this technique. Shows you the playground and gives you the confidence to jump in. Couple of questions: Does this mean you’re using low as the price? Why is that? Is there a way to have the custom overlay introduce a “hold” action?
was thinking of this too, lesser trading transaction (holding your current trading position) might had increased your profits/losses, but definitely will reduce your trading commission charges
Thanks a bunch @Josh, yep using low as the price here. TBH it was probably a brain fart moment, could easily switch out the column for Close as well though!
Great video mate! I just wanted to ask what is probably a silly question, but apart from the way a system is 'trained', is there a fundamental difference between using RL and a genetic algorithm to optimise a strategy? I'm new to RL so forgive the naivety!
So this is my ELI5 explanation for this: GAs: Starts by creating a random set of solutions > use natural selection to 'kill off' not so great solutions > apply mutation and crossover to 'breed' ultimately the best solutions > natural selection again > and so on > apply best solution in prod RL: Explores an environment > learns a value-function that attempts to maximise reward > exploits this value function to maximise reward > balances exploring the environment and exploiting it to ultimately make the most of possible actions > apply value function in prod
So I just took tomorrows price and made it my custom indicator. Because I figured if the model can't decide what to do when it knows the future price then this model is probabply not any good at all. And it absolutely didn't know what to do after 500k timesteps of training. It just buys all the time, all the dots are green. Am I doing something wrong? I have just two columns in my signal_features: price for today and price for tomorrow. No other changes relative to your code. Thanks!
This is a super basic model at this stage, I wouldn't be expecting maximum performance without some additional tweaks. Kind of weird you're only getting buy signals though, is that on historical data as well?
@@NicholasRenotte I found an error in my code, accidentally removed a line where we set df index. Did not affect the end result though. Now buys and sells are somewhat random, not all buys, but it's still inadequate. And by the way I changed the second number from future price to future price difference. So the model just needs to return 0 if the number is less than zero and 1 if it is greater than zero, but it's unable to do even that. Which is a little disappointing to me. Is there some easy tweaks I could do to make this trivial model work?
Great tutorial! I learned a lot from PT1 and PT2! Thoughts on how to increase the # of actions to more than Buy/Sell? Next vid:) Thanks again for the content, keep em coming!
@@NicholasRenotte I am working on a project with options data, actions I would like to include for the environment, [Buy, Sell, Hold, Short_Sell, Cover, [Sell-Short_Sell, Cover-Buy]]. Expanding beyond buy/sell actions would be great. Appreciate you taking time to respond!
First of all thank you very much for these awesome videos. As a begginner in this amazing world of RL (having finance background) I have some questions concerning the reward function of this agent. Does the reward == performance of the next day? or does it holds position?
@@NicholasRenotte Well more of an advice -- I'm trying to create my own start up and feel like ML would be useful someday down the line. As a data scientist, do you have any advice for a new start up?
What is the code we can use to actually predict for the next day, what a trade should be? How can we actually input last few days data and have it return BUY or SELL?
I've looked for such videos on RL and you are the only one that covers the right topic with such clear explanations. In this specific video I wonder whether would be OK to run XGBoost, an SVM and an LSTM (also pattern indicators like engulfing) and give the results also as an indicator to the RL agent. If so I would appreciate a video like this but anyway you are one of the bests on this topic, thanks Nico!
@@NicholasRenotte Thank you Nico, I already did it and the result was 1.143 which means 14% (it's a tequila brand in the mexican stock market) but in the graph from nearly 175 points only 4 out of them are red, so even with the 14% I think something must be wrong, a further explanation on the interpretation when you get there will be highly appreciated. And again, thanks a lot!
Thank you Nicholas for your fascinating job!!!! When I'm training the model I have : | explained_variance | 0.986 | | fps | 150 | | nupdates | 4927500 | | policy_entropy | 2.13e-05 | | total_timesteps | 246375 | | value_loss | 0.0186 | BUT the following logs are TOTAL disgusting. Will it keep that particular good settings or am I supposed to wait for a more stable convergence?
Thank you so much Nicholas! These vids have been amazing. I have a question. Can you work with panel data in this framework? As in, we are optimizing for GME, but how does the data model look like when we have two different stocks? Multi-index? Or maybe train different agents? Thanks in advanced. Like the content a lot. Here before you blow up :)
Heya @Adam, I believe this only works with a single asset (in it's current state). Butttt you could train multiple agents sequentially by looping through the frame and filtering by stock or asset! e.g. Psedocode for stock in df['stock_name'].unique(): data = df[df['stock_name'] ==stock] ## train model Want a vid on this? And, thanks a ton for checking out the video, stoked you liked it!
@@NicholasRenotte Thank you for replying! Yeah! Seemed like the framework works for an asset only, hence one would have to iterate over the assets. The thing is that maybe it doesn't take into account the balancing of how much is invested into each stock. Maybe normalizing using the reward function would be a good idea for asset allocation. Apart from the expected return, of course. Thank you for these videos. I've been trying to get into RL and DRL since it's probably the thing I find most interesting and mind-bending in Machine Learning and AI. Your videos have introduced me to it, from a practical perspective also, in a smooth way.
@@aliriano15 agreed, I actually had a thought to use something like linear programming for optimal asset allocation, possibly leveraging something like the efficient frontier. And yes! I struggled with RL for so long, there wans't anything practical enough to allow a newbie to jump into it. I've got a lot more planned which takes it past the beginner level but still with a practical spin to it! As an aside, feels good to be getting back into finance after a while, all of the stuff from university is coming back but it's now a whole lot more intesting with ML involved!
@@NicholasRenotteplease don’t let this thought die! I’ve been looking all over for a tutorial that links RL and optimal asset allocation (portfolio allocation). BTW: awesome content, love your vids!! Tks
Your video are wonderful. How would you do a stock prediction for a number of companies at a go....say 100 companies? If its not too much do you have a video on deep reinforcement relearning
I’ve created a neural network indicator and all I have to do is determine which input values produce the the most accurate output... There has to be some system that can do this automatically.
I missed something. At 18:48, why did you set prices to be the 'Low"? You can't really enter trades based on the low price, because that's not determined until the day is over. Right?
Heya @Wayne, hmmm, agreed! I think it would've been closer to reality (or at least easier to implement) if I chose Close prices. Keep in mind you could do this with real time bid/asks as well, ideally for this you would want significantly more granular data.
@Nicholas Renotte its a great tutorial. But I doubt that this approach is a real price prediction as you use indicators that were calculated on the same prices that the model tries to predict. I think that "price" or Y parameter should be the price of the next tick (row of the dataframe). Shouldn't 'Low' or 'Close'' price be shifted down?
Hi S. A. it only has the previous windowed_timesteps to attempt to produce an action! We're not actually trying to forecast the projected price here but rather what the agent should do given the current set of prices.
Hi Nick, thanks for uploading this ado, it's very helpful , is it possible that you can make the ado that we can implement the training model into the real world trading like live trading account. Thanks again for sharing.
This is interesting. Does gym A2C shift the target field? It looks like your target is "low". But if you don't shift(-1) that column are you really predicting anything? Or are you predicting what the low was on the same day for each data row?
Hey Nicholas, Great Content for RL . could you show us an example on how to take this learnt Model and use it in real environment as in real time trading scenario, that would be a great tutorial, since its not there in any channel.
How do I force it though to not make short trades? I only want to train it to make long trades. Also, I would want it to sell before it tries to buy again (Buy then Sell). Is this stuff configurable with Gym-Anytrading?
Great tutorial, thank you... Can you tell me what is your setup for your tutorial? Python environment version and any reason for tensorflow 1.1.5... Thanks in advance
Great video learned so much thanks! Not sure how you can use this for Bitcoin though since it appears gymanytrading has only stocks and FTX default environments? maybe a good short video would be how to extend this to a crypto environment.
Just found an answer to this question. lagged changes are not needed for independent variables because reward (target variable) is calculated using one step ahead target variable. so all is right in this universe..
Hi Nicolas, Thanks for the great tutorial. Could you please explain how we can input number of the shares and the constraint for balance in each episode.
Amazing video! I'm trying to implement this but in Keras-rl2, however I'm having some trouble getting LSTM model implemented, do you have a guide somewhere I can refer to for such a task, much thanks!
Hi Nicholas, I am wondering if there is option to produce performance report? specifically, P&L ,Trade list, trade statistics(avg. win ,avg loss, total short/long ect) Detailed equity curve . Lets assume I have hundreds of data pre processors or custom indicators. Will the algorithm use all and likely to curve fit, or select the most frequently occurring instances ? Lastly, it would be really useful if you can have an example for active portfolio diversification using 10-20 different shares.
@Nicolas Renotte thanks man, great explanation. could you please do a more "advanced" series where you cover callbacks for stoping training.. and maybe how to connect this to a live data fee (like a broker's API) that would be the real applications for this. A tutorial on how to build your own trading Env would also be very cool in case one wants to take different conditions into account Thanks again man
Thanks for this amazing video. Could you please explain some troubleshooting options and hyper-parameter tuning available when the model is apparently not learning anything (explained variance and loss move wildly and stay far from 1 and 0 respectively even after 100-200k timesteps). What are the firs things to try? Data preprocessing, choosing a different model or policy, changing the window and frame sizes, etc. Thank you.
@@NicholasRenotte I'm not sure I see over-fitting. Even at around 10k steps the variance varies between, for example, -8.82e-05 (at 4000 steps) and 0.741 (4500 steps).
hi 2 questions. If i wanted to do portfolio optimization using RL, is that much different to this? And secondly, what is the baseline model you'd compare this to? Would you just take the expected return as average return over the period?. Actually, I also heard that we shouldn't be using price in ML because it is non statiionary. we should just be using returns and ideally really just log returns.
Take a look at a library called FinRl they've actually got a portfolio opt environment already setup! I'm going to be answering some of the other q's in the new trading bot tutorial!
@@NicholasRenotte aha funnily enough im using it rn. it's great! 1 question; how can i get the agent to predict what to do tomorrow? Do i give it today's values when the market closes as well as yesterday's closing prices? (so it can calculate ROI). I thought to give it expected returns but i think that's wrong because id have to predict all technical indicator values
hi, is there any possibility to access variable from explained_variance? to stop learning when it reaches a good value. Another question, is it normal for the location and number of points of purchase and sale to always be the same? they are never reduced in quantity, they only change action in the same location. Thank you very much for the help you gave me in the later video.
Got a better way to do it using reward, will show how to do it in the next RL video that's coming up! I'm actually going to be rebuilding the environment from scratch, I noticed a few weird things in it myself tbh.
@@NicholasRenotte great! I'll be waiting for that video! I think I understand that gym-anytrading environments are for general analysis and not for use in a real-time bot.
@@guillermogallogarcia9217 awesome, it's out now: th-cam.com/video/Mut_u40Sqz4/w-d-xo.html let me know how you go! Edit: FYI this is the RL course, I've got the trading envs coming soon!
I would remove the rows with missing mean values with *dropna* instead of setting them to 0. I assume the zero values could train the RL into wrong directions?
How do you make a custom environment to buy & exit, sell short & cover? Can it be trained select approptiate trade size and to scale in and out of trades?
It would be really cool to see how such a model can be connected to real-time data. I don't really understand how i can use model.predict() without creating an env, to predict() only on given data... maybe you have some tutorial?
Thanks a lot for the Video Nicholas I have two questions to ask you : In "info" output there are three arguments : "total_reward" , "total_profit" and "position" -- What's the difference between "total_reward" and "total_profit" ? -- what's the meaning of "position : 1" (for me there are many positions...is it saying that the major position taken is LONG here ?
@@NicholasRenotte Why did you delete you're answer ? (don't be shy if you don't know...but you can say "I think that it is" even if you think that we expect you to be EXPERT which is not TRUE but it's not a shame . We are learning all together . What we expect from you is to make videos with non basics errors that's it....:)
@@ihebbibani7122 oh no, I had a laugh at myself when you replied 🤣. I responded but I think I missed the reply, my bad. I actually started digging into the code for you, profit is meant to be dollar profit, reward is percentage return so .98 means a loss of 2%. It's not super well documented unfortunately so took a little while.
So once you train this agent, how do you actually run the agent live? I've used the freq-trade bot to trade before, but I don't understand how to actually use the trained agent to do trading.
Why did you use 'Low' values as prices? The agent shall make a decision at the 'Open' price of the day. We never know about the 'Low' of the day before the day is closed
Hi Nick, Thanks for your video, Somehow, i cannot run the code in jupyter notebook, as the code uses tensorflow 1.x and googlecolab only support googlecolab tensorflow 2.x and which tensorflow 2.x only works with stablebaselines3, can you kindly how how can we upgrade the code to tensorflow 2.x with stablebaselines3 ? Thanks
@@NicholasRenotte Can you please do a short video on how to get trading signals dynamically based (instead of just backtesting). If you could integrate it to a live exchange like Binance, it would be awesome aswell! Thanks!
Thanks alot for creating this video. Can you please also guide us to add one more action space i.e "Hold" or do nothing. As currently the agent trades on every single day. Thanks in advance
Great video, I love the way you teach. I've tried your code using different stocsk, but when using GME the learn stabilize(explained variance always above 0.95) after some minutes while when using other stocks it doesn't stabilize, da you have any idea why this happen?
Heya @Marco, I think it's always going to vary depending on what security you use. I think the GME model is solving as it's learning to trade the short squeeze, but really it's seeing it and learning to go Long. Have you watched the follow up video, I go through how to solve and train on more stable data: th-cam.com/video/q-Uw9gC3D4o/w-d-xo.html
I would love to know how i can give the bot multiple ticker information at once and get seperate but/sell outputs for every symbol. I think there is some interaction between different stocks/cryptos, so I feel like it would be beneficial to give the bot more information
Heya @leader c, done, I show how to do it in this video: th-cam.com/video/nRHjymV2PX8/w-d-xo.html and in this code base: github.com/nicknochnack/StableBaselinesRL/blob/main/Stable%20Baselines%20Tutorial.ipynb
I keep getting the message ValueError: Error: The environment must be vectorized when using recurrent policies. When I try to run the model.predict(obs) line. Any ideas what I can do to fix this. I used the custom environment like in the video to use custom indicators.
I tried it with 8 years of daily Facebook data. That's a total of about 2100 trading days. I used 60% for training and the rest for testing. 1m learning steps. 0.5 profit. Did I use enough steps? The explained variance doesn't stabilise.
One thing is bothering me. We got through whole process and ended up with agent with some experience. How it is possible to save this data and use in real trading bot?
Heya @Dawid, you can save the model using the save function: E.g. model.save('mysavedmodel') Then reload it when you need it alongside the environment: A2C.load('mysavedmodel', env=env)
Not quick but it is possible! I actually found it easier to just save the model every x thousand steps and grab the best one. Got a vid planned for this!
Why are you using the low column? I would use the close column in real life we don't know the low until the end of the period but it's too late to buy at that price
@@NicholasRenotte Hi Sir; I am working on this project of yours and wish to enhance the model by adding a few more indicators and connecting it to live account. Indicators part is done. I don't understand where should I place the buy and sell functions of the exchange and I previously made an a2c model wherein I have the actor, critic and shared model saved on disk, Mind guiding how to load and use it in your environment, if that's a good way to run from bugs of that environment. Kind regards, Tasneem
FYI you didn't take into account the initial bad data that SMA12 causes. That is, when setting window_size and frame_bound instead of setting frame_bound start equal to the widow_size you should set it to windows_size + 11(or higher) because the first 11 values of SMA are going to be bad values due to not having enough data to calculate. Otherwise, during training you'll be having 11 bad observations per round. So window_size=12, frame_bound=(12,50) should be window_size=12, frame_bound=(23,50).
Ah you're right, I just zeroed it out rather than starting from when the full SMA commences. Solid pickup @Carlos!
Subscribed! Love that someone shows how to use RL in real-world applications
I think it's a huge learning curve but trying to get there! Still a bunch I'm learning around it everyday! Rebuilt the entire environment for trading this morning and realised I wasn't iterating through my data *doh* either way, trying to share the stuff I learn along the journey!
Really interesting series of videos! Clarified a lot of things I've been struggling to get my head around! Looking forward to more videos in this series, great work!
Thanks so much @Mark, a lot more planned in the RL Trading series!
You don't need add_signals and CustomEnv if you calculate all your indicators in the Dataframe before create the environment and then pass the data to the environment?
Well done! One thing that would be interesting to see would be the implementation of the model where you mock stream “real-time” data to the model and allow it to potentially return a buy or sell signal so that it can be closer to real world usage. Thanks!
That would actually be awesome, some of the subscribers mentioned hooking it up to the live Binance API and running it using that. I've got three model videos planned in this series:
1. Adding the training callback
2. Hooking up to the Binance API for real time data
3. (Possibly) trying out live trading on a security
@@NicholasRenotte Those 3 would be fantastic, keep up the amazing work!
@@NicholasRenotte When will you do it? It has been 7 months? :))
Amazing video! I learned a lot.
Could you also talk about choice of hyper-parameters such as learning rate, number of neurons, number of layers, etc. and maybe how to use callbacks to get a better model.
Definitely will schedule it for a next video @Ixonok!
@@NicholasRenotte Been enjoying your videos or RL for a while... Did you ever get around to creating the next videos with the callback, hyperparameters, etc? That would really take things to the next level! Great stuff!
Getting to learn a lot about RL! Looking forward to the upcoming parts on Python Trading Series:)
YESS, awesome @Vishal! I'm getting back to some RL vids over the next couple of weeks stay tuned!
This is a great series of videos! What's the right kind of callback to implement a break and save a good model? I can find a way to do it in the documentation based on reward, but should it be done with the explained variance instead? Thanks again 🙏🏻
I saw several people below ask this same question I had, but not answered. I was curious as to why you're using the low value. Wouldn't you want to use the close value, or am I misunderstanding how it works? Thanks, love the videos
Hey Nicholas, huge thanks! I stumbled upon your RL videos while I was searching for some tutorials for RL. I’ve built tons of custom indicators for LSTM models and I’ve gone deep down that rabbit hole. I’m excited to try applying them to RL. I actually have a really good custom loss function for trading that beast mse & Mae. Hit me up if you’re curious about that one. Cheers! And again, great content!
Thanks so much @Alec, definitely, let's chat on Discord: discord.gg/mtTTwYkB29
Sick!. In a next tutorial could you use binance's API to automatically get data to python and dinamically plot it. It sounds simple but it can be challenging.
Oooooh yeah, that would actually be sweet!! Might do that this weekend, thoughts?
vote for this
@@docdies4376 yah? Alright, I've got two other ideas that I'm going to be working on tomorrow. Might sneak this one in as well!
Can you please do a short video on how to get trading signals dynamically based (instead of just backtesting). If you could integrate it to a live exchange like Binance, it would be awesome aswell! Awesome vid!
Yup, will be doing it as part of our next big project.
Thanks so much. Such a great starting point for this technique. Shows you the playground and gives you the confidence to jump in.
Couple of questions:
Does this mean you’re using low as the price? Why is that?
Is there a way to have the custom overlay introduce a “hold” action?
was thinking of this too, lesser trading transaction (holding your current trading position) might had increased your profits/losses, but definitely will reduce your trading commission charges
Thanks a bunch @Josh, yep using low as the price here. TBH it was probably a brain fart moment, could easily switch out the column for Close as well though!
Hai Nick, Thanks for your video. May god bless you.
So cool! Do you think its possible to change the reward function to something like sharpe ratio?
You’d have to implement it yourself, there’s some good papers out there on it
What @Jordan said ^. You can formate it in a custom environment but I don't think it's out of the box with Anytrading.
Thanks for the great tutorial. I am just thinking about relearning during the production. Do you have a tipp?
Great video mate! I just wanted to ask what is probably a silly question, but apart from the way a system is 'trained', is there a fundamental difference between using RL and a genetic algorithm to optimise a strategy? I'm new to RL so forgive the naivety!
So this is my ELI5 explanation for this:
GAs: Starts by creating a random set of solutions > use natural selection to 'kill off' not so great solutions > apply mutation and crossover to 'breed' ultimately the best solutions > natural selection again > and so on > apply best solution in prod
RL: Explores an environment > learns a value-function that attempts to maximise reward > exploits this value function to maximise reward > balances exploring the environment and exploiting it to ultimately make the most of possible actions > apply value function in prod
We need a video on how to connect it Binance and go brrrr pls!!!!!
Besides that, awesome work! Thank you. 😊
So I just took tomorrows price and made it my custom indicator. Because I figured if the model can't decide what to do when it knows the future price then this model is probabply not any good at all. And it absolutely didn't know what to do after 500k timesteps of training. It just buys all the time, all the dots are green. Am I doing something wrong? I have just two columns in my signal_features: price for today and price for tomorrow. No other changes relative to your code. Thanks!
This is a super basic model at this stage, I wouldn't be expecting maximum performance without some additional tweaks. Kind of weird you're only getting buy signals though, is that on historical data as well?
@@NicholasRenotte I found an error in my code, accidentally removed a line where we set df index. Did not affect the end result though. Now buys and sells are somewhat random, not all buys, but it's still inadequate.
And by the way I changed the second number from future price to future price difference. So the model just needs to return 0 if the number is less than zero and 1 if it is greater than zero, but it's unable to do even that. Which is a little disappointing to me.
Is there some easy tweaks I could do to make this trivial model work?
Great tutorial! I learned a lot from PT1 and PT2! Thoughts on how to increase the # of actions to more than Buy/Sell? Next vid:) Thanks again for the content, keep em coming!
Ya, will definitely do some more in this space, soooo glad you liked it @mtlong! What other actions would you add apart from maybe hold?
@@NicholasRenotte I am working on a project with options data, actions I would like to include for the environment, [Buy, Sell, Hold, Short_Sell, Cover, [Sell-Short_Sell, Cover-Buy]]. Expanding beyond buy/sell actions would be great. Appreciate you taking time to respond!
trader_env.py, came across class Actions and class Positions to expand on the actions/positions. Need to add the logic that goes with new actions:)
@@mtlong8403 ah, awesome! Didn't really think of some of those, awesome suggestion though!
First of all thank you very much for these awesome videos.
As a begginner in this amazing world of RL (having finance background) I have some questions concerning the reward function of this agent. Does the reward == performance of the next day? or does it holds position?
Performance is linked to profit from trades and yep, it holds as well I believe!
I’ve learnt so much from your channel!
AWESOME, super glad to hear @Chuks! Anything else you'd like to see?
@@NicholasRenotte Well more of an advice -- I'm trying to create my own start up and feel like ML would be useful someday down the line. As a data scientist, do you have any advice for a new start up?
@@chuksgrinage7238 actually that would make an awesome video! Alright, I'll get something together on it!
@@NicholasRenotte ayyyyyyyyyy
@@chuksgrinage7238 yeeeeahhyaa!
One of my favourite tutorials
Thanks soo much @Ashley!
What is the code we can use to actually predict for the next day, what a trade should be? How can we actually input last few days data and have it return BUY or SELL?
I've looked for such videos on RL and you are the only one that covers the right topic with such clear explanations. In this specific video I wonder whether would be OK to run XGBoost, an SVM and an LSTM (also pattern indicators like engulfing) and give the results also as an indicator to the RL agent. If so I would appreciate a video like this but anyway you are one of the bests on this topic, thanks Nico!
Could definitely do that @Jorge, going to be doing a ton more in this space once the Python course is done :)
@@NicholasRenotte Thank you Nico, I already did it and the result was 1.143 which means 14% (it's a tequila brand in the mexican stock market) but in the graph from nearly 175 points only 4 out of them are red, so even with the 14% I think something must be wrong, a further explanation on the interpretation when you get there will be highly appreciated. And again, thanks a lot!
Thank you Nicholas for your fascinating job!!!!
When I'm training the model I have :
| explained_variance | 0.986 |
| fps | 150 |
| nupdates | 4927500 |
| policy_entropy | 2.13e-05 |
| total_timesteps | 246375 |
| value_loss | 0.0186 |
BUT the following logs are TOTAL disgusting.
Will it keep that particular good settings or am I supposed to wait for a more stable convergence?
it was very nice video. Can you please create some video using FinRL ? That would be helpful
Thank you so much Nicholas! These vids have been amazing.
I have a question. Can you work with panel data in this framework? As in, we are optimizing for GME, but how does the data model look like when we have two different stocks? Multi-index? Or maybe train different agents?
Thanks in advanced. Like the content a lot. Here before you blow up :)
Heya @Adam, I believe this only works with a single asset (in it's current state). Butttt you could train multiple agents sequentially by looping through the frame and filtering by stock or asset!
e.g. Psedocode
for stock in df['stock_name'].unique():
data = df[df['stock_name'] ==stock]
## train model
Want a vid on this? And, thanks a ton for checking out the video, stoked you liked it!
@@NicholasRenotte Thank you for replying!
Yeah! Seemed like the framework works for an asset only, hence one would have to iterate over the assets.
The thing is that maybe it doesn't take into account the balancing of how much is invested into each stock. Maybe normalizing using the reward function would be a good idea for asset allocation. Apart from the expected return, of course.
Thank you for these videos. I've been trying to get into RL and DRL since it's probably the thing I find most interesting and mind-bending in Machine Learning and AI. Your videos have introduced me to it, from a practical perspective also, in a smooth way.
@@aliriano15 agreed, I actually had a thought to use something like linear programming for optimal asset allocation, possibly leveraging something like the efficient frontier. And yes! I struggled with RL for so long, there wans't anything practical enough to allow a newbie to jump into it. I've got a lot more planned which takes it past the beginner level but still with a practical spin to it!
As an aside, feels good to be getting back into finance after a while, all of the stuff from university is coming back but it's now a whole lot more intesting with ML involved!
@@NicholasRenotte Indeed!
Thank you so much for your content! I'm hooked on your channel haha
@@NicholasRenotteplease don’t let this thought die! I’ve been looking all over for a tutorial that links RL and optimal asset allocation (portfolio allocation). BTW: awesome content, love your vids!! Tks
Your video are wonderful.
How would you do a stock prediction for a number of companies at a go....say 100 companies?
If its not too much do you have a video on deep reinforcement relearning
I’ve created a neural network indicator and all I have to do is determine which input values produce the the most accurate output... There has to be some system that can do this automatically.
I missed something. At 18:48, why did you set prices to be the 'Low"? You can't really enter trades based on the low price, because that's not determined until the day is over. Right?
Heya @Wayne, hmmm, agreed! I think it would've been closer to reality (or at least easier to implement) if I chose Close prices. Keep in mind you could do this with real time bid/asks as well, ideally for this you would want significantly more granular data.
@Nicholas Renotte its a great tutorial. But I doubt that this approach is a real price prediction as you use indicators that were calculated on the same prices that the model tries to predict. I think that "price" or Y parameter should be the price of the next tick (row of the dataframe). Shouldn't 'Low' or 'Close'' price be shifted down?
Hi S. A. it only has the previous windowed_timesteps to attempt to produce an action! We're not actually trying to forecast the projected price here but rather what the agent should do given the current set of prices.
Hi Nick, thanks for uploading this ado, it's very helpful , is it possible that you can make the ado that we can implement the training model into the real world trading like live trading account.
Thanks again for sharing.
Hello Nicholas, Great Video! I was wondering if we can pass an image in the custom env?
This is interesting. Does gym A2C shift the target field? It looks like your target is "low". But if you don't shift(-1) that column are you really predicting anything? Or are you predicting what the low was on the same day for each data row?
yes I think this needs to be shifted..
Hey Nicholas, Great Content for RL . could you show us an example on how to take this learnt Model and use it in real environment as in real time trading scenario, that would be a great tutorial, since its not there in any channel.
Ya! Thats the plan, going to drill into RL trading and a few other niche topics over the coming weeks!
How do I force it though to not make short trades? I only want to train it to make long trades. Also, I would want it to sell before it tries to buy again (Buy then Sell). Is this stuff configurable with Gym-Anytrading?
Great tutorial, thank you... Can you tell me what is your setup for your tutorial? Python environment version and any reason for tensorflow 1.1.5... Thanks in advance
Heya @MrHtungh, I'm using Analconda with Python 3.7.3. I used TF 1.15 as the reinforcement library requires a tensorflow version below v2.
@@NicholasRenotte "Analconda"?.... well that's one way to make extra money on the side....
@@johnquarto 🤣🤣🤣 gotta make that GPU money some how!
Great video learned so much thanks! Not sure how you can use this for Bitcoin though since it appears gymanytrading has only stocks and FTX default environments? maybe a good short video would be how to extend this to a crypto environment.
Hey this is a great video. I think the price variable should be shifted up by one for training right. or to say dependent variables should be lagged.
Just found an answer to this question. lagged changes are not needed for independent variables because reward (target variable) is calculated using one step ahead target variable. so all is right in this universe..
Hi Nicolas, Thanks for the great tutorial. Could you please explain how we can input number of the shares and the constraint for balance in each episode.
Working on this now, I'm actually including some balance management inside the new custom environment.
This is a question to know why are you using the 'Low' column instead of the 'Close' on the line prices in 19:06?
Amazing video!
I'm trying to implement this but in Keras-rl2, however I'm having some trouble getting LSTM model implemented, do you have a guide somewhere I can refer to for such a task, much thanks!
Hi Nicholas,
I am wondering if there is option to produce performance report? specifically, P&L ,Trade list, trade statistics(avg. win ,avg loss, total short/long ect) Detailed equity curve .
Lets assume I have hundreds of data pre processors or custom indicators. Will the algorithm use all and likely to curve fit, or select the most frequently occurring instances ?
Lastly, it would be really useful if you can have an example for active portfolio diversification using 10-20 different shares.
@Nicolas Renotte thanks man, great explanation.
could you please do a more "advanced" series where you cover callbacks for stoping training.. and maybe how to connect this to a live data fee (like a broker's API) that would be the real applications for this.
A tutorial on how to build your own trading Env would also be very cool in case one wants to take different conditions into account
Thanks again man
Definitely! I'm going to be doing a continuing series on it, all the vids will be here: th-cam.com/play/PLgNJO2hghbmjMiRgCrpTbZzq0o_hINadE.html
Thanks for this amazing video.
Could you please explain some troubleshooting options and hyper-parameter tuning available when the model is apparently not learning anything (explained variance and loss move wildly and stay far from 1 and 0 respectively even after 100-200k timesteps). What are the firs things to try? Data preprocessing, choosing a different model or policy, changing the window and frame sizes, etc.
Thank you.
Heya @A N, at any stage did the model get a higher than average explained variance? 100-200k steps might be heading into the realm of overfitting.
@@NicholasRenotte I'm not sure I see over-fitting. Even at around 10k steps the variance varies between, for example, -8.82e-05 (at 4000 steps) and 0.741 (4500 steps).
@@AsafNaymarkadv could also be a difficult dataset to solve, GME in this case has the huge spike which throws everything out.
hi 2 questions.
If i wanted to do portfolio optimization using RL, is that much different to this?
And secondly, what is the baseline model you'd compare this to? Would you just take the expected return as average return over the period?.
Actually, I also heard that we shouldn't be using price in ML because it is non statiionary. we should just be using returns and ideally really just log returns.
Take a look at a library called FinRl they've actually got a portfolio opt environment already setup! I'm going to be answering some of the other q's in the new trading bot tutorial!
@@NicholasRenotte aha funnily enough im using it rn. it's great! 1 question; how can i get the agent to predict what to do tomorrow? Do i give it today's values when the market closes as well as yesterday's closing prices? (so it can calculate ROI). I thought to give it expected returns but i think that's wrong because id have to predict all technical indicator values
This channel is amazing
Thanks a bunch @David!
Hey, Nicholas! This works with historical data right. If I have model which predicting, how can I implement RL into it?
you are the best! Thanks for sharing
Great video. Could you please make a video about creating a custom environment for trading multiple stocks?
You got it @Thuong, going to be doing a bigger series on ML for finance soon!
There is a course by quantra dealing this. ... reinforcement learning for trading is the name
hi, is there any possibility to access variable from explained_variance? to stop learning when it reaches a good value.
Another question, is it normal for the location and number of points of purchase and sale to always be the same? they are never reduced in quantity, they only change action in the same location. Thank you very much for the help you gave me in the later video.
Got a better way to do it using reward, will show how to do it in the next RL video that's coming up! I'm actually going to be rebuilding the environment from scratch, I noticed a few weird things in it myself tbh.
@@NicholasRenotte Please do it, I have the same problem with exṕlained_variance. Thank you very much
@@NicholasRenotte great! I'll be waiting for that video! I think I understand that gym-anytrading environments are for general analysis and not for use in a real-time bot.
@@guillermogallogarcia9217 awesome, it's out now: th-cam.com/video/Mut_u40Sqz4/w-d-xo.html let me know how you go! Edit: FYI this is the RL course, I've got the trading envs coming soon!
Please can you show callback function as well and also saving the model for future stock analysis
Stay tuned @Satish, definitely coming!
love your content
I would remove the rows with missing mean values with *dropna* instead of setting them to 0. I assume the zero values could train the RL into wrong directions?
How do you make a custom environment to buy & exit, sell short & cover? Can it be trained select approptiate trade size and to scale in and out of trades?
ready to do it, lets get to it!!
Helll yeahhhhh!! What did you think?
You could use yfinance to get processed data. Btw Great Tutorials.
Nice! Will check it out!
It would be really cool to see how such a model can be connected to real-time data.
I don't really understand how i can use model.predict() without creating an env, to predict() only on given data...
maybe you have some tutorial?
Thanks for reinforcement learning brother
Anytime, glad you're enjoying it @Ameer!
Thanks a lot for the Video Nicholas
I have two questions to ask you :
In "info" output there are three arguments : "total_reward" , "total_profit" and "position"
-- What's the difference between "total_reward" and "total_profit" ?
-- what's the meaning of "position : 1" (for me there are many positions...is it saying that the major position taken is LONG here ?
I appreciate that soo much 🙏 thanks man!!
@@NicholasRenotte would appreciate an answer to my questions if possible lol
@@NicholasRenotte Why did you delete you're answer ? (don't be shy if you don't know...but you can say "I think that it is" even if you think that we expect you to be EXPERT which is not TRUE but it's not a shame . We are learning all together . What we expect from you is to make videos with non basics errors that's it....:)
@@ihebbibani7122 oh no, I had a laugh at myself when you replied 🤣. I responded but I think I missed the reply, my bad. I actually started digging into the code for you, profit is meant to be dollar profit, reward is percentage return so .98 means a loss of 2%. It's not super well documented unfortunately so took a little while.
@@NicholasRenotte Thanks a lot for your effort . Appreciate it. I asked also for "position" do you have any idea ? Thx
Another question, how do you restrict model from trading with some signal such as time you want the model to trade within or minimum volume, etc.?
Kinda tricky with RL, I would probably be implementing this with trading rules instead.
So once you train this agent, how do you actually run the agent live? I've used the freq-trade bot to trade before, but I don't understand how to actually use the trained agent to do trading.
Will be doing a bigger end to end tutorial on how to implement Jacob!
This is so cool.. how about adding the orders data into this model?
amazing content, thanks man!
Thanks for checking it out :)!
Hey Nick , Have you uploaded next step of this running on real environment yet ?? Could not find
Not yet coming!
Why did you use 'Low' values as prices? The agent shall make a decision at the 'Open' price of the day. We never know about the 'Low' of the day before the day is closed
I think I was having an off day tbh @Matt, should've definitely been using Open or Close!
Renotte You we’re having a ‘Low’ day 😂
Hi Nick, Thanks for your video, Somehow, i cannot run the code in jupyter notebook, as the code uses tensorflow 1.x and googlecolab only support googlecolab tensorflow 2.x and which tensorflow 2.x only works with stablebaselines3, can you kindly how how can we upgrade the code to tensorflow 2.x with stablebaselines3 ?
Thanks
thanks... learned a lot..
Would be cool if you integrated it to a crypto exchange like Binance for live trading in this RL series. That would "Wrap it up'" nicely :)
That's the gameplan @B B!
@@NicholasRenotte Can you please do a short video on how to get trading signals dynamically based (instead of just backtesting). If you could integrate it to a live exchange like Binance, it would be awesome aswell! Thanks!
@@jarvadetoro7004 will go into this in depth once we get back to the finance series!
Thanks alot for creating this video. Can you please also guide us to add one more action space i.e "Hold" or do nothing. As currently the agent trades on every single day.
Thanks in advance
Building a custom environment for this atm! Stay tuned.
Great video, I love the way you teach. I've tried your code using different stocsk, but when using GME the learn stabilize(explained variance always above 0.95) after some minutes while when using other stocks it doesn't stabilize, da you have any idea why this happen?
Heya @Marco, I think it's always going to vary depending on what security you use. I think the GME model is solving as it's learning to trade the short squeeze, but really it's seeing it and learning to go Long. Have you watched the follow up video, I go through how to solve and train on more stable data: th-cam.com/video/q-Uw9gC3D4o/w-d-xo.html
@@NicholasRenotte Yes, I've used some signal indicators as you suggest in video, but the training is unstable when I don't use GME stocks
@@Marco-hr2is I've got more videos coming on this topic, producing a stable algo will be one of them!
Is there a reason why you use the *Low* instead of *Close* price?
Same question here
Hi Nicholas! How would you go about training RL model on multiple ticker names?
Few different ways but could have a stacked array with each representing a separate ticker or use a dictionary per ticker.
I would love to know how i can give the bot multiple ticker information at once and get seperate but/sell outputs for every symbol. I think there is some interaction between different stocks/cryptos, so I feel like it would be beneficial to give the bot more information
Heya @Marcel, so multiple asset classes at once?
Great video, but I really want to know how to save and read training parameters, thank you!
Heya @leader c, done, I show how to do it in this video: th-cam.com/video/nRHjymV2PX8/w-d-xo.html and in this code base: github.com/nicknochnack/StableBaselinesRL/blob/main/Stable%20Baselines%20Tutorial.ipynb
Thank you so much!
Why does everyone think generic off the shelf indicators stuffed into a Python environment is going to change anything or improve trading performance?
Thanks for this video…
Great, info, it would be useful if you can amend program train target to average profit per trade
I am surprised that there is no third action for "Hold" in this model, because I don't think it makes sense to either "buy" or "sell" in *each* step.
Curious, is it possible to integrate this into mql4 and use neutral for automation trading?
Brilllant. is there a way to see transactions history ?
Yep, I'm going to be doing a deep dive into building something like this soon.
@@NicholasRenotte hey man have u made the video about this yet ? i would highly appreciate it, would also pay for a quick 1 on 1 call !!
I keep getting the message ValueError: Error: The environment must be vectorized when using recurrent policies. When I try to run the model.predict(obs) line. Any ideas what I can do to fix this. I used the custom environment like in the video to use custom indicators.
light weight babyyyy
But how you are going to apply this in real world, how the trained but is going to place real trade?
I tried it with 8 years of daily Facebook data. That's a total of about 2100 trading days. I used 60% for training and the rest for testing. 1m learning steps. 0.5 profit. Did I use enough steps? The explained variance doesn't stabilise.
5m time steps resulted in a total profit of 0.94. 98% buy signals.
That would mean it's at a 0.5 loss right? And you used the stock model @Vincent?
@@NicholasRenotte yes that's correct
@@vincentroye yeah that's not exactly the best result! Got some additional work coming for this model though! Stay tuned.
One thing is bothering me. We got through whole process and ended up with agent with some experience. How it is possible to save this data and use in real trading bot?
Heya @Dawid, you can save the model using the save function:
E.g. model.save('mysavedmodel')
Then reload it when you need it alongside the environment:
A2C.load('mysavedmodel', env=env)
@@NicholasRenotte thank you. I should have looked at this topic in documentation :)
@@dawidjurys4498 no stress, always happy to help! How's the performance of your model looking?
Please make a video on cryptocurrency trading bot using Open AI Gym..
Is there any quick way to put an early stop in?
Not quick but it is possible! I actually found it easier to just save the model every x thousand steps and grab the best one. Got a vid planned for this!
I dont understand why only Buy and Sell states and no hold? can hold be included so the model knows when to close the position?
Me to! Id like to understand better how algo buy buy buy and then sell sell.
is there a way to alter the rewards gained at each action?
May I ask why we use “Low” instead of “close”,any one can help ?
Why are you using the low column? I would use the close column in real life we don't know the low until the end of the period but it's too late to buy at that price
by the way, I love what you"re doing! really interesting stuff going on here :)
🤟
Ayyye, glad you liked it!
Does this model save ? Can it learn and remember it?
Sure can, you can use model.save to save the model down!
@@NicholasRenotte Hi Sir; I am working on this project of yours and wish to enhance the model by adding a few more indicators and connecting it to live account. Indicators part is done. I don't understand where should I place the buy and sell functions of the exchange and I previously made an a2c model wherein I have the actor, critic and shared model saved on disk, Mind guiding how to load and use it in your environment, if that's a good way to run from bugs of that environment.
Kind regards,
Tasneem
should_follow = true if video.content == "dope" and thumbnill == "Perfect"
i think upper line compile to true
Hahahah, thanks Ali!
It's not a profit: it sells on rising trend.