Reinforcement Learning for Trading Tutorial | $GME RL Python Trading
ฝัง
- เผยแพร่เมื่อ 30 ก.ค. 2024
- Heard about RL?
What about $GME?
Well, they’re both in the news a helluva lot right now.
So why not bring them together.
In this video you’ll learn how to build the beginnings of a Python Trading Bot using Reinforcement Learning. And better yet you’ll be able to do it by bringing in your own securities, in this case you’ll be working with Gamestop stock prices from MarketWatch.
The video doesn’t go through advanced strategies but gives you an idea of what’s involved to begin to leverage RL in a Finance/Trading environment!
In this video, you'll learn :
1. Working with the OpenAI Gym gym-Anytrading for Reinforcement Learning Trading
2. Training a Trading Bot with Python Reinforcement Learning using Stable Baselines
3. Loading in GME Trading data for training a custom RL Trading Bot
Get the code: github.com/nicknochnack/Reinf...
Chapters:
0:00 - Start
3:12 - Installing Gym-Anytrading and Dependencies
5:06 - Importing Dependencies
9:09 - Loading Gamestop Marketwatch data using Pandas
13:51 - Pushing Custom Data into the Gym-Anytrading Environment
18:11 - Testing the Trading Environment
24:18 - Training the Reinforcement Learning Agent
31:55 - Evaluating Model Performance
Oh, and don't forget to connect with me!
LinkedIn: / nicholasrenotte
Facebook: / nickrenotte
GitHub: github.com/nicknochnack
Patreon: / nicholasrenotte
Join the Discussion on Discord: / discord
Happy coding!
Nick
P.s. Let me know how you go and drop a comment if you need a hand! - วิทยาศาสตร์และเทคโนโลยี
![The moment we stopped understanding AI [AlexNet]](http://i.ytimg.com/vi/UZDiGooFs54/mqdefault.jpg)
Please keep making videos! You're a real treasure explaining RL so well!
I'm just learning it at school and you really just helped me understand a lot of it. Thank you!
Thanks so much @Pedro!
@@NicholasRenotte u77⁷⁷7⁷the
Am trying to make a project like this one and have almost no experience in it, but the learning curve just got easier thanks to you
Can't express how much I enjoy your videos. Amazing job making the projects as practical as possible and looking forward to more ML videos!
Extremely interesting and easy to understand! I'd like to learn more about the pros and cons of other gym trading environments. Thanks for all the time you spend producing these tutorials. You're helping a lot of people like me.
Definitely @Vincent. Glad you enjoyed the video. Will find some other trading envs that we can take a look at, I had another one on my list already that I've tested.
Fantastic tutorial! Some of the libs are a bit old now. I got it working on lambda stack with the following changes:
1. Use latest tensorflow-gpu and tensorflow
2. Change "stable-baselines" to "stable-baselines3"
3. Change "MlpLstmPolicy" to "MlpPolicy'"
Cheers
thnx!
Thanks!!!!!
Thanks !!!
what about stocks-v0, i cant get it to work. it says gym doesnt know it
@@user-pt1dd1wr4t You can pip install the gymnasium instead of gym and import it like "import gymnasium as gym".
Super helpful! Thank you for explaining everything in easy to understand terms
Would love to see more videos like these in depth. And a video on stock price predictions using neural networks
Just so everyone knows, you need to add df.sort_index() so the data isn't reversed. The model is training and predicting on reversed data. Gym-anytrading does not automatically sort by the date index.
Yah! Good pick up, fixed it in the second part to this tutorial!
Battled with this for like two days until I noticed, came to write this same comment here!
Hey man your videos are great. Please continue with reinforcement learning so that we can learn to apply RL algorithms. I have been searching for Application of RL algorithms but it is hard to come by and finally got your videos. Really great videos you make. Keep going please.
Thanks @shouvik, got plenty coming. Not stopping anytime soon!!
would love to see more in-depth video, specially about custom signal features
Done, will add it to the list. I enjoyed getting back into the finance side of things. What do you think about extending out the trading RL series a little more?
Check it out, custom indicators included: th-cam.com/video/q-Uw9gC3D4o/w-d-xo.html
@@NicholasRenotte that would be great!
@@vincentroye the extended series? or the indicators vid?
@@NicholasRenotte the extended series :)
Nicholas your work is impressive and community is growing, Perfect. The community always can expect useful set of instructions/information's about AI and as now how to model/teach RL agents. The RL is really awesome but rather very abstract. so it requires lot of studying. Your effort in promoting this branch of AI noticeable. Thanks also for the stable-baselines tips. Have a nice day.
@@blockchainbacker.4740 Hello Nicholas, I do not have any problem with investments since I invest only in education. You provide one of the best YT content. Thanks
Heya @Markus, that was someone impersonating me. They were blocked from the channel, click the user name in case it looks weird next time! But as always, thank you soooo much for checking out the videos. The new RL trading one is out as well: th-cam.com/video/q-Uw9gC3D4o/w-d-xo.html
@@NicholasRenotte I can't really understand how someone can be "impolite" generally. Your YT channel is awesome and following mini RL series very impressive. You display somehow very innovative way where RL can be deployed. Good luck with you innovative way of mind.
Thanks sooo much@@markusbuchholz3518 means a lot to me!!
would really love to see a more in depth video, heck would love to see more video from you. I am learning a lot so thank you! new subscribers and been binge watching your stuff. good works
Thanks soo much @izzrose1997, anything you'd like to see?
Hi Nicholas. Love your videos. I probably learn more watching your youtube videos than my data science classes back in college. I just wanted to know if we can add plain vanilla indicators such as 200 EMA, MACD, RSI as input vectors to train our reinforcement learning algorithm. These indicators can improve the performance of our algorithm.
Could you please explain in depth the best way to find the explained_variance near 1? And also maybe talk about the overfitting of the model and how it perfoms using cv time split. Thank you Nicholas, really good content!
would be cool if you could throw some basic technical indicators like RSI and OBV into the df to see if it helps the agent. Also, would appreciate it if you could add callbacks to stop training at the optimum level.
Keep up the good work and looking forward to the next vid of this series :)
Thanks so much @BB, let me know what you think, have RSI, OBV and SMA covered: th-cam.com/video/q-Uw9gC3D4o/w-d-xo.html
Awesome work, this would be a good series for you build out with other features and all the things you're more than happy to expand on :) . Also looked at other videos you have made, very clear and great videos!
Thanks so much @stockinvestor1, I'm definitely going to do more in this space. So far three more tutorials planned for RL trading but open to any additional ideas!
Thanks for the video, nicely explained and very interesting! Do you have any video or information about adding the sentiment to this reinforcement learning model?
Great lecture , really helped me a lot . Can you please explain what are the different states in the environment and what is the Y axis in the final performance plot.
Thank you so much again for the wonderful tutorial.
Excellent video! I'm wondering if you could add video on portfolio optimization using RL in the future? This would be more related to real-world trading environment where one needs to determine ratio allocation among multiple assets...
Definitely, I've thought about doing this with discrete optimization. Possibly looking at something like leveraging the efficient frontier.
I trimmed the data down to just the closing prices, and the algorithm is training a lot better. I highly recommend others do that as well.
Ha, awesome. How much better are we talking here?
Love the way you explain! it would be great if you can add a few technical indicators and market sentiment as features.
Thank you for such clear, hands-on tutorials on reinforcement learning. I have a couple of questions, though.
I've learned elsewhere that an RL agent requires a trade-off between exploration and exploitation. I didn't see this specifically mentioned in this video. Is there a reason for that? Perhaps it's not advisable to use any exploration/exploitation trade-offs in trading algorithms, or maybe this specific RL model doesn't support it. I would appreciate it if you could help me understand these considerations.
Additionally, I would love to see an example of an RL agent being trained with new data while in operation. I believe the official terminology is "online learning" or "continual learning." Please consider making a video that covers that topic as well.
Hi Nicholas, Excellent explanations, really commendable,
can you please have one video with using callback, looking forward for some more knowledge sessions from you!
Yup, got it planned and coming @Vivek!
excellent tutorial and easy to understand. thx! would like to see video on using the callback for model.learn
You got it @Wong Johnny, got a bunch of stuff planned for our ML trader series!
Hi Nicholas, should we perform a train test split on the df and, if so, do stationary variables and feature standardization of the variables help with improving the model?
Great, Thank alot for all the work you put in into this tutorial!
Thanks Nicholas! This is amazing. Is it possible or reasonal to train bot using all stocks data rather than GME price data itself?
your RL stuff is fantastic thanks for doing it!! 💎 🙌 bro.
Also yes please do more RL trading stuff! Different action spaces & custom environments!
You got it, I think I'm going to bump it up on the list as well @Ed!
Hii! New Fan here :) Would love to see that, too! Maybe with a hold position... or trading fees. Just as an idea! Keep up the great work man
@@weigthcut you got it! Will likely be doing a mega series on ML for finance/trading!
Is there any method by which we can plot the rewards vs the episodes graph?
Your videos are really helpfull in learning Reinforcement learning. Thank you
you can add callback to the learn method, where you can specify any custom item you want to plot
@Nicholas Renotte thank you the video is very good, but may I ask a question I'm still confused in part 1 determining the window_size 5 and the frame_ bound (10,100), will the frame bound display the best 90 days of data or what? thanks for the explanation later
Thanks Nicholas, great video as usual
Do you plan to make a tutorial for multi-agent RL and multi-agent environments?
It's on the list, a little far out, but it's definitely planned @Omar.
excellent tutorial! keep up your good work! i read below that you are planning to add the callback to the code! looking heavily forward to that!
Thanks so much @Peter, definitely I've got it planned in the next couple of weeks! Code is done, just need to record!
@@NicholasRenotte Love to hear that! BTW just saw I was at the same company as you ;). Pinged you at some social media, if you are interested in connecting.
@@TheJourneyToMyFreedom definitely, always intested in connecting!
@@NicholasRenotte You already accepted ;)
@@TheJourneyToMyFreedom perfect!! Will jump onto LinkedIn!
How do I force it though to not make short trades? I only want to train it to make long trades. Also, I would want it to sell before it tries to buy again (Buy then Sell). Is this stuff configurable with Gym-Anytrading?
Can you do an update on this ? Since tensorflow 1.15.0 is not available anymore and seems they changed so much that i just cannot get this to work with tensorflow 2
Awesome! You explain very well!! Do you have a tutorial where you adopt TA(RSI, MACD, BB etc..) to the chart to see if the agent perform better or if was traded only on TA.
Sure do, check this out: th-cam.com/video/q-Uw9gC3D4o/w-d-xo.html
Hello, I am trying to train an agent on a custom environment that I made and I was wondering if there is any way to increase how often the performance metrics like explained_variance and value_loss pop up during the training process.
That's great! Please do more of these
You got it, got a bigger series starting up! Saving them all here: th-cam.com/play/PLgNJO2hghbmjMiRgCrpTbZzq0o_hINadE.html
Umfortunately I get by the end of part three the following errot: attribute error: 'gym_logger' has no attribute 'MIN_LEVEL'. I tried the solutions on stackoverflow without success. Do you have any idea?
Great video, highly educational.
My guess is that the granularity was not sufficient as things can change wildly during the day and between market open. Minute granularity in 5 minute trading blocks may have worked better
Solid suggestion!
You make understanding so simple . Love your work thank you for making such videos .
So glad you enjoyed it @Krishna!
I was wondering why aren't we split testing the data into train and test prior to training the model in reinforcement learning?
Great content Nicholas! I have a small doubt. While the dataframe has different prices (Open, High, Low and Close), which price does the model use? How does it decide to use which one?
I think I used open, I should've used close 😅 you can change which you pass though.
You are an angel. Thanks a million for all you are doing by making knowledge free and easy to use.
How would I go about saving/loading models? Let's say I want to implement a callback that saves the model with the highest explained variance and then load it and serve it new data as it comes in through the terminal?
Is there a way to do this without using stable-baselines, and use keras instead?
from stable_baselines import A2C is not working.
It gives a DLL error. Can someone help in sorting that?
This is a great video. Good work @Nicholas! Is it possible to save the developed model as a pickle file? Eg. if we need to deploy the model in a production.
Normally you just save the trained model weights as a h5 and reload into keras! I show how to do it with stable baselines in the RL Course ont he channel.
Man your videos are awesome. We need more about adding other features (or creating them).. Thank you
Help me, I want to know your Python and Anaconda versions. I have run but it gives a lot of errors in gym and tensorflow
Hey Nicholas, it was an interesting video! Do you have any video on creating our own environment or something to suggest?
with stable baselines 3 they removed mlplstm policy, are there others that can be used in its place?
i just wonder if the "Estimation Part" is correct ??
you use
```
env = gym.make(...)
obs = env.reset()
```
but the problem is that did you load the previously trained model ?? when using the code, `env` will be replaced with a new one, is that right ??
more, you 'KeyInterrupted' the training part, will the model save the process where it had been ??
Can I use this on a mac? It says GPU tensor flow isn't mac supported. thank you.
I had try PPO (action masking) to learn based on OHLCV data, but the model can not to converge.
while importing A2C i got this error "ModuleNotFoundError: No module named 'tensorflow.contrib' " how to resolve
any news?
How can I add a stop loss action when the profit drops to certain value?
Very nice video! I am interested in the callback functionality to stop the model at a certain threshold of accuracy. Thanks!
as always your RL stuff are fascinating and easy to grasp!
Thanks so much @Bite sized!
I want to pass in a few more data to the model along with the price, such as the market cap, the change in market cap, the difference in 'high' price and 'low' price, maybe something more to experiment with.
How do I do this?
Thanks in advance
Greetings. Could you add or direct to a resource that shows how to do the final step, which is using the output for actual trading? Thanks
based on that you used mlplstm policy, is your method deep reinforcement learning?
Hello, what version of python did you use for this video?
Has anyone else noticed that when he runs evaluations, he gets a different total profit. See 34:53 and 35:55 when he reruns the evaluation. The frame_bounds are the same (90,110) for both evals. When evaluating the same trained model on the same range of data, shouldn't one expect the same total profit?
I haven't used an agent like this, but I wonder how you know if the model has overfit the data when you go forward? I also wonder how a bot like is is supposed to handle anomalous events.
I think it might have been tbh @Varuna, we're going to be handling it properly when we build up our bot for the ml trader series with better backtesting!
Hello Nicholas! First of all thank you for your videos they are really intersting! Can you explain a little bit more the meaning of the explained variance and the value_loss? It will help to understand why they need to be respectively closed to 1 and 0. Do you have also a automatical way to select the best moment to stop the fitting ? Thank you in advance
Check this out: th-cam.com/video/eBCU-tqLGfQ/w-d-xo.html
Will this work if I do not have a gpu in my system? Thank you
Have you made a profitable bot that just trades forever?
Great Job!! Love ur videos ! Keep the good work : D. I didnt now nothing about Python Code and RL and thanks to ur awsome videos im learning fastfast
Yesss! Awesome to hear João, what are you currently working on?
Im planin a deep RL to play sliter.io : ))
@@joaobentes8391 niceeee, I saw an example of that a couple of nights ago using Gym Universe. Sounds awesome!! Would love to see a snippet once you've got it trained.
great content and nice explaination. Love the vids
Thanks so much @Giacomo! Glad you're enjoying them, anything else you'd like to see?
@@NicholasRenotte it would be interesting to do a principal component analysis on a huge database of indicators to see which are less correlated and bring the most variance to the data. That way you know that you are using the right indicators (there are so many out there). Otherwise use forecasting models as indicators in which basing your trading strategy on. For example a logistic regression function saying will price go up or down ? An ARIMA checking the direction, if it forecasts that the price will go up or down. Once you have several confermation indicators, enter the trade and then add the proper risk management to the system. No idea, I am still trying to figure all of this out. Im coming from tradingviews pinescript, which is nowhere near what you can do in python. I do have a bit of knowledge of data science, but between studying it and actually applying it, there is lots of pracite to be done.
@@NicholasRenotte For sure though I think the whole crypto space is much more interesting as there is plenty of data to choose from. Especially from the blockchain it self not only necessarily price and volume data. Just some food for thought, maybe that gives you some ideas for next videos
How would i change the x-axis of the plot to show the dates?
Note: Stable baselines only work with Tensor Flow one. I'm going through tutorials on reinforcement learning and it's hard to find something relatively current and easy to follow.
Take a look at the latest Mario tutorial, that uses Stable baselines 3 @Michal!
Thanks for the great content!!!
Thanks so much @01bit!!
Thanks for such great content!
I am very much interested in RL and robotics and I have been working to build a robo dog which uses RL to learn to walk and can recognize faces and can understand gestures and voice commands, could you please do this kind of video? :)
I still don’t understand what the red dots and green dots represent. 'shorts' means sell??
Please provide detailed steps to setup this. Unable to run this as unable to install tensorflow 1.15
How is it possible that reward is positif and yet profit says that we are loosing money ?? Can you explain this ?
Outstanding content because you explain everything but do it quickly and clearly.
Thanks so much @Varuna, glad you liked it!
Thank you for creating good content. This is really helpful!
How do I add stopLoss Take profit they are very critical for a trades success. Even hedging a loss making trade and make it go in profit or use some custom logic i.e. 7 up and 1 down i.e. if a particular trade is loss making trade then put an opposite trade with higher lot and increase it if it is still loss making , strategy is called martingale. I mean how can I give an input to the model say if RSI is greater than 70 stop buying for example.
Hey TensorFlow-GPU has been discontinued, can anyone help me that wether this will run fine with CUDA or any other version of pytorch or not.
Very useful stuff. Thanks Nicholas.
Doing some RL trading now @Adam?!
@@NicholasRenotte I'm just learning. I was on an expensive algotrading course, but the guy didn't explain as well as you do. Since you're an IBM guy, I was wondering if you could give us a course on Qiskit and Quantum Machine Learning please. Thank you.
@@ButchCassidyAndSundanceKid oooooh now you're talking my language! Been keeping that one a secret but I've definitely got something coming in that space 😉
@@NicholasRenotte My understanding of Qiskit and Quantum Machine Learning is still very limited, all these Hadamard Gates, Shor's Algorithm concepts are very abstract and not easy to grapple with. And without a full understanding, one cannot proceed further to Quantum Machine Learning. I have just finished watching your Reinforcement Learning series, it taught me a different way of writing RL code (I used to use tf.agent which is very clunky and difficult to understand). Thank you Nicholas. Keep up with the momentum. Look forward to more of your tutorials.
Really great video!
I am trying to do RL with energy engineering, this tutorial is as close I have gotten because it uses recorded data. Any recommendations or examples of code that tackles optimal solutions for energy storage, micro grids, refridgeration, etc?
Ooooh, I'm actually looking at energy and resource opt at work rn. I don't have anything quite yet but this might help in terms of setting up the custom environment that will likely be needed: th-cam.com/video/bD6V3rcr_54/w-d-xo.html
Excuse me, After obtaining the model, how we can run the model in the realworl day by day? You have ti insert each day the data with Gym?
Nope, can bypass and pass through the state to the trained model!
Hi Nicholas, great video can you explain call back function where the model can stop when reach expected accuracy?
Yup, I show it in part 5: th-cam.com/video/Mut_u40Sqz4/w-d-xo.html
@@NicholasRenotte Thanks so much
thx very much. it's very useful for me
i am getting an error:
AttributeError: module 'stable_baselines.logger' has no attribute 'get_level'
The data is sorted by date descending. Neither in your code nor in gym_anytrading the data is sorted "properly". Do you do it on purpose?
Nope, that's a screw up. Sorted in the follow up tutorial: th-cam.com/video/q-Uw9gC3D4o/w-d-xo.html
Thanks for the video, i guess it would be great if you show how to create custom features with financial formulas
oh, you've already created such video, thanks:)
Hello , How to get the corresponding dates to the given signals ? Thanks in advance
Inside of the dataframe you can filter using df.loc['your date here'] once the date is set as the index!
Hi i am getting ValueError: too many values to unpack (expected 4) near n_state, reward, done, info = env.step(action) under Build environment. Can anyone help me with this
Hi Nicholas, I'd like to see a video on making a callback function if you have time.
Thanks very much for your work, this is very informative
Heya! Check out the reinforment learning course :) all is revealed!
@@NicholasRenotte Nice one. Thanks
As I am new to all this still trying to get my hands around Deep Learning I have been looking at maybe going with reinforcement learning. The only question I have is it seems that we only are using RL to train any given model and then use that in 'production'. What I'm looking for is something which will allow it to always be in learning mode which it then could adapt as the environment around it changes. For this is RL the best way to go?
Heya @Jeff, you're going down the right path. From a deployment standpoint I've only shown interation. In production like projects what you'll actually do is have continuous retraining and deployment. The process is the same however the cycle is changed. So for example what you would do is:
1. Train your model
2. Deploy it to production
3. Overnight or at a set schedule, bring in new data
4. Load your existing model and retrain it on the new data
5. Redeploy the model
@@NicholasRenotte What I want to try and do really is to give the robot reward areas and from there have it train itself on what to do next. This way the robot would evolve more or less. The only items I will need to look into is how to recharge it. But at the end give it a camera, 4 sensors (cliff sensors), a charger and a pickup device like the Vector Robot has. From there it learns to move and explore and what not all the while knowing only that it's doing the right thing when getting a reward. BUT yes, before I get to this level I need to first go back to your very first video and start at the very beginning. :) The only issue I am having now is I can't get my Jetson Nano (b01) Jetpack 4.5 fully installed and working. :(
+++ As always very informative and clearly explained. Will you a better result when you stop training at 0.95 explained variance?
Ideally yes @Henk, I think having an improved loss metric will help as well. Got a few more vids to do in the CV space then will head back to RL soon!
hie Nic How do you stop the algorithm from running all a hundred thousand iterations in training the model
You can stop the cell using Jupyter and it'll stop training :)
Very useful!
Haha would be amazed if it could predict a short squeeze 😋
Black swans will always be an issue for AI based trading, or human trading for that matter.
Ikr @FuZZbaLLbee, I think I’d just be creating RL agents 24/7 chasing alpha if there was the chance of predicting them!
Was genuinely curious how a quick RL agent was going to perform in this case. Definitely will do a little work on exploring integration with external data sources e.g. forecasting reddit sentiment and bringing it as a leading signal.
So if I wanted to predict future actions and profits, can I use "frame_bound=(len(df)-5,len(df)), window_size=5" for the next 5 days?
Would repeatedly call it using a sliding window, would need forecasted prices if you wanted future actions.
@@NicholasRenotte Thank you for the info!
please make this video step by step beginners can not understand what are you doing also i tried every thing and can not pass step 0 !!
Checked out the RL course?
Could you please redirect us to any video (or text) on how to add callbacks to automatically stop the learning when the explained variance achieves an arbitrary target (e.g greater than 0.94) . thanks
Great video mate, would love to see the in-depth video, but this does look like a method that is far too easy to overfit with. What do you think? Thanks again for the vid!
Yep, agreed! It probably didn't help that I picked a stock that had wild variance. Having early stopping will help, the model trained so fast it was hard to stop when it reached an appropriate level of EV. I'm going to pump out some more detailed videos on how to get something that is actually a little more stable working plus custom indicators, integration, etc. Thoughts?
@@NicholasRenotte That would be awesome! I agree with the custom indicators and integration :) those videos would be really helpful, some like a moving average would be a great intro I think, do you know if you can do multiasset trading using the RL environments that you're using?
@@btkb1427 got it, I'll add an SMA or something along those lines! I think you could have a multi-asset model but from my (very brief) research it sounds like individual asset models are more the norm. You would probably run individual asset models with an overarching funds management model is my best guess!