Great tutorial. Everything put together in the most basic form and it works very well. As a feedback I can say that inside .act() call and when calling forward of target_net, grad could be disabled with torch.no_grad context and it'd run much faster since computation graph won't be created.
That was pretty cool, and clear explanation and code. Many Thanks for this. Albeit I don't think I will ever quite understand Squeeze and UnSqueeze() removing and adding pytorch dimensions. I guess in practice I would add Save model (pytorch dict parameters) every so often, with early stopping etc, since running RL algorithms can be painfully slow experience in practice. Also where/when to copy ptorch tensors onto Cuda devices to speed up the algorithm on GPUs where available. But there are examples and code out there on how to do this, and can easily applied to your code.
Very nice description of DQN, do you have the code in a repository ?, I tried to write the code as you described in the video, but I am getting some errors.
Great Video, what you said at 30:00 really resonated with me. I have the same problem! my models learns fine, up to a point where it chooses to take a dive. The thing it that it does this about half the times. Sometimes it trains fine. Would you mind telling me what the bug was, because there is a high chance I have the same one! Cheers!
Maybe it's an easy task, but can you make a tutorial on how we can find the shortest path of Graph as we use the graph as an environment on Deep RL ? since all the tutorials on youtube are using the GYM library as an environment, changing the environment to the graph theory will be nice and you are easy to understand, and you have ways to exposes the idea easily
@@manelmimi8197 Although I haven't done research on this, many games can be modeled as MDPs and it shouldn't be so different from a typical gym environment. If the action space changes between nodes, you may run into trouble there due to the limitations of neural nets. But I feel confident that there is research on this topic, and probably examples on github.
Great video - I could only get it to run in Colab, not in VSCode or Pycharm. If I can just figure out how to render it in Colab... I will be viewing the next one too...
Thanks for this tutorial! One question is in the Nature paper they have one For-loop nested into another, whereas in your code you do not do this. Was there any particular reason why?
Very nice tutorial. It is good to see how you implemented this paper and making it as easy as you can. My question will be if you try to implement this on colab, there is an error. How do you suggest we go about implementing this for a smooth running. I a really new to this topic and I am trying to learn it as much as possible. Thank you.
12:00 Deep Q Learning can only be used with agents that have a discrete number of actions. What Reinforcement Learning algorithms use continuous actions?
Do you think this algorithm could be used to control a stepper motor? How many actions can this algorithm output? Because a stepper motor needs to be commanded to make a certain number of steps. Maybe between 0 and 100. I know that is a lot of actions to learn. Would you recommend PPO instead?
Assuming you are observing some environmental state between steps of the motor and that the motor at least probabilistically affects those observations, some form of reinforcement learning should be able to work. I recommend experimenting with different algorithms, you will probably want to use more sophisticated variants of Q-learning instead of this vanilla implementation.
@@brthor1117 Thank you for responding to all my comments. I will be modelling a robot with stepper motors. The motor will receive a command to move and be unresponsive until it stops. It will learn not to make bad moves. I think I'm going to implement a MOMPO algorithm next. I've seen video of PPO and MOMPO and the MOMPO looks much more smooth. I haven't found a PyTorch implementation of MOMPO, but a few of MPO. Can I ask, is the difference a penalty if the prediction goes outside -1 and 1?
@@Throwingness "is the difference a penalty if the prediction goes outside -1 and 1?" I'm not totally sure what you are asking there. If the rewards are scaled too high it can affect the scale of the advantages in algorithms like PPO and cause difficulty in training. Commonly some type of reward normalization is used to offset this. RL algorithms can require millions of observations to reach decent performance and that can be difficult to collect using a robot in real time. There has been success training RL algorithms on robotic simulations and then transferring that into the real world.
I have checked multiple times, but i can't seem to find any differences (except for the expecting 4, because of the new return parameter) obss = np.asarray([t[0] for t in transitions]) ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (32,) + inhomogeneous part. i can't for the life of me get why this happens
Hi I tried following your tutorial but i got the following error: ---> 17 new_obs, rew, done, _ = env.step(action) the error: too many values to unpack (expected 4) any idea on what the problem could be? thanks!
maybe a change in the api of the env.step() function from gym. You'll have to put it under a debugger or print the return value from that function and take a look at the tuple it returns.
Heyy can you do a video on finding the optimal set of input parameters, for the cases when we know the output? It's like a reverse engineering problem, Is it possible to use RL?
Some editors allow you to place multiple cursors and edit at each cursor simultaneously. In this case pycharm lets you alt-click to place another cursor. VS Code and Sublime Text are two others that let you do this.
The videos are organized as a successive set of tutorials on DQN variants that lead up to a Rainbow implementation. This video is part 1 and covers the vanilla DQN theory and implementation.
@@brthor1117 Well My average reward isn't getting above 80, even though I've used the same parameters as yous. How much time on average CartPole-v0 DQN takes for training?
@@AshishSingh-753 There are other frameworks & languages you can use as well. For example, caffe in c++ or torch in lua. Most practitioners use python though and a lot of the examples you will find will be in python.
Your channel is a gem! Thank you for making such quality content!
👌🏻
Great tutorial. Everything put together in the most basic form and it works very well. As a feedback I can say that inside .act() call and when calling forward of target_net, grad could be disabled with torch.no_grad context and it'd run much faster since computation graph won't be created.
That was pretty cool, and clear explanation and code. Many Thanks for this.
Albeit I don't think I will ever quite understand Squeeze and UnSqueeze() removing and adding pytorch dimensions.
I guess in practice I would add Save model (pytorch dict parameters) every so often, with early stopping etc, since running RL algorithms can be painfully slow experience in practice.
Also where/when to copy ptorch tensors onto Cuda devices to speed up the algorithm on GPUs where available. But there are examples and code out there on how to do this, and can easily applied to your code.
Check out the next video in the series where we add gpu support: th-cam.com/video/tsy1mgB7hB0/w-d-xo.html
Great video. This is the 4th I watched and only one that didn't have 300 plus lines of code in it. This is the hand up I needed.
😁glad it helped
Very nice description of DQN, do you have the code in a repository ?, I tried to write the code as you described in the video, but I am getting some errors.
Great Video, what you said at 30:00 really resonated with me. I have the same problem! my models learns fine, up to a point where it chooses to take a dive. The thing it that it does this about half the times. Sometimes it trains fine. Would you mind telling me what the bug was, because there is a high chance I have the same one!
Cheers!
Maybe it's an easy task, but can you make a tutorial on how we can find the shortest path of Graph as we use the graph as an environment on Deep RL ? since all the tutorials on youtube are using the GYM library as an environment, changing the environment to the graph theory will be nice and you are easy to understand, and you have ways to exposes the idea easily
I'm not sure what you're trying to accomplish here. An RL agent that learns djikstra's?
@@brthor1117 create an environnement from an MDP model and training the DQN agent to this environnement
@@manelmimi8197 Although I haven't done research on this, many games can be modeled as MDPs and it shouldn't be so different from a typical gym environment. If the action space changes between nodes, you may run into trouble there due to the limitations of neural nets. But I feel confident that there is research on this topic, and probably examples on github.
Good tutorial, though due to environment versions, it is required more additional steps.
Great video - I could only get it to run in Colab, not in VSCode or Pycharm. If I can just figure out how to render it in Colab...
I will be viewing the next one too...
My env.render() is not working. I tried with all the render modes also including rgb_array
Plz help
No idea, best to search for issues related to the gym package, as there is nothing non-standard related to its usage here.
use ensembles and add auxiliary tasks to each deep q network in ensemble for any game
Your website link is not working. Can you please provide the code as well?
Thanks for this tutorial! One question is in the Nature paper they have one For-loop nested into another, whereas in your code you do not do this. Was there any particular reason why?
I do not remember, perhaps it was a simpler way of writing the code.
Super helpful, GOAT
Why don't we keep track of episode reward values when we are initializing? I am asking as it is occurring 1000 times. Isn't this value important?
Very nice tutorial. It is good to see how you implemented this paper and making it as easy as you can. My question will be if you try to implement this on colab, there is an error. How do you suggest we go about implementing this for a smooth running. I a really new to this topic and I am trying to learn it as much as possible. Thank you.
This should work fine in colab, any error is most likely a bug in your program.
nice
keep the grind
one day u will suceed
🙌🙌🙌
Thank you for the awesome explanation, it was really helpful :)
😁🙌
12:00 Deep Q Learning can only be used with agents that have a discrete number of actions.
What Reinforcement Learning algorithms use continuous actions?
algorithms in the policy gradient space such as PPO, and others
Very good clear code :) Thanks!
hi. i did the code, but i don't understand why my output step is starting from 11,000 and not from 0 as you did?
Very nice video, thank you !!
Thanks for the comment!
Do you think this algorithm could be used to control a stepper motor? How many actions can this algorithm output? Because a stepper motor needs to be commanded to make a certain number of steps. Maybe between 0 and 100. I know that is a lot of actions to learn. Would you recommend PPO instead?
Assuming you are observing some environmental state between steps of the motor and that the motor at least probabilistically affects those observations, some form of reinforcement learning should be able to work. I recommend experimenting with different algorithms, you will probably want to use more sophisticated variants of Q-learning instead of this vanilla implementation.
@@brthor1117 Thank you for responding to all my comments. I will be modelling a robot with stepper motors. The motor will receive a command to move and be unresponsive until it stops. It will learn not to make bad moves.
I think I'm going to implement a MOMPO algorithm next. I've seen video of PPO and MOMPO and the MOMPO looks much more smooth.
I haven't found a PyTorch implementation of MOMPO, but a few of MPO. Can I ask, is the difference a penalty if the prediction goes outside -1 and 1?
@@Throwingness "is the difference a penalty if the prediction goes outside -1 and 1?" I'm not totally sure what you are asking there. If the rewards are scaled too high it can affect the scale of the advantages in algorithms like PPO and cause difficulty in training. Commonly some type of reward normalization is used to offset this. RL algorithms can require millions of observations to reach decent performance and that can be difficult to collect using a robot in real time. There has been success training RL algorithms on robotic simulations and then transferring that into the real world.
Thank you for the step by step explanation. Is it possible to do a tutorial of DQN on image classification, like MNIST? 😍
👉th-cam.com/video/td0TjL5tznc/w-d-xo.html
Great video.
I have checked multiple times, but i can't seem to find any differences (except for the expecting 4, because of the new return parameter)
obss = np.asarray([t[0] for t in transitions])
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (32,) + inhomogeneous part.
i can't for the life of me get why this happens
Same here...
Env.reset() returns a tuple
you have to make env.reset()[0] to get the observation
Idk why but my model is not learning
Shouldn't you detach the targets in the loss function?
Hi I tried following your tutorial but i got the following error: ---> 17 new_obs, rew, done, _ = env.step(action) the error: too many values to unpack (expected 4) any idea on what the problem could be? thanks!
maybe a change in the api of the env.step() function from gym. You'll have to put it under a debugger or print the return value from that function and take a look at the tuple it returns.
Im getting the error -> can't convert np.ndarray of type numpy.object_. torch as tensor !
I have never hit that error, maybe try torch.from_numpy pytorch.org/docs/stable/generated/torch.from_numpy.html
Thanks. Can i download the code from somewhere?
Heyy can you do a video on finding the optimal set of input parameters, for the cases when we know the output? It's like a reverse engineering problem, Is it possible to use RL?
Optimal set of input parameters? Like function optimization? You can probably go for SGD directly if it's a mathematical function.
how much memory on your laptop do you need for this not to have error of 'cuda out of memory'.
note that laptop has shared memory of the gpu and cpu.
I don't know about that kind of setup. IIRC this tutorial runs the algorithm on the cpu only.
not related to DQN how did you write the same line in 4 line at once at 22:04 ?
Some editors allow you to place multiple cursors and edit at each cursor simultaneously. In this case pycharm lets you alt-click to place another cursor. VS Code and Sublime Text are two others that let you do this.
@@brthor1117 thank you
29:30 wysi
Nice
Good paper!
The paper was pretty good once I finally understood it.
Hi i need help with my project can you help me?
no
This is not DQN in the Rainbow version.
The videos are organized as a successive set of tutorials on DQN variants that lead up to a Rainbow implementation. This video is part 1 and covers the vanilla DQN theory and implementation.
Would you mind providing the link to Github?
This is a follow along video and has no provided code. There are many sample DQN implementations available if all you need is some code.
@@brthor1117 Well My average reward isn't getting above 80, even though I've used the same parameters as yous. How much time on average CartPole-v0 DQN takes for training?
@@uonliaquat7957 IIRC about 2 minutes. Likely you have a bug.
Jackson Scott Taylor Ruth Hernandez Michael
Cool
👍
How much python is needed for reinforcement learning
@@AshishSingh-753 There are other frameworks & languages you can use as well. For example, caffe in c++ or torch in lua. Most practitioners use python though and a lot of the examples you will find will be in python.
anyone who coded this and found it working, could you please share the code. I coded it and got error. @brthor could you help with my error?
i have a problem, how to fix this?
target_net.load_state_dict(online_net.state.dict())
'Network' object has no attribute 'state'
.state.dict() -> .state_dict()