Q-Learning: Model Free Reinforcement Learning and Temporal Difference Learning
ฝัง
- เผยแพร่เมื่อ 30 พ.ค. 2024
- Here we describe Q-learning, which is one of the most popular methods in reinforcement learning. Q-learning is a type of temporal difference learning. We discuss other TD algorithms, such as SARSA, and connections to biological learning through dopamine. Q-learning is also one of the most common frameworks for deep reinforcement learning.
Citable link for this video: doi.org/10.52843/cassyni.ss11hp
This is a lecture in a series on reinforcement learning, following the new Chapter 11 from the 2nd edition of our book "Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control" by Brunton and Kutz
Book Website: databookuw.com
Book PDF: databookuw.com/databook.pdf
Amazon: www.amazon.com/Data-Driven-Sc...
Brunton Website: eigensteve.com
This video was produced at the University of Washington - วิทยาศาสตร์และเทคโนโลยี
I personally love the big picture perspective that Prof. Brunton always shows. Please, continue to make these high quality videos!
👍👍👍
More casual example for TD-learning:
Imagine a curious robot exploring a maze, searching for a hidden treasure. Unlike other methods that wait until it finds the treasure to learn, TD learning is all about learning on the fly. It uses what it already knows (like the estimated value of different paths) and immediate feedback (rewards) to improve its predictions about future moves.
- The robot keeps track of a Q-value (Q(s_t, a_t)) for each path, which tells it how good it thinks that path is based on its past experiences.
- When it takes a path and gets a Q-value (or reward) (like finding a clue), it compares that reward to what it expected (based on the Q-value - r_t + \gamma Q(s_{t + 1}, a_{t + 1})). This difference is called the prediction error.
- If the reward is better than expected (positive error, or r_t + \gamma V^{old}(s_{t + 1}) - V^{old}(s_t) > 0), the robot increases the Q-value for that path, making it seem more attractive next time.
- If the reward is worse than expected (negative error, or r_t + \gamma V^{old}(s_{t + 1}) - V^{old}(s_t) < 0), the robot decreases the Q-value, steering it away from less promising paths.
CS PhD student here. This video provides such amazing content. Highly recommended.
Thank you for the outstanding production quality and content of these lectures! I especially enjoy the structure diagram organizing the different RL methods.
Professor I must sincerely thank you for the astonishing quality of this video. You were able to clearly explain an advanced concept without simplifying, going into the details and providing brilliant insights. Also I sincerely thank you for saving my GPA from my R.L. exam 😆
Great videos. Definitely keep your face in the thumbnails. Even if the channel name doesn't ring a bell, sometimes I find your videos by just the thumbnails and always enjoy them
Prof. Burton, you are amazing. I never expected someone to take so much of time to explain a concept about TD. I'm one of the few people who hate reading text books to understand concepts. I rather see a video or learn about it from class.
Thanks a lot
Thank you dear Prof Brunton for this outstanding lecture. The detailed explanations and focus on subtleties are so important , Looking forward to your next videos.
I was hoping that your next video would have been about Q-learning, and here it comes!
Thank you so much for using very relevant analogies and very clear explanations. I think I have a much better grasp of the concepts behind Temporal Difference learning now.
I enjoy your talks. They are very clear and well structured and have the right level of detail. Thank you,
Thank you! It's a great video. My understanding in TD learning was deepened a lot.
this was the best explanation ever!
thank you so much, professor!
The videos are wonderful! Thank you, professor.
Excellent class! Extremely easy to understand!
You gave the best explanations I've ever seen!
Hi Prof. Brunton. Great vídeo as always! Please keep producing quality ML content
For MC you only get the reward at the end and then divide it up among all the states. But for TD, if you are only taking one step forward, where does the reward come from? A little confused here.
I do like the description of Q Learning. I had come up with another analogy for why it makes sense. If you took the action of going out to a party, and then happened to make some mistakes while there, we wouldn't want to say "you should never go out again." We'd want to reinforce the action of going out based on the best* possible outcome of that night, not the suboptimal action that was taken once there.
These are fantastic lectures, I use these as an alternative explaination to David Silvers DeepmindxUCL 2015 lectures on the same topic, the different perspective really suits how my brain understands RL. Thank you!!
This is the best RL tutorial on the internet.
Thank you for the clear picture. It was really well explained and others already mentioned, now I can say that I understand these techniques quite fairly well. 🙏
The video quality is incredible lol and all the concept is discussed extremely clear OMG!!
Brilliant masterpiece bro KEEP GOING !!
Thanks a bundle Steve, this was really well explained!
Great lecture! Concerning the difference between SARSA and Q-Learning, I didn't get the emphasis on Q-Learning being better for exploration. In principle, one can choose a epsilon-greedy for both methods. As a matter of fact, the SARSA method is defined in Sutton's book with an epsilon-greedy policy. I get the point that the TD target of Q-Learning does not depend on the policy itself and, therefore, is called an off-policy method. However, if one can choose a exploratory policy (e.g., epsilon-greedy) for both methods, why would SARSA be safer or less exploratory?
I got the impression he was asserting that the updates to the quality function can/will/often become an undesireable feedback loop when non-optimized states are used, and I would infer that means the training steps done on those states would have an undesireably high probability of entering such states.
What you said does seem to be convincing evidence otherwise though.
That is a correct comment. The difference between the two lies in the Q-function updates. The way you choose your action is orthogonal (and can be more/less exploratory in either method). Also, from the video, Q-learning comes off as "better" method than SARSA, at least in problems where you don't need safe exploration, which is not accurate. It's more like a trade-off, where no method is clearly better.
I love your videos in general. I think though, that this particular one needs a bit of a revision. Hope you don't see this as a critique, but rather as constructive feedback.
I very like your descriptions about deep learning
Somehow I find that the explanations given by Prof. Brunton are easier to understand than those provided by video lectures from Stanford (which are also available on TH-cam).
Steve: thank you again. I appreciate your work. Trying to help, let me say that I believe there is a small typo, at minute 5:29 you wrote π(s,a) = argmax_a Q(s,a), should it be written π(s) = argmax_a Q(s,a)? Also, at time 22:35, the first equation has a sum over k, should it be over n? Anyway, this is a very good video.
Thanks a lot. Not just math but also the intuition that i was looking for
Read the chapter, and I've been waiting for this video for a while. Happy to know I'm the first to comment :) Thanks, Steve. Can you explain more the bias (and especially the variance) in RL in a later video ?
Great explanation..very clear!
Thank you so much for these lectures sir!
Thanksssss steveeee. I couldnt understand nothing before this video.Thanks again
This whole series was good, but this one pushed me past my confusion. My neural network finally learned to play tic-tac-toe!
Hi Steve, Thanks for Amazing! lecture. I think the mail challenge in RL is designing our own custom environment (Multiple states and actions). It will be a great help if you can upload some lecture, suggest some link to do this job. Other comments are also welcome. Currently, I am doing some experiment on Retail pricing optimization using offline data. Looking forward.
Thanks a lot, Prof. Brunton!
In the Monte Carlo method, you have the reward discounted by gamma. Why do you discount a reward function for the entire episode?
Furthermore, 1/n(R) would not the average reward for any step k.
Thanks for the fantastic explanation!
Such a great video, thank you!
I would like to know what is being used to project the equations. It is apparently not video editing because he points directly to them.
One thing that seems to be either an error or just inconsistent notation is the use of TD(N) to mean an N-step TD. It seems like the value in the parentheses is supposed the value of lambda, not the number of steps of TD. TD(0) apparently should be read as, "N-step TD when lambda = 0, " while TD(1) means, "N-step TD when lambda = 1." I'm basing this off of the book "Reinforcement Learning: An Introduction - Second edition" by Sutton and Barto.
Loud and clear.
Great lecture, but I guess your definition of on/off policy is different from the definition of Sutton/Barto. On policy doesn't necessarily mean you always take the optimal action. "[on-policy] learns action values not for the optimal policy, but for a near-optimal policy that still explores" [excerpt from a different chapter but also valid for TD].
SARSA usually still follows a epsilon-greedy strategy.
Hello :) thank you for the video. I have a small question. I don't fully understand the difference between TD(0) and SARSA. Indeed, if SARSA uses the optimal action 'a' at each time step 'k', doesn't the Q-function in SARSA equal the Value function in TD(0) ? Or was there an error in my understanding ? Can you please help me see more clearly ? :)
For TD0, Why not say we do alpha x the new value (s_k+1) + (1 - alpha) x the old value (s_k). It's a very basic update method...
Can you explain how off policy q learning can take a suboptimal route sometimes if you are always taking the max of the actions presented?
brilliant explanation
being a phd student of you must be a gift :D
Very well explained.
Great lecture!
Excellent Explaination
woww thank you , so well explained with lot of patience .. god bless
One question, in terms of updating the Q function using the observed(real) reward at state k + 1, how do we know the observed(real) reward at state k + 1 since it is one timestamp in future?
22:35 As someone else mentioned, the first equation has a sum over k, shouldn't it be over n?
there is a question that pops up in my head; if SARSA is an on-policy method, then is it OK to use e-greedy algorithm in SARSA? as you mentioned it always take into account taking the safe and on-policy action rather than random and off-policy actions?
Nice talk al always. A question: at minute 5:00, does pi depend on a? It shoud not, right? The same holds for the previous video.
Pi of s,a should only be a function of s since the RHS calculates a. What am I missing?
what is the reward for r_k if we don't receive a reward after every action? Is it just assumed zero and the value is based only off the quality function?
I cannot access the new chapter in the 2nd edition. Has anybody accessed the link?
Brilliant! Thanks 🙏
In value able Content.. Cant thank enough.
Great video! Would be nice a simple example
Could you please explain it with some examples, that will be really helpful to understand these formulas, thanks!
Great stuff
ty 4 great explain greeze from switzerland :)
This is amazing content.
I just have a question (still struggle with the concept of on and off policy)... at 30:18, the max Q (in Q-learning) .. which Q is it old or new ?
old, but at next state $s_{k+1}$
Help a lot with my AI course final!!!!
top prof!
Thank youuu !
You are the best:))))))
very useful
wow, nice
👍
A+
I don't think the comparison of q learning and sarsa is accurate.
isnt q-learning, learning to play master yi
too much talk too few examples
Instead of using so many words to explain, why couldn't you just use a couple of examples to explain the relationship between Q(s, a) new vs Q(s', a')? It would be so easy to understand through examples.
It’s a theoretical formula.
Instead of using so many words to complain, why couldn’t you just make a video illustrating these yourself?
@@stevewu1920worst take ever