Reinforcement Learning, by the Book
ฝัง
- เผยแพร่เมื่อ 4 มิ.ย. 2024
- The machine learning consultancy: truetheta.io
Want to work together? See here: truetheta.io/about/#want-to-w...
Part one of a six part series on Reinforcement Learning. If you want to understand the fundamentals in a short amount of time, you're in the right place.
SOCIAL MEDIA
LinkedIn : / dj-rich-90b91753
Twitter : / duanejrich
Github: github.com/Duane321
Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: / mutualinformation
SOURCES
[1] R. Sutton and A. Barto. Reinforcement learning: An Introduction (2nd Ed). MIT Press, 2018.
[2] H. Hasselt, et al. RL Lecture Series, Deepmind and UCL, 2021, • DeepMind x UCL | Deep ...
[3] D. Silver, Lecture 1: Introduction to Reinforcement Learning, Deepmind, 2015, • RL Course by David Sil...
[4] Y. Wang, Pricing at Lyft, Lyft, 2022, eng.lyft.com/pricing-at-lyft-...
[5] A. Irpan, Deep Reinforcement Learning Doesn't Work Yet, 2018, www.alexirpan.com/2018/02/14/...
[6] D. Silver, et al., Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, Deepmind, 2017.
[7] J. Schrittwieser, et al. Mastering Atari, Go, Chess and Shogi by Planning with a
Learned Model, Deepmind, 2020.
[8] R. Roy, et al. PrefixRL: Optimization of Parallel Prefix Circuits using Deep
Reinforcement Learning, Nvidia, 2022
[9] J. Degrave, et al. Magnetic control of tokamak plasmas through deep reinforcement learning, Nature, 2021.
SOURCE NOTES
[1] is my primary source for this series. It largely determined the notation and the set of topics and motivated much of the commentary. This series could not be what it is without the comprehensive and consistent presentation of this vast subject provided by this text. If you're interested in learning more, I highly recommend this text.
In preparation, I also took Deepmind's course [2], with the intend to understand a different perspective. The material is similar, though not entirely. Notably, Deepmind's problem statement involves agents receiving observations, which are summarized into an agent-specific state. This is a more application-ready problem statement. Early drafts of the series included this version of the problem, but it was cut, since it complicated following topics. Overall, I learned about RL substantially from this course.
The Maze demonstration of the value function comes from David Silver's 2015 lecture [3].
[4] - [9] are sources for all papers and blogs referenced at the start of the video. It includes a blog [4] where Lyft describes how RL is used in their pricing algorithms.
NOTES
With regards to the statement "RL won't be the same revolution that Neural Networks were. That's OK - NNs are quite a high bar". I should elaborate. I anticipate a response of, "You can't separate RL and NN, since much of the most impactful applications of RL involve NNS". Yes, comparing these technologies is fraught. In effect, I'm presuming that if a widespread subscription to RL's problem statement is enabled by more performant NNs, that is part of the RL migration. That's not fair if we're to attribute successes to either RL or NN. In my view, this attribution isn't important. I'm merely making the claim that RL will become a primary component in many production systems. The comparison of RL vs NNs is an accidental symptom of how I pitched the RL trend. In retrospect, I would have phrased this differently.
TIMESTAMP
0:00 The Trend of Reinforcement Learning
2:46 A Six Part Series
3:24 A Finite Markov Decision Process and Our Goal
9:02 An Example MDP
12:49 State and Action Value Functions
15:00 An Example of a State Value Function
16:28 The Assumptions
17:58 Watch the Next Video!
CORRECTIONS
1) In the MASE, I have a -16 where there should be a -14 (Thank you rogiervdw).
Let's take a quick break -- immediate cut and continues to next section. Got me good :)
Mmmmmm coding
how are you so clear with your word choice, expressions and tone? it's like you're uploading into my brain directly. I'm spreading the word, you deserve to be an educational youtube superstar.
You deserve a million followers
Agreed. Help spread the word.
the content is too specialized, this is like grad school level cs/stats, i doubt there are like a million of those
@@akshaygulabrao4423, Have you been to Google around lunchtime? I believe there might be a million of them right there.
He has some great intuitions. I am lucky to found him..
I found your plain English explanations of the equations particularly helpful. Thank you!
After the release of the AlphaTensor paper, this is a pretty timely video, and I wholeheartedly agree with your statement about RL becoming much more relevant outside the realms of simply solving games. Great start to the series, and I look forward to the rest.
Reinforcement learning is so fascinating, I'm so looking forward to the next videos!
Reinforcement learning is so fascinating, I'm so looking forward to the next videos, as well!
Absolutely phenomenal clarity in your explanation. A million thanks for producing this. I’m about to binge watch the series :-)
Love it Andrew! Thank you!
And if you know anyone else who is interested in RL, it would be huge for me if you shared it with them :)
I can only agree! Very, VERY, very! nice!
discovered your channel looking for Fisher Information explanations, and I can tell your channel's content is really good for overview and have nice explanations, and love that you mention that you need to get into books and maths to go in details
love your content
keep it up !
I really like how you go through a concrete episode. That makes the formalism come to life. Best into to RL I've gone through. Thanks!
Thanks - it's nice to hear from people who appreciate the same things I do
I found your channel looking for RL, I started watching your videos and you are amazing, your explanations and examples are very clear, I hope you continue making many more videos!
Thank you! I''m doing my best :)
I wish these videos were available when I started learning RL years ago. Nice work!
This is phenomenal maths communication. better than anything I have received in my three year undergraduate degree
In order to learn Deep learning especially RL, I've been revising my algebta calculus for 2 months. extraordinairily, I found your channel
This is one of the best, if not the best, explanation I have seen!
Can't wait to watch the upcoming videos in the series!
You won't have to wait long :)
amazing work, looking forward to all the parts
This video is criminally under-watched. It perfectly clears up the core taxonomy of RL that I was confused about up until now.
University of Alberta's Coursera course and Sutton & Barto's book hurt my ignorant self; but you ease some of the pain. Thank you.
At 05:24, I think $r \in \mathcal{R} \subseteq \mathbb{R}$ is more appropriate than $ r \in \mathcal{R} \in mathbb{R} $. Loved the video. Looking forward to watching the whole series.
Ah yes.. that is a better.. silly mistake. Not quite a big enough error to warrant a re-upload, but I appreciate the note
I'm doing a quick refresher, and I am loving your videos, thank you so much, you've clearly done a lot of hard work, both understanding and explaining in a structured and logical way, and it is beautiful!
Also, if I may, I want to leave a note for those who are just starting, while I still remember the parts that caused me trouble years ago when I myself was learning this.
Despite my great respect to Sutton, both the book and the historical development of RL have not been straightforward, both suffer from idiosyncrasies, ad-hoc solutions and sometime confusion.
My advice to a student would be - it's not just you, it's the topic. Take a deep breath and DECOMPOSE what you are reading.
Take, for example, the simplest formula in RL, `P(s',r|s,a)`. It often throws people off, because s',r is a joint distribution. Forget it for a moment, assume that P(r'|s') is deterministic (that's what many coding examples do), and you'll be left with a much clearer form `P(s'|s,a)`, with deterministic function `R(s)`.
Another similar situation is the term "policy", and specifically policy being probabilistic. I would advise the first time a student reads the book to forget about this probabilistic nature of the policy. Having done that we have a much clearer implementation of the policy function, in fact, the standard coding interview dynamic programming question like knapsack problem, where we populate a memoization table. That table is nothing else than policy. And if we instead of storing it, we replace the table with a neural net, you got the idea..
Having decomposed things this way, you'll see how everything downstream is going to be just this or that block augmented with one more variable or replaced with a neural net.
For more advanced students, I'd strongly advise to look into optimal control, as it's like a "smarter big brother of RL", the environment is known, opening doors for proper math analytical solutions, you'll see RL better after visiting this parallel universe, and as a bonus - fall in love with Laplace transform.
PS: "Dynamic programming" is a misnomer, there is nothing dynamic nor it is about the modern meaning of the word "programming" 🤦🏻♂ .
Thanks for the comment Mat! I am currently doing my masters in RL and I do get the concepts but the real problem is in the code itself, trying to implement it from scratch is like reinventing the wheel I should say, but I think this is an important step for me to actually understand the underlying concepts. Understanding concepts from books is something and implementing it is literally something ELSE.
Handsoff the best explanation/teacher video I've seen in my entire life.
well done !
Thank you dude!
What a great lecture! I have been reading Sutton’s book, however the material is often dense and abstract. I like how you visualise the computation processes for us, it really helps understanding the concepts clearly.
Thank you. Looking forward to the rest of the series.
When I was reading the book originally, I was trying to think of these processes in my head. I was sure many others were trying the same.. and that's a big motivator for this series
Big thanks for comeback!
Found this channel too late. This is an amazing overview. I said "extremely cool" exactly when the instructor did in the video, bonus points!! :D
We're on the same page :)
Starting this series now, really looking forward to it! Your videos have been great so far.
Love it Tobias - the series will treat you well!
Ooooooooo look at these energetic edits and the new lighting! Very nice!!
My 3 yrs old son probably would understand what RL is by watching this video. The most clear and distinct video of RL I found on TH-cam so far. Thanks for sharing it!
You have one smart son! :) Thanks for watching!
These videos are really quite good. I'm going through David Silver's course, then using your videos as a great review and reinforce my understanding. You really do a great job of identifying the concepts that noobs like me would have trouble with and then thoroughly explaining them. I also appreciate that there's no fluff, but lots of substance. Keep up the great work!
Thank you very much Miro - glad they're helping. David Silver's lectures have been a solid source for these videos. Also, Hado Van Hasselt's recent series is excellent. I've put in the time for animations and info-density, but those guys are the pros!
AWESOME VIDEO! Best I've seen on the topic BY FAR
ha yes you get what I'm going for! thanks!
Really like how you break problems down to a low level, including spelling out explicitly what all the variables & their symbology mean. Ty!
Yea, that's where the confusion happens. I do it because it confuses me when people don't!
@@Mutual_Information you've also nailed the balance too. it's equally as tedious when people read out and overly explain
Always the best. I was looking forward to your next block of videos... and they didn't disappoint! :)
Happy to hear that :) More on it's way
Wow, one of the most explanatory videos on RL.
My goodness you are an amazing teacher, keep it up. I have read up to the end of chapter 3 in Sutton and Barto so far - and it was all a bit nebulous in my mind but these videos really helped collate my intuitions on the basics of the topic.
That's excellent to hear. You're exactly the audience I'm going for. Sometimes I wonder if these concepts are only clear as I write them, but not necessarily clear to the listener. So it's nice to know it lands.
17:00 Yes, it’s true that the assumptions about being able to reference a world state do not really apply.
But I feel that essentially, this is a large part of what we learn when we develop a skill.
Your memory and experience is a gradually developing world state. And thus, experts do repeatedly reference an internal world state as they conduct the task.
The trained agent is like an experienced expert when we analogize this over to real life.
Props to you man!! This video was absolutely great and explains the topic so good. Thanks a lot and i hope i will understand the next video as easy as this one...
Great video! Excited to see the rest of this series. Nice work!
Thanks MVP ;)
Looking forward!!
My Man! Right on time as my RL project in university is starting haha ty♥️
Wow! Just beautiful.
your channel deserves 10 million followers. very good content! ⚡️
A 1000X compliment - thank you!
Awesome Content, Clearly Explained🔥🔥
Dziękujemy.
I highly recommend people here interested in RL study from Sutton and Barto. I own three RL books (including S&B) and 5-6 if you count books that include some RL methods but aren't entirely about it. S&B is honestly one of the best self-study textbooks I have encountered, it is a classic for a reason. It is much better and easier to learn RL from it than the other books I have (and indeed any source). S&B does a great job in building intuitons and motivations and it orders topics in the perfect order from start to finish (at least for parts I and II, I am less familiar with part III). MI's videos are great, but it is hard to grasp without reading S&B first. If you really want to understand this stuff, I highly recommend reading from S&B directly in addition to videos like these.
Agreed. There's no competing with the original source - it's a phenomenal text
Thank you.
Truly great video!!!
Thank you!
it is amazing works !!!! Please keep publishing RL video, we r desperately to see it !!!!
Ok I will! But I need a favor from you - tell EVERYONE! ;)
I do hope you can keep this up when you become famous
Very nice introduction into this topic! I was just about to start in RL and i find this video extremely motivating
The hope is to lower the cost of learning (from reading a textbook to watching a video) - sounds like it's work! Though, you should def still read the book lol
@@Mutual_Information You are doing a great service. I know i know, i will read it haha
Thank you ! Very good video 👍
I don't comment on YT. but sir you are a great teacher. please keep going.
Thanks, editing a video now, so no plans of slowing down
Thank you for your excellent content
and thank you for watching
Awesome!
Finally, someone understand his audience.
Keep up the awesome work
Thanks for this series.....@9:50, you nailed it.
This is so detailed. I love it! :)
As you know, the details matter - glad you like it!
Thanks for making this, the educational value is miles ahead of most material you can find online.
Just a nitpick, if you don't mind: at 5:15 the second ∈ should be ⊂ to denote subset (or ⊆, depending on the convention)
Yea someone else pointed that out. Oops! Not quite a big enough deal to warrant a re-upload, but thanks anyway
@@Mutual_Information I demand a re-upload.
@@AlisonStuff lol I accept demands from anyone other than my sister
Wow. Really amazing topic.
Thank you - yea it's quite a hot topic (though not as hot as it was 4 years ago).
Keep pushing through. I bet your videos will hit big soon!
So your videos Are very concrete and technical. do you have any video ideas for the big picture around reinforcement learning or other data science topics? Perhaps like how do you think like a data scientist or how do you foster creativity when working with a data set?
Fortunately I've become quite good at being patient with growth. I'm keeping my expectations quite reasonable.
Regarding big picture stuff - I'd like to do advice pieces, but they aren't high in the queue currently. I'm eager to have some solid TH-cam wins under my belt before I speak very generally.. but I will one day. Maybe starting early next year.
I bought this book 2 months ago. I just couldn't understand it. something a bit off. you don't know how valuable your video is to give me the idea on what is the book trying to tell me. Thanks you! I am so excited to continue my learning.
You are welcome! That's exact the circumstance I was trying to solve for :)
Absolutely amazing playlist! Also, at 5:11, calR is a subset of the real numbers; not an element of the real numbers.
ha yea you found it. That's a mistake, but too subtle to warrant a re-upload
Thanks!
No thank you Abdulfatah!
Great production value!!
As does ML Street Talk! Thanks for the love
Oh gosh, had no clue what you're talking about while watching the Fisher Information video...but that RL one is a pure gold! I read mentioned RL brick (e..g. book by Sutton and Barto) and I can hardly imagine better explanation and better summary of the MDP and RL general concept. I hope you can follow up with a Python code. If that's going to be as simple, concise, precise and informative as the video it will be wonderful. Thank you!
Thank you! There is *some* Python code, but not that much. The Python code gets light usage, so I haven't invested in it very much.
But! I do have a strong habit of answering questions here. So ask away if you have questions. And you can ask many - totally happy to get you to a place where you think you have a good grasp of the subject.
@@Mutual_Information Sorry for a stupid question: what do you mean by "Python code gets light usage"? It's my English 🙁
@@marcin.sobocinski ah sorry - i meant “in the past, people don’t use the Python code very much. So I haven’t spent a lot of time on Python code for these videos.”
Honestly, I really comment but this is an amazing video!
Terrific, very well explained and well paced. Minor typo: @16:02, where it says -16 it should say -14 (doesn’t affect the best policy but might avoid confusion of the careful observer)
Ah damn, you're right.. man I really thought I checked the sh*t out of that. lol oh well. I've included a note in the description - thank you!
How about using RL instead of PID regulation? (like adaptive control)
Before I make that comparison, I'd do a quick video on PID controllers. It would be harder to discuss using RL *instead* of PID because I'm not familiar with any data/papers which discuss what happens when that migration happens.
But PID controls themselves are quite cool. Very simple and very effective tools. Now I want to do that video.. hmm
Finally 🫡
6:27 Anyone knows the use case to understand Policy-pi as a probability distribution? In RL, most case I encountered, policy-pi is a specific choice of action. Thanks in advance.
There are some problems where the optimal strategy is not deterministic. The book gives a toy example (I think in the policy gradient chapter). Outside of that, I could image poker being an environment where it pays to be genuinely randomly at points.
@@Mutual_Information Thanks so much for the response, never expected to get a response from the man himself, never mind this fast... :D Thanks for the clarification.
This is such a beautiful explanation. I spent several years getting my head around RL, q-learning, policy gradients, etc and have several "gaps" in my understanding. Your way of explaining is so much on point and yet simple to understand, - a mark of real knowledge. I'm looking forward to watching these lessons. Thank you!
PS. came here from Yannic Kilcher :) th-cam.com/video/r8wiBA3ZaQE/w-d-xo.html
Glad you enjoyed it Igor ! Yea, Yannic's shoutout was a nice boost :)
Great Intro! I ended up with a small question though: At 8:16 you present the formula of the return with the steps of the summation being k=t+1, while t should be an element of (let's just say it like that for the sake of this question) whole number {0,1,2,...}. Now, would that mean that k=0 in all cases? Thereby reducing gamma to 1, since it's always calculated to the power of 0?
E.g.: t=5, k=5+1=6, gamma^(6-5-1)=1
Hope I didn't experess myself to confusingly...
G_t is the sum of future rewards at time t. So if there are 4 periods in an episode and the reward is always 1.2, then e.g. G_1 = gamma^(2-1-1)*1.2 + gamma^(3-1-1)*1.2 + gamma^(4-1-1)*1.2. In other words, k changes from 2 to 3 to 4 in the exponent of gamma and t is fixed over the sum, so gamma isn't always a fixed power
Does that answer your Q?
@@Mutual_Information yes, thank you very much. I didn't watch the full video yet, cause I looked at it in my lunch break and time was up, but I'll make sure to watch the rest today. Thanks a lot for the swift response!
Anyone know this algorithm?
"Generalized Data Distribution Iteration"
As a control system researcher, how can I make use of this for making a contribution and producing research papers
Hm, well this is just an intro series - I won't find anything cutting edge, but maybe it'll inspire some directions?
Well explained! But in 13.40, in the episodic case when we have a time limit T I think the action and time value function are time (t) dependent, because here the number of expected rewards starting in state s really depends on how many possible rewards are left. So in this case t should be another variable in the functions name. Independence makes only sense for the infinite case I think
I thought about it and it also makes sense when T is the stopping time defined as the first t for which S_t enters a final state. But this is Not completely trivial i think. If T is some finite constant the value should be time dependent. Just in case anybody else stumbles across the same problem..
Great content. The only thing that irritates me a lit bit and even in the book it does not develop so much is what the expectation value function exactly express mathematically, I mean, what is taking the expectation E_pi[G | s], as G can be an infinite sum of Rewards that depends on previous states and actions, for me this notation just express the concept in a high level way.
I know what you're referring to. E[] is an operator where you have to imagine an integration and the specifics of that integration are omitted, but they matter! Ultimately, you just get used to it..
Assuming that a process is Markov is not a strong assumption. All processes can be made Markov by including all relevant (past) variables/states in the state variables.
That's true in principle, but it's hard/impossible in practice for real world environments. E.g. markovian models are used alot for financial prediction, but no one has written down a state space that fully captures the history of the economy and stock market. If you just say the latent space *is* all historical observations, that's a huge state space, and so you're not getting anything out of the markov assumption.
I already subscribed
ついに来たか！
I subscribed, but if you fix the backgrounds on your monitors to actually align continuously, I'll share your channel with a few friends. As it stands, I have to keep this between us, I can't be associated with this kind of thing.
lol!
Are you still working at Lyft full-time? Can't imagine how much work goes into these videos... no way you are able to do this in your free time?! Anyways, happy to see new videos from you, keep it up :)
Ha thank you - yea I have a full time job. That’s why I hadn’t posted all year - it does take time!
Hey... How can we connect bro
Twitter DM?
Excellent series, but i do need to get that wallpaper haha
Here you go: github.com/Duane321/mutual_information/blob/main/computer_background/background.png
1:00
if the results for each action are random, they are meaningless and how could you possibly learn anything?
They have a random component, but they aren't completely random. If I say action A gives you a 100 dollars plus or minus 10 dollars with equal probability and action B gives you 20 dollars plus or minus 2 dollars with equal probability, which would you prefer? Both outcomes are random, but one is clearly better.
@@Mutual_Information but in your example the action has nothing to do with the reward that is received, it's completely random. There is no $100 or $20 comparison to be made, the rewards are all -1, 0, 1, and chosen at random regardless of what action is taken.
@@Torqable Oh i see. In that example, I'm just trying to show the mechanics of the interaction between the policy and environment. But there are some slight differences. The up action makes 1 a bit more likely than -1 as compared to the down action.
@@Mutual_Information well I'm very new to this so apologies if I'm just not getting it. But I thought the probabilities were supposed to be a sort of confidence in receiving a reward vs a penalty and therefore used to select an action, not used to select a reward after an action is taken. Is that incorrect? Also I know it's just an example but wouldn't down sometimes be correct? Am I over-thinking it?
@@Torqable I don't think you're over thinking it. It's not an easy subject. What you're thinking about is the quality of the policy, and this is just a very weak policy.. and so it looks like nothing is happening. But that's something specific to this case. I'd just continue to consider example and eventually, it'll make sense. In other words, it's hard to form a full view with just this example.
I came for the money. I left with a triple PhD in astrophysics and quantum mechanics.
came here from ML news
Getting Yannic's shoutout is a big help. Otherwise it's hard to get attention on such in-the-weeds topics.
If you actually wrote this out as python code, rather than a bunch of fancy looking math, it's not that impressive
"They use RL to contain nuclear plasma, which is really cool, and not just practical." I think you're pretty wrong here.
Well if I said that, that would be wrong. I try to be careful with my words. I said the paper shows they can do this, not that this is actually done.
But, with such subtly is probably not how the audience interprets it. And so I see your point. A caveat would have been appropriate.
Thanks!
Wow! Biggest donation ever! Thank you!!