I have one question about the lecture: Dr. Brunskill claimed that "if the process is deterministic, then the return and the value function will be equivalent". But the reward by definition is a random variable, i.e. for the same state, there maybe different rewards. So even the process is deterministic where we know the exact next state, we are still not sure the realized reward for that state. Why then the value and return are equivalent for the deterministic process?
About the gammas being in GP has a very good interpretation in finance and I believe it stems from there and is just not mathematical. It does have some mathematical properties though. It's to do with interest which means if we earn 1 now and there's 10% interest, then after 1 year the it is 1.1 which means if after 1 year if I am earning 1, it is equivalent to earning 0.909 now and since interest are always in 10 to 20 25% range ballpark, this gives us rough values of gamma as 0.8 to 0.9 or so. A gamma of 0.5 would mean I would leverage the reward such that it would double in following time step. This is compounded over time and that is how it's a GP. However, this would imply if I have a reward on 1 this year, I can leverage it over following years (collect interest) which seems reasonable to think in terms of learning from experience early on in a sense... However this is my understanding and might be biased..
How return function is different from value function ? How come return will be different from value function when process is not stochastic .( both having sum of reward )
we said if policy is deterministic we can simplify value function to Vπk(s) = r(s, π(s)) + γXs0∈Sp(s0|s, π(s))Vπk−1(s0) but how we can write max(a) Q(s,a) >= V(s) when policy is deterministic and we can choose just one action?
@@gravitas8297 Does beamer allow annotation ? I thought it was a latex class for making presentations ? I wanted to know the annotation tool she is using for iPad. That would be really helpful .
I'm under its spell. I had the pleasure of reading something similar, and I was under its spell. "The Art of Saying No: Mastering Boundaries for a Fulfilling Life" by Samuel Dawn
G o resto x ela quiser vir me CP g vi agora só r ela e e horário então só r r viu se ela quiser e te amo e o valor e horário da manhã r viu se e o valor e horário
25:47
Conjecture: inverse exists if gamma in [0,1), and fails to exist if gamma=1.
Easy to check for 1 or 2 state systems.
True, for gamma < 1 the matrix is strictly diagonally dominant, thus invertible
I have one question about the lecture: Dr. Brunskill claimed that "if the process is deterministic, then the return and the value function will be equivalent". But the reward by definition is a random variable, i.e. for the same state, there maybe different rewards. So even the process is deterministic where we know the exact next state, we are still not sure the realized reward for that state. Why then the value and return are equivalent for the deterministic process?
Can the common or good questions of piazza be put up somewhere to refer to?
About the gammas being in GP has a very good interpretation in finance and I believe it stems from there and is just not mathematical. It does have some mathematical properties though. It's to do with interest which means if we earn 1 now and there's 10% interest, then after 1 year the it is 1.1 which means if after 1 year if I am earning 1, it is equivalent to earning 0.909 now and since interest are always in 10 to 20 25% range ballpark, this gives us rough values of gamma as 0.8 to 0.9 or so. A gamma of 0.5 would mean I would leverage the reward such that it would double in following time step. This is compounded over time and that is how it's a GP. However, this would imply if I have a reward on 1 this year, I can leverage it over following years (collect interest) which seems reasonable to think in terms of learning from experience early on in a sense... However this is my understanding and might be biased..
Does anybody understand how did she get to 2nd step of the equation on 1:11:56?
We dont care about a or a'. Suppposed that BV_k >= BV_j, a_j = a' making the maximum of BV_j. When a_j = a, we get BV_j{a_j=a}
How return function is different from value function ? How come return will be different from value function when process is not stochastic .( both having sum of reward )
we said if policy is deterministic we can simplify value function to Vπk(s) = r(s, π(s)) + γXs0∈Sp(s0|s, π(s))Vπk−1(s0) but how we can write max(a) Q(s,a) >= V(s) when policy is deterministic and we can choose just one action?
Thank you for sharing the contents
What is the tool that Prof Emma is using for the presentation and annotation, it looks really helpful?
Beamer? I guess
@@gravitas8297 Does beamer allow annotation ? I thought it was a latex class for making presentations ? I wanted to know the annotation tool she is using for iPad. That would be really helpful .
@@adityanarendra5886 Err I haven't tried that sorry :(
47:13 Someone just asked what I wanted to! 😂
I'm under its spell. I had the pleasure of reading something similar, and I was under its spell. "The Art of Saying No: Mastering Boundaries for a Fulfilling Life" by Samuel Dawn
V tú e horário normal e o valor da entrada e o valor e horário normal e o valor da taxa de ontem e o valor e horário normal e
G o resto x ela quiser vir me CP g vi agora só r ela e e horário então só r r viu se ela quiser e te amo e o valor e horário da manhã r viu se e o valor e horário