00:00 review of last lecture 03:14 sub-problems for strings/sequences(suffixes, prefixes, substrings) 05:55 parenthesization 23:56 edit distance 41:35 running time of edit distance 42:40 knapsack 50:06 running time of knapsack
That's actually an amazing conclusion - that the knapsack problem is not polynomial. It makes sense, but I'd never thought about that till Erik explained it. Kudos.
What I've been missing from these lectures so far (particularly for the knapsack problem) is the insight that _'recursion + memoization DP'_ can be much more efficient than _'bottom up DP'_ if you don't ultimately need the solutions to _all_ sub-problems in order to solve the original problem. For instance, if you have a knapsack capacity of 10 and four item-sizes 1, 3, 6 and 8, then you'll never need solutions for a remaining capacity of 5 or 8, because no combination of items take up size 5 or size 2. I suppose you can get around this by reformulating the set of sub-problems to exclude those, but it'll be very inconvenient to systematically iterate over such an awkward set.
+Javier Torrez: I suspect you didn't fully read my comment. Try to run Erik's knapsack problem bottom-up and then recursive, on the example input I give above. Then notice how the recursive version never has to solve the "remaining capacity = 5" and "remaining capacity = 8" sub-problems. Their solution is not required. But with the bottom-up approach, you won't know this until after you've already solved them.
True that, but we have to also remember recursion use stack and less efficient than the iterative solution. In both case, time and space complexity are the same if you consider the big O notation. I personally find debugging is way easier with bottom-up iterative approach.
@Nil, I'd say in cases where all subproblems need to be explored (Fib, matrix multiplication, etc.) then bottoms up is more efficient because it has better constant factors. But in the case of Knapsack, where you don't need to explore all subproblems, I'd say recursion+memoization is probably faster. Either way, the runtime is the same.The difference is negligible.
It always irks me when professor say something is "super easy". But, once you I understand the concept clearly, I can see why someone would describe it like that
In some examples I've seen for edit distance, they also include an implicit zero-cost "no op" operation when x[i]==y[j] (which could also be modeled with a zero-cost replace(x[i], y[j]) when x[i]==y[j]), even though in the problem statement they say operations are only "insert, delete, and replace" with equal costs. This lecture doesn't discuss what to do when x[i]==y[j], but since he DOES say that replacement cost could be variable, I suppose we have to assume that the cost to replace a character with the identical character is zero. Without some version of "no op", I think you'll end up with a different min cost and set of operations than the examples.
One interesting observation I had while watching this video is the DAG of the knapsack problem. This goes back to graph transformation problems from previous lectures. You can picture there been S levels for each weight and N items/nodes at each levels. So you'd have S*N vertices. The edges point downwards and signify adding an item to backpack. Edge weight are the values. You'd find the longest path from Start to End where they are both 'dummy' nodes. All vertices have an edge to 'End' of length 0. The start node would have edges to all 'N' items are the levels corresponding to their weight,
I believe that when you go by suffix, you could also go by prefix, and vice-versa. E.g. you can solve the knapsack also by prefix of items, changing the recurrence relation and base case accordingly.
What makes MIT great is they still use chalk boards instead of powerpoints to give lectures. It really keeps students engaged. Professors and students almost everywhere else have gotten lazy.
In my first semester teaching I used the Powerpoints supplied by the author of the text I was using. And never again. I prefer my students to stay awake.
604 00:34:04,940 --> 00:34:09,020 And three possibilities are insert, delete, or replace. about this, why not "do nothing if x[i] = y[j]" as another possibility?
Correct me if I am wrong. I think this possibility has been included in the first case of replace, where c = c', mean the characters are the same, hence we don't need to change anything and the cost is equal to 0.
hi 盧宥諭 , thx for ur help!, I think prof Erik mentioned what you said. 536 00:30:03,960 --> 00:30:06,450 Because replacement says, I don't pay anything 537 00:30:06,450 --> 00:30:09,710 if the characters match exactly, otherwise I pay everything. And also, taking look at hw 7 of 6-046J, 2005 version : ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-046j-introduction-to-algorithms-sma-5503-fall-2005/assignments/ps7sol.pdf there is another way of seeing this problem : the problem explicitly listed "move right" as an operation. I think maybe prof Erik is going a little fast when discussing edit distance? So he did not repeat? re iterate? some assumptions? points easily confused? Or at that point, students didn't ask this question? But, I guess maybe that's the pace for todays ultra competitive world?
Why is the number of subproblems n*S for Knapsack and not just n? If there are n items that you can take with you, you have a subproblem for each of those items, no?
why he took subproblem as sufffix for knapsack as it should be subset of list of items ,cause we dont really care about sequence of items,can anyone explain please
Maybe I am late as hell, but when talking about edit distance in such detail that we have a value dependent replacement costs, then why did the lecturer discarded options of symmetric operations on x, y. Specifically, he mentioned inserting in x, and deleting in x, and left out deleting from y and inserting into y, but if the deletion cost is different than the insertion cost then the following example would have incorrect solution: deletion cost = 1 insertion cost = 10 x = "" y = "ABC" since the original solution only cares about how can we manipulate x to get y the result would be 30, because we cannot delete anything from an empty string. If this asymmetry is present, then the editDistance would not satisfy the requirements of a distance function, but still at 36:05 he states "[these three options] covers every possible thing you can do". Please, if you find any flaws in my logic, then do correct me, I am really excited to have a chat about this topic
Csaba, I think you are right. The catch is that the edit distance calculations are generally asymmetric. We want to find minimum number of edits in string a in order to change it into string b. In this process we leave b unchanged. This can help:- stackoverflow.com/questions/32550864/edit-distance-algorithm-explanation
bro it is like for i : str for j : str you are always considering substring from i to j and you will get the combinations from loops mentioned above so it o(n^2)
watching those videos, I want to be an elite / scientist this is for you. To me, it's time for regular people. those does not make sense to regular people. make sense to scientist.
10:30 - Why wouldn't we find all of the pairs that produced single numbers and do those first? Or, more generally, seek to produce the smallest result matrices possible? I would scan the list and find the smallest result I could produce, and produce it. Then repeat until done.
Your matrix of optimal values varies 1. on the index of an item in the list and 2. on the capacity on the knapsack - and the knapsack can be filled to level up to S.
I'm confused about the parenthesization problem, in the beginning he explains row * col is n^2 and then the multiply operation to another col would be n^2 which I get, but how does changing the order of the parentheses change the complexity to n? He still multiplies row * col and then col...what am I missing?
Professor Erik mentioned that in Knapsack we iterate over the suffix of items [i:] but we are actually iterating over the prefixes of item. Can someone else confirm?
@Yunqi Shen Its DP[N, X] = 0. Where n in the number of items and X is the max weight allowed. When we reach the Nth item, it means we don't have any more items to consider
@@anamm3071 It is suffix itself, you have a bigger subproblems at i= 0 i.e max(dp(1,size-1)+Vi,dp(1,size)) , but as you reach n-1 term you have only one subproblem.
I’m I the only one confused about the fact that the number of substrings is O(n^2) because I think it’s of the order O(n!) which is closer to O(n^n) than O(n^2) -- I need help with this in case I’m doing something wrong 🤲🏾
+Luis Daniel Mesa Velasquez I think he ran out of time to get to the base cases. The way I implemented it, you would have a case if capacity - i.size < 0 and skip the current i instead of finding the max of the skip and the add. Does that make sense? If you want, I can share my code for it.
Shouldn't the time for parenthesization be n^2 instead of n^3? Can someone explain why he multiplied the total recursion time with the number of choices at each step? It doesn't feel correct.
He isn't counting the inside recursions here just like he didn't on other examples. But note that these are 'DP(i,j) =', not 'DP(n) ='. So there are n^2 recurrences * n time/subproblem. Actually it is (n^2)/2 since you have the 'i
Kind of expected better quality lectures from Dr. Demaine. It's good but only for those who already have a good understanding of dynamic programming. Otherwise tough luck.
How would you solve a coin change with unlimited coins.. ie knapsack with unbounded. I dont see a way to solve it efficiently using prefix/suffix way- leetcode.com/problems/coin-change/
Seems like he was having a rought week cause he kept erasing and messing up on the chalkboard. if I was student in that class writing in pen, i would despise him for making my notes all scribbly.
Parenthesization 6:09
Edit distance 24:17
Knapsack 43:14
Thanks a lot!
thanks :)
I forgot what the problem was being solved at ~23, lol
thanks
just watch full lection guys all these problems matter in ur programming future.And dont act like u dont have time for high quality content.
One of the best lectures on DP I have ever seen. I used to be afraid of DP. Love for Erik from Bangladesh! You are doing a great job.
Can’t agree more.
This video made me realize how all of my cs projects for college incorporated aspects of dynamic prohramming
The best explanation of "edit distance", it's recurrence with a splendid illustration.
Indeed, Erik did a good job, but I think the best explanation came from Algorithms (2006) by Dasgupta, highly recommended
recursion and then 'bottom-uptify' is my favorite strategy now for construction DP algorithms
It is funny how also some of MIT students don't get this at first. It gives me feeling I'm not alone
00:00 review of last lecture
03:14 sub-problems for strings/sequences(suffixes, prefixes, substrings)
05:55 parenthesization
23:56 edit distance
41:35 running time of edit distance
42:40 knapsack
50:06 running time of knapsack
Note that the lecture notes say to maximize the cost for Edit Distance, where as the lecture say's to minimize. I think it should be minimize.
That's actually an amazing conclusion - that the knapsack problem is not polynomial. It makes sense, but I'd never thought about that till Erik explained it. Kudos.
can you explain a little bit? I don't understand the ending part of the lecture.zc
"This one's gonna blow your mind" at 42:42.... Loved the lecture
Everyone better like Eriks videos
This is one the best lectures I’ve seen on DP.
What I've been missing from these lectures so far (particularly for the knapsack problem) is the insight that _'recursion + memoization DP'_ can be much more efficient than _'bottom up DP'_ if you don't ultimately need the solutions to _all_ sub-problems in order to solve the original problem.
For instance, if you have a knapsack capacity of 10 and four item-sizes 1, 3, 6 and 8, then you'll never need solutions for a remaining capacity of 5 or 8, because no combination of items take up size 5 or size 2.
I suppose you can get around this by reformulating the set of sub-problems to exclude those, but it'll be very inconvenient to systematically iterate over such an awkward set.
+Javier Torrez: I suspect you didn't fully read my comment.
Try to run Erik's knapsack problem bottom-up and then recursive, on the example input I give above. Then notice how the recursive version never has to solve the "remaining capacity = 5" and "remaining capacity = 8" sub-problems. Their solution is not required. But with the bottom-up approach, you won't know this until after you've already solved them.
You're right, looks like I misunderstood your comment - deleted my comment.
Very interesting observation. Thanks!
True that, but we have to also remember recursion use stack and less efficient than the iterative solution. In both case, time and space complexity are the same if you consider the big O notation. I personally find debugging is way easier with bottom-up iterative approach.
@Nil, I'd say in cases where all subproblems need to be explored (Fib, matrix multiplication, etc.) then bottoms up is more efficient because it has better constant factors. But in the case of Knapsack, where you don't need to explore all subproblems, I'd say recursion+memoization is probably faster.
Either way, the runtime is the same.The difference is negligible.
It always irks me when professor say something is "super easy". But, once you I understand the concept clearly, I can see why someone would describe it like that
One of the best lectures Ive ever had. Erik is amazing!
29:44, if the answer we're looking for doesn't have to be a meaningful word, then "iello" and "heglo" also meet the criterion
it's the same length as hello, but ihegllo can be one.
In some examples I've seen for edit distance, they also include an implicit zero-cost "no op" operation when x[i]==y[j] (which could also be modeled with a zero-cost replace(x[i], y[j]) when x[i]==y[j]), even though in the problem statement they say operations are only "insert, delete, and replace" with equal costs. This lecture doesn't discuss what to do when x[i]==y[j], but since he DOES say that replacement cost could be variable, I suppose we have to assume that the cost to replace a character with the identical character is zero. Without some version of "no op", I think you'll end up with a different min cost and set of operations than the examples.
Edit Distance: 24:00
One interesting observation I had while watching this video is the DAG of the knapsack problem. This goes back to graph transformation problems from previous lectures. You can picture there been S levels for each weight and N items/nodes at each levels. So you'd have S*N vertices. The edges point downwards and signify adding an item to backpack. Edge weight are the values. You'd find the longest path from Start to End where they are both 'dummy' nodes. All vertices have an edge to 'End' of length 0. The start node would have edges to all 'N' items are the levels corresponding to their weight,
I believe that when you go by suffix, you could also go by prefix, and vice-versa. E.g. you can solve the knapsack also by prefix of items, changing the recurrence relation and base case accordingly.
just before he starts with Knapsack at 42:40 .."this is gonna blow your minds" ...whispers "hopefully"
I learned more things from this lecture for free than in university where my parents paid hundreds of thousands of dollars.
What makes MIT great is they still use chalk boards instead of powerpoints to give lectures. It really keeps students engaged. Professors and students almost everywhere else have gotten lazy.
this was uploaded 10 years ago mate
@@thedavidji I've watched more recent lectures as well mate. Can be a general comment.
In my first semester teaching I used the Powerpoints supplied by the author of the text I was using. And never again. I prefer my students to stay awake.
longest common substring example was great. awesome setup too
longest common sebsequence 28:00
it was a good run. goodbye guy and thank you.
604 00:34:04,940 --> 00:34:09,020 And three possibilities are insert, delete, or replace.
about this, why not "do nothing if x[i] = y[j]" as another possibility?
Correct me if I am wrong. I think this possibility has been included in the first case of replace, where c = c', mean the characters are the same, hence we don't need to change anything and the cost is equal to 0.
hi 盧宥諭 , thx for ur help!,
I think prof Erik mentioned what you said.
536 00:30:03,960 --> 00:30:06,450 Because replacement says, I don't pay anything
537 00:30:06,450 --> 00:30:09,710 if the characters match exactly, otherwise I pay everything.
And also, taking look at hw 7 of 6-046J, 2005 version :
ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-046j-introduction-to-algorithms-sma-5503-fall-2005/assignments/ps7sol.pdf
there is another way of seeing this problem :
the problem explicitly listed "move right" as an operation.
I think maybe prof Erik is going a little fast when discussing edit distance?
So he did not repeat? re iterate? some assumptions? points easily confused?
Or at that point, students didn't ask this question?
But, I guess maybe that's the pace for todays ultra competitive world?
Whoa now I seem to understand dynamic programming much much better!
I believe that great teachers are very rare. Richard Feynman was one such great teacher. Erik Demaine is apparently another.
23:50 - The only way to understand this crystal clear, is to look examples and write code to solve it.
knapsack at 42:41
To iterate, human
To recurse, divine
To DP, devil
Could someone help me with what "guess" is? I don't quite understand what it is
Mr. demain is amazing! this is just pure gold, thank you.
28:00 Longest Common Subsequence Problem (LCS)
Why is the number of subproblems n*S for Knapsack and not just n? If there are n items that you can take with you, you have a subproblem for each of those items, no?
I’m a first year computer science student and now I’m scared
that's what she said
@1:54 "bottom-up version of the DP"
Allen Day check lecture 19 of this series (dynamic programming 1). he explains in third version of Fibonacci solution
The funniest thing is that the two other people in the comment thread don't get the joke hahahha
DP: "careful brute force"
Watch this video twice, you will be amazed.
Blew my mind
I love't this prof.
Amazing!
why he took subproblem as sufffix for knapsack as it should be subset of list of items ,cause we dont really care about sequence of items,can anyone explain please
Can we do knapsack in W space in top down?
Maybe I am late as hell, but when talking about edit distance in such detail that we have a value dependent replacement costs, then why did the lecturer discarded options of symmetric operations on x, y.
Specifically, he mentioned inserting in x, and deleting in x, and left out deleting from y and inserting into y, but if the deletion cost is different than the insertion cost then the following example would have incorrect solution:
deletion cost = 1
insertion cost = 10
x = ""
y = "ABC"
since the original solution only cares about how can we manipulate x to get y the result would be 30, because we cannot delete anything from an empty string.
If this asymmetry is present, then the editDistance would not satisfy the requirements of a distance function, but still at 36:05 he states "[these three options] covers every possible thing you can do".
Please, if you find any flaws in my logic, then do correct me, I am really excited to have a chat about this topic
Csaba, I think you are right. The catch is that the edit distance calculations are generally asymmetric. We want to find minimum number of edits in string a in order to change it into string b. In this process we leave b unchanged.
This can help:-
stackoverflow.com/questions/32550864/edit-distance-algorithm-explanation
Can we solve a suffix sub-problem using prefix technique? If not, how to identify prefix suffix and a sub-string sub-problem. TIA :)
Why would one want to insert in the longest common subsequence ?? Shouldn't the cost of insertions be infinity for the problem ?? I don't follow
Insertion in x basically means we are deleting that letter from y and moving forward.
@@prakarshsaxena2063 , could you please elaborate a bit more, I still couldn't get it. Many thanks
I prefer to think of like as below
𝐿𝐶𝑆(𝑖,𝑗) = 1 + 𝐿𝐶𝑆(𝑖+1, 𝑗+1) 𝑖𝑓 𝑥[𝑖]=𝑦[𝑗]
= max(𝐿𝐶𝑆(𝑖, 𝑗+1), 𝐿𝐶𝑆(𝑖+1, 𝑗)) 𝑂𝑇𝐻𝐸𝑅𝑊𝐼𝑆𝐸
5:56 - How are there N^2 substrings? I see N! substrings. There are N of length 1, N-1 of length 2, N-2 of length 3, ... and 1 of length N.
bro it is like
for i : str
for j : str
you are always considering substring from i to j and you will get the combinations from loops mentioned above so it o(n^2)
N + (N - 1) + (N -2) + ... + 1 = N*(N + 1) / 2
DP(i,j) = min { DP(i,k) +DP(k+1,j) + cost of last matrix multiplication}, i think....
how do you get the minimum while you are adding all the elements. That sounds like a sum. min() is comparison.
Thats the par I don't understand.
Such a great lecture. Salute.
Knapsack -> 44:10
watching those videos, I want to be an elite / scientist this is for you. To me, it's time for regular people. those does not make sense to regular people. make sense to scientist.
10:30 - Why wouldn't we find all of the pairs that produced single numbers and do those first? Or, more generally, seek to produce the smallest result matrices possible? I would scan the list and find the smallest result I could produce, and produce it. Then repeat until done.
awesome lecture, thank you
In Knapsack, why number of subproblem is N times S? @49.10
Your matrix of optimal values varies 1. on the index of an item in the list and 2. on the capacity on the knapsack - and the knapsack can be filled to level up to S.
Did anyone implement edit distance as described in the video?
Love knapsack
How is the number of subproblems for the parenthesization problem n^2?
Here: stackoverflow.com/questions/24901537/how-many-substrings-of-a-string (the second answer's really good)
I'm confused about the parenthesization problem, in the beginning he explains row * col is n^2 and then the multiply operation to another col would be n^2 which I get, but how does changing the order of the parentheses change the complexity to n? He still multiplies row * col and then col...what am I missing?
It's specific to matrix multiplication.
why number of subproblem is (n*S)
Best dp video
I finnally found this
Professor Erik mentioned that in Knapsack we iterate over the suffix of items [i:] but we are actually iterating over the prefixes of item. Can someone else confirm?
@Yunqi Shen Its DP[N, X] = 0. Where n in the number of items and X is the max weight allowed. When we reach the Nth item, it means we don't have any more items to consider
@@anamm3071 It is suffix itself, you have a bigger subproblems at i= 0 i.e max(dp(1,size-1)+Vi,dp(1,size)) , but as you reach n-1 term you have only one subproblem.
I’m I the only one confused about the fact that the number of substrings is O(n^2) because I think it’s of the order O(n!) which is closer to O(n^n) than O(n^2) -- I need help with this in case I’m doing something wrong 🤲🏾
Woo! Canada represent!
Did he mean 2n instead of n^2 when he said the total number of substrings for a string of length n
No. It's n^2
Good one 🤙🏽
Thank you!
30:55 Continues Edit Distance
Where does he check if X in Knapsack is greater than 0?
+Luis Daniel Mesa Velasquez
I think he ran out of time to get to the base cases. The way I implemented it, you would have a case if capacity - i.size < 0 and skip the current i instead of finding the max of the skip and the add.
Does that make sense? If you want, I can share my code for it.
Gerardo Veltri thanks! you can always share code, nothing wrong with that =)
Thanks.
Shouldn't the time for parenthesization be n^2 instead of n^3? Can someone explain why he multiplied the total recursion time with the number of choices at each step? It doesn't feel correct.
He isn't counting the inside recursions here just like he didn't on other examples. But note that these are 'DP(i,j) =', not 'DP(n) ='. So there are n^2 recurrences * n time/subproblem. Actually it is (n^2)/2 since you have the 'i
"With difficulty..." LMAO:)
What ?
Kind of expected better quality lectures from Dr. Demaine. It's good but only for those who already have a good understanding of dynamic programming. Otherwise tough luck.
This is the 3rd lecture in the series. Try watching the first DP lecture by him and then coming back to this one
@@JanacMeena I did but I still hold to my original comment.
try to solve at least 20 dp questions and then ckme back
@@luojihencha I have and still hold to my original comment. He's good only if you already have a basic knowledge of DP. His pedagogy isn't great.
How would you solve a coin change with unlimited coins.. ie knapsack with unbounded. I dont see a way to solve it efficiently using prefix/suffix way- leetcode.com/problems/coin-change/
confuse about the topological order stuff
I recommend viewing Erik's top sort lecture
vivienne do cheers. :)
+Xubing Liang cheers :)
Seems like he was having a rought week cause he kept erasing and messing up on the chalkboard. if I was student in that class writing in pen, i would despise him for making my notes all scribbly.