Ok, it took me a bit to understand why the std::mem::replace method works. So here is how I see it: Normally, element is a mutable reference to self.slice[0]. This mutable reference has lifetime 'next, because it is defined within the next function that has a lifetime of 'next. We need to make it have a lifetime of 'iter instead. The only mutable reference to self.slice with a lifetime of 'iter is self.slice itself. So when you do 'let slice = &mut self.slice;', you are creating a mutable reference to self.slice, with lifetime 'next since it is defined in the next function. Then std::mem::replace takes that mutable reference with lifetime 'next, **dereferences** it and sees a mutable reference to self.slice with lifetime 'iter, and returns it to us which is exactly what we want. It is the fact that it dereferences the 'dest' function parameter and returning it (with the lifetime associated with it) which makes it work. -I think it would have been a bit clearer if at --1:06:23--, you would have used 'let slice2 = std::mem::replace(slice, &mut []);' because it is a bit confusing when you see that slice is being replaced with '&mut []', and it's old value is being bound to slice again.- You did this as 1:08:26, I should have watched it through before commenting! I had to draw it out to see how it worked, but I'm glad I did because I think I get it now :D
As a C++ programmer, this is really not that easier. Implementing an iterator of a non-const container in C++ is much easier, but of course also a bit more error prone as you don't get lifetime checks.
@@embeddor2230 of course it's much easier if you do it wrong and full of bugs. the point is that doing the correct thing has the same difficulty in both Rust and C++, but Rust prevents you from writing bullshit code
56:40 I think it comes down to a simple fact. If the data is guaranteed to exist (immutably), and it won't be changed ever (because we enforced mutable borrow laws, so you can't read the data after it has been mutably borrowed), and because we know how long the data will last for (up to the highest lifetime), then because of all these conditions, we can say for certain that the data referenced there will ALWAYS be the same, so it is safe to lengthen it to the longest lifetime in that case (regardless that it was borrowed as 'next). I suppose that in this case, the reference from 'next effectively == 'a
I really like your instruction style, Ryan; thanks for doing this. @35:30 I think the question was this: If I write a function that returns an iterator, should the function signature specify the concrete iterator type that is returned, or should it use "impl Iterator"? And the answer, I believe, is that in general it is better to use "impl Iterator." To be more precise about it, fn foo(...) -> impl Iterator {...} is better than fn foo(...) -> MyIterator {...} It is better in that it preserves flexibility in the implementation -- later you might want to return some other type. If all you promised was that you would return SOMETHING that implements Iterator, then you can make that change. If you promised that you would return a MyIterator, then changing that would be a breaking change in your API.
Hey, I got a question: isn't it weird that we can std::mem::replace effectively self.slice temporarily with &mut []? AFAIK shouldn't any contents of self.slice need to have a lifetime of 'iter (which is clearly not the case with &mut [], which should live on the function stack)? Is this just a special elision trick where the compiler proves that self.slice is never actually invalid or am I missing something?
I haven't written a line of rust yet (cant figure out a project to do) but it would seem that, in the first iteration of the Iterator implementation where you check if the slice.is_empty and return none, that code could panic with a index out of bounds/range "exception" when self.slice has a single item in it. (Unless slice[1..] returns differently than i would expect.) edit I would also like to note that I appreciate it when you repeat the questions from chat (which you have done consistently throughout this and other videos).
I've thought the same thing, so I've actually tested it and it works correctly when the collection contains one element. It seems that slice[1..] does not panic even though the slice has only one element at that moment. Then I tested the following: 1) make collection be a Vec of one element, 2) 'let second_collection = &collection[1..];' works, but 'let second_collection = &collection[2..];' does not, and 'let second_collection = &collection[1];' also does not work. This leads me to believe that when you use slice notation, then you can refer to one element after the last element, but when you refer to *just* the one element after the last (or even further), then it doesn't work. Why it works like this, I'm not entirely sure as I haven't looked into that, but it was interesting to say the least :)
I also thought it would be an index out of bounds. Apparently, one index more than the last index is ok as long as you are using slice notation (otherwise it would be the traditional index out of bounds if not a slice). This finally explains why [1..] never had an error. It'll just simply return an empty slice, or if it's exactly at the end of the index, it'll return the current element I think it's kinda useful though, since this way you can get an item or none. If it went out of index one index after (as normally it should), then all you could do it slice at the last index instead which would return only the last item, and you'd never be able to return an empty slice If you test it out on Python, it works the same way in fact. (Although it might be worth noting you can keep going up and up in numbers without an error, unlike in Rust where you can't) a = [1,2,3] a[2:] // = [3] a[3:] // = []
Where you said that T must live as long as struct, that literally made this make more sense than all the countless guides I read 25:00 "it's more code" \*ends up being less code\*
Some people feel shame over the realization "Why didn't I think if this? It's so obvious now that I see it!". I don't, because that feeling just means that I learned something that I know I would have needed several times before, and - more importantly - will need again in the future. Going back to review an ugly hack I did in a tokio Decoder.. :)
You can move out of self.slice directly by passing &mut self.slice into std::mem::replace(). No need for a placeholder borrow. Thanks for another great stream Ryan, do you know if there’s work being done to make forwarding of lifetimes like this easier?
Q: Is there a differance between std::mem::replace() and core::mem::replace()? why exist in std? I'm coming at Rust from microcontroller space where there is no OS. My impression is that Rust is only recently viable here. The no GC is very attractive for microcontrollers where I want dynamic behaviour with accurate timing always.
Rust crates can be #[no_std], in which case they do not include the standard library std. The core library is always present. std re-exports some items from core.
Question: what is the a problem with using 'iter lifetime for the mut self in the next function? That is, instead of using fn next(&'next mut self), why not just use fn next(&'iter mut self) to enforce the same lifetime is used for both the iterator and each item?
The next function is defined by the Iterator trait in std which has no idea about our `'iter` lifetime. Change the next function to take `&'iter` and it will complain that that's not the right signature for the next function.
So basically it prevents the situation that you borrow a book at the library due next month and find out while reading that the pages inside have to be returned tomorrow 🤔
More like it prevents you trying to modify pages of the book that you have already handed out to the students to eat. It also prevents you to hand out the same page for multiple students to eat, which would lead to a situation of multiple students eating the same page. Ok this example has derailed.
Hello, I'm very interested in how &self.slice[1..] works If we have 1 element left in the slice, then we can get it through [0], but if we use [1] then it will be out of range, then how does &self.slice[1..] work when we have 1 element?
Great video... as usual But there is just one thing that bothers me. It's been mentioned multiple times throughout the video by people in chat that it is impossible to implement mutable iterator without unsafe code and Ryan telling all of us to just wait and see how it is done. Then he shows us "a trick" with std::mem::replace but if you look at its implementation (doc.rust-lang.org/std/mem/fn.replace.html - click on src link) then you see that this method is just a small unsafe block, essentially meaning that we couldn't avoid using unsafe
multiple readers are just as a big problem as writers, as he states right before: imagine the slice becomes invalid while your readers still point to it, the multiple readers would fail or access invalid memory. If you want to support multiple readers you need to keep track of them, e.g. with smart pointers like Rc. For a single reader the compiler can keep track of it for you. At least that is my understanding.
@@MichaelKefeder reaaaalllyy? I wouldn't have thought so, because one reader would still be a problem for the writer, as they would require synchronization Unless you meant the reader as in the writer, because writers can both read and write
@@Dorumin I was answering for the assumption that multiple readers are no problem, when they are. I guess that's why he said it multiple times, without noticing because it is true as well. Sure in the video he is talking about handing out a mutable reference, which can only be handed out once since this gives read/write access, where saying "multiple writers" would be more clear.
Just like in "The Raiders of the Lost Arc". Fingers crossed the borrow checker isn't sending a giant rolling boulder your way. Couldn't this dummy replace trick be added to the standard library as a means to grab the original pointer?
Sooo... MyMutableIterator::next needs to last as long as MyMutableIterator last, and it is understandable, but why then the `next` function has to have its own " 'next" lifetime? Would not it work if the `next` function would have a " 'iter" lifetime? Like this: fn next(& 'iter mut self) -> Option { ... } and then the function body would look the same as the immutable one.
If you try that you'll see that Rust complains that the type sugnature of `next` no longer matches how `next` is declared in the Iterator trait. So you're correct in your thinking, but next does not generally borrow `self` for `'iter` it borrows it for the lifetime that we've called `'next`.
Hi and thanks for this video. I learn rust right now and of course borrowing and life time are really a little montain to grow. I will see over videos from you channel and one of my question actually is, ilke a lot on persons i guess, i comes from object programming and with rust, it seems not to be a good paradigm to code efficiently, because of the temptation to referencing multiples objects each overs for example. Is exists some good tutorial or book who talk about good rust paradigms and how to organize and write good rust program ? Thanks and have a good day.
why do you only get the conflicting requirements error when you do it with mutable references? Why doesn't it complain when you do it with immutable references?
If you have a mutable reference to self, you can either borrow mutably for as long as you have the reference or borrow immutably for as long as the self lives. If you return a longer immutable borrow than the mutable borrow you were given, you wont' be able to mutably borrow (i.e. call next) until the returned element has been destroyed. In the immutable case, as you have a mutable rerefence to Self in next, you know that there are no other mutable references to slice. As there are no mutable references to slice, you can take a borrow from slice for as long as the slice lives. Or in other words, because you have a mutable reference, immutable reference lifetimes can be extended.
I had the exact same issue this weekend and I found the book lacking a bit in this context. What would be the in your opinion another way to get the same result without using the mem::replace trick? Can you extend the lifetime of the returned value to the struct lifetime? Thanks for the content anyway! ;)
The trick is in mem::replace - it makes the iterator itself an owner of this slice pointer. The split_first then is able to return two pointers that aren't connected to any lifetime (and you can do that, because if you are the only one pointing to a memory chunk, you can use split_first to make two mutable pointers to the same chunk as long as they don't overlap). Since the function is now the owner, it can do with these two owned mutable pointers whatever necessary - return first one from function and assign the second one back to the mutable iterator state.
i used rust last time but i find it overly complex..its not much different from c++ except no more header files..other than that, its not much different from c++..its anoyying to use..is it really worth it? making linked list in rust alone make me wanna break my keyboard
Rust is certainly not the easiest language to learn or use. If it doesn't fit any of your use cases, there's no harm in not using it. Linked lists are something you almost never need in Rust and implementing one in safe code requires an intermediate knowledge of the borrower checker so it's not really beginner friendly. I made a video on this: th-cam.com/video/IiDHTIsmUi4/w-d-xo.html
![[LIVE] : ONE ลุมพินี 89 | คู่เอก "ยอดไอคิว vs คิริลล์"](http://i.ytimg.com/vi/Z3vYkHJ_5Ig/mqdefault.jpg)
Ok, it took me a bit to understand why the std::mem::replace method works. So here is how I see it:
Normally, element is a mutable reference to self.slice[0]. This mutable reference has lifetime 'next, because it is defined within the next function that has a lifetime of 'next. We need to make it have a lifetime of 'iter instead. The only mutable reference to self.slice with a lifetime of 'iter is self.slice itself.
So when you do 'let slice = &mut self.slice;', you are creating a mutable reference to self.slice, with lifetime 'next since it is defined in the next function. Then std::mem::replace takes that mutable reference with lifetime 'next, **dereferences** it and sees a mutable reference to self.slice with lifetime 'iter, and returns it to us which is exactly what we want. It is the fact that it dereferences the 'dest' function parameter and returning it (with the lifetime associated with it) which makes it work.
-I think it would have been a bit clearer if at --1:06:23--, you would have used 'let slice2 = std::mem::replace(slice, &mut []);' because it is a bit confusing when you see that slice is being replaced with '&mut []', and it's old value is being bound to slice again.- You did this as 1:08:26, I should have watched it through before commenting!
I had to draw it out to see how it worked, but I'm glad I did because I think I get it now :D
I was thinking of calling it a day, but then this comment helped me get it finally. Thanks man.
@@ihve I'm glad it helped!
C & C++ Programmers would be like, ow this is much easier. people coming from GC language " what the heck have i been watching for an hour "
As a C++ programmer, this is really not that easier. Implementing an iterator of a non-const container in C++ is much easier, but of course also a bit more error prone as you don't get lifetime checks.
@@embeddor2230 of course it's much easier if you do it wrong and full of bugs. the point is that doing the correct thing has the same difficulty in both Rust and C++, but Rust prevents you from writing bullshit code
TIL (finally) why you have to annotate lifetimes and generic parameters twice in the impl signature!
Thank you Ryan again for the awesome tutorial!
56:40 I think it comes down to a simple fact. If the data is guaranteed to exist (immutably), and it won't be changed ever (because we enforced mutable borrow laws, so you can't read the data after it has been mutably borrowed), and because we know how long the data will last for (up to the highest lifetime), then because of all these conditions, we can say for certain that the data referenced there will ALWAYS be the same, so it is safe to lengthen it to the longest lifetime in that case (regardless that it was borrowed as 'next). I suppose that in this case, the reference from 'next effectively == 'a
For the longest time I kept thinking, "who is this know-it-all Chad fella? And why is he asking all the questions?"!! 🤦🏽♂️
Love the content.
Finally I found someone discussing Lifetimes up to advance usage. Ive been struggling to understand lifetimes.
Checkout Crust of Rust too - Jon Gjengset - he does really deep dives on topics.
I really like your instruction style, Ryan; thanks for doing this.
@35:30 I think the question was this:
If I write a function that returns an iterator, should the function signature specify the concrete iterator type that is returned, or should it use "impl Iterator"? And the answer, I believe, is that in general it is better to use "impl Iterator."
To be more precise about it,
fn foo(...) -> impl Iterator {...}
is better than
fn foo(...) -> MyIterator {...}
It is better in that it preserves flexibility in the implementation -- later you might want to return some other type. If all you promised was that you would return SOMETHING that implements Iterator, then you can make that change. If you promised that you would return a MyIterator, then changing that would be a breaking change in your API.
Your content really is the best. Thank you. I’m totally inspired.
This is by far the best video about Rust lifetime. thank you so much. Would be great if you can post more videos about Rust like async 😊
A totally underrated guy. Thank you, Ryan!
Hey, I got a question: isn't it weird that we can std::mem::replace effectively self.slice temporarily with &mut []? AFAIK shouldn't any contents of self.slice need to have a lifetime of 'iter (which is clearly not the case with &mut [], which should live on the function stack)? Is this just a special elision trick where the compiler proves that self.slice is never actually invalid or am I missing something?
`&mut []` (or empty slices in general) has a static lifetime and thus can effectively have any lifetime.
@@RyanLevicksVideos thanks mate! so it's a special case then - &mut [something] wouldn't work. Was just wondering.
I haven't written a line of rust yet (cant figure out a project to do) but it would seem that, in the first iteration of the Iterator implementation where you check if the slice.is_empty and return none, that code could panic with a index out of bounds/range "exception" when self.slice has a single item in it. (Unless slice[1..] returns differently than i would expect.)
edit
I would also like to note that I appreciate it when you repeat the questions from chat (which you have done consistently throughout this and other videos).
I've thought the same thing, so I've actually tested it and it works correctly when the collection contains one element. It seems that slice[1..] does not panic even though the slice has only one element at that moment. Then I tested the following: 1) make collection be a Vec of one element, 2) 'let second_collection = &collection[1..];' works, but 'let second_collection = &collection[2..];' does not, and 'let second_collection = &collection[1];' also does not work. This leads me to believe that when you use slice notation, then you can refer to one element after the last element, but when you refer to *just* the one element after the last (or even further), then it doesn't work. Why it works like this, I'm not entirely sure as I haven't looked into that, but it was interesting to say the least :)
@@Oguz286 Cool, thanks for your reply and for taking the time to look into this.
I also thought it would be an index out of bounds. Apparently, one index more than the last index is ok as long as you are using slice notation (otherwise it would be the traditional index out of bounds if not a slice). This finally explains why [1..] never had an error. It'll just simply return an empty slice, or if it's exactly at the end of the index, it'll return the current element
I think it's kinda useful though, since this way you can get an item or none. If it went out of index one index after (as normally it should), then all you could do it slice at the last index instead which would return only the last item, and you'd never be able to return an empty slice
If you test it out on Python, it works the same way in fact. (Although it might be worth noting you can keep going up and up in numbers without an error, unlike in Rust where you can't)
a = [1,2,3]
a[2:] // = [3]
a[3:] // = []
@@Oguz286Interesting. I’ll have to play with this now. 😊
1:07:37 Isn't it a bit better to use std::mem::take() instead? You're replacing it with a default, and take() should do just that
my exact thought
Where you said that T must live as long as struct, that literally made this make more sense than all the countless guides I read
25:00 "it's more code" \*ends up being less code\*
Some people feel shame over the realization "Why didn't I think if this? It's so obvious now that I see it!". I don't, because that feeling just means that I learned something that I know I would have needed several times before, and - more importantly - will need again in the future.
Going back to review an ugly hack I did in a tokio Decoder.. :)
This is amazing. I finally understand something of Lifetime. Your content is gold! :D
This is great content! Please keep it up. I don't spend much time on twitch but will start to try and watch you live
You can move out of self.slice directly by passing &mut self.slice into std::mem::replace(). No need for a placeholder borrow. Thanks for another great stream Ryan, do you know if there’s work being done to make forwarding of lifetimes like this easier?
BTW, your cursor is black against black - it was very hard to see when you were point out items in the code.
Thank you, Ryan. You are really good at explaining Rust. Your voice is awesome, too.
Q: Is there a differance between std::mem::replace() and core::mem::replace()? why exist in std? I'm coming at Rust from microcontroller space where there is no OS. My impression is that Rust is only recently viable here. The no GC is very attractive for microcontrollers where I want dynamic behaviour with accurate timing always.
Rust crates can be #[no_std], in which case they do not include the standard library std. The core library is always present. std re-exports some items from core.
Bro we need your new videos :(( where are you :((
Insightful, mYbe the most insightful explanation and demo of lifetimes. Thanks!
Amazingly helpful video. Thank you
Question: what is the a problem with using 'iter lifetime for the mut self in the next function?
That is, instead of using fn next(&'next mut self), why not just use fn next(&'iter mut self) to enforce the same lifetime is used for both the iterator and each item?
The next function is defined by the Iterator trait in std which has no idea about our `'iter` lifetime. Change the next function to take `&'iter` and it will complain that that's not the right signature for the next function.
I see. Thanks!
So basically it prevents the situation that you borrow a book at the library due next month and find out while reading that the pages inside have to be returned tomorrow 🤔
More like it prevents you trying to modify pages of the book that you have already handed out to the students to eat. It also prevents you to hand out the same page for multiple students to eat, which would lead to a situation of multiple students eating the same page. Ok this example has derailed.
This was a really nice and helpful explanation for lifetimes! Thanks!
Hello, I'm very interested in how &self.slice[1..] works
If we have 1 element left in the slice, then we can get it through [0], but if we use [1] then it will be out of range, then how does &self.slice[1..] work when we have 1 element?
That puzzled me as well. I guess I’ll have to code it up and step through to see what’s going on. Or maybe it’s explained elsewhere in the comments?
Great video... as usual
But there is just one thing that bothers me. It's been mentioned multiple times throughout the video by people in chat that it is impossible to implement mutable iterator without unsafe code and Ryan telling all of us to just wait and see how it is done. Then he shows us "a trick" with std::mem::replace but if you look at its implementation (doc.rust-lang.org/std/mem/fn.replace.html - click on src link) then you see that this method is just a small unsafe block, essentially meaning that we couldn't avoid using unsafe
Simply fabulous!
Does mem::replace () accepts a pointer to a pointer ?
This is blood magic (memory being the blood).
Thanks Ryan. As always, very insightful.
48:48, did you mean "we can't have multiple 'writers' at the same time?" (i heard 'readers')
multiple readers are just as a big problem as writers, as he states right before: imagine the slice becomes invalid while your readers still point to it, the multiple readers would fail or access invalid memory. If you want to support multiple readers you need to keep track of them, e.g. with smart pointers like Rc. For a single reader the compiler can keep track of it for you. At least that is my understanding.
@@MichaelKefeder reaaaalllyy? I wouldn't have thought so, because one reader would still be a problem for the writer, as they would require synchronization
Unless you meant the reader as in the writer, because writers can both read and write
@@Dorumin I was answering for the assumption that multiple readers are no problem, when they are. I guess that's why he said it multiple times, without noticing because it is true as well. Sure in the video he is talking about handing out a mutable reference, which can only be handed out once since this gives read/write access, where saying "multiple writers" would be more clear.
Super nice! Very well explained! Just subscribed to the twitch channel.
Just like in "The Raiders of the Lost Arc". Fingers crossed the borrow checker isn't sending a giant rolling boulder your way.
Couldn't this dummy replace trick be added to the standard library as a means to grab the original pointer?
what linting extension is being used for your vscode? That's pretty neat.
Awesome, Thank you!
How is it that split_first is not named car_cdr ... I don't get it.
Thanks Ryan!
Thank you for this!
Sooo... MyMutableIterator::next needs to last as long as MyMutableIterator last, and it is understandable, but why then the `next` function has to have its own " 'next" lifetime? Would not it work if the `next` function would have a " 'iter" lifetime? Like this:
fn next(& 'iter mut self) -> Option {
...
}
and then the function body would look the same as the immutable one.
If you try that you'll see that Rust complains that the type sugnature of `next` no longer matches how `next` is declared in the Iterator trait. So you're correct in your thinking, but next does not generally borrow `self` for `'iter` it borrows it for the lifetime that we've called `'next`.
Hi and thanks for this video. I learn rust right now and of course borrowing and life time are really a little montain to grow.
I will see over videos from you channel and one of my question actually is, ilke a lot on persons i guess, i comes from object programming and with rust, it seems not to be a good paradigm to code efficiently, because of the temptation to referencing multiples objects each overs for example.
Is exists some good tutorial or book who talk about good rust paradigms and how to organize and write good rust program ?
Thanks and have a good day.
Thanks Ryan! Very helpful.
Would be very cool if you'd put a list of your vs code extensions in some link in the description
you helped me a lot, thanks
I'll be your friend. U r awesome.
So... the mutable borrow of the empty slice... does it have a lifetime of `static?
I believe mutable borrow is just a pointer. Not even the slice itself is 'static unless it's anotated as such.
Thank you so much for this one!
why do you only get the conflicting requirements error when you do it with mutable references? Why doesn't it complain when you do it with immutable references?
If you have a mutable reference to self, you can either borrow mutably for as long as you have the reference or borrow immutably for as long as the self lives. If you return a longer immutable borrow than the mutable borrow you were given, you wont' be able to mutably borrow (i.e. call next) until the returned element has been destroyed.
In the immutable case,
as you have a mutable rerefence to Self in next, you know that there are no other mutable references to slice. As there are no mutable references to slice, you can take a borrow from slice for as long as the slice lives. Or in other words, because you have a mutable reference, immutable reference lifetimes can be extended.
awesome!! love your rust content
I had the exact same issue this weekend and I found the book lacking a bit in this context. What would be the in your opinion another way to get the same result without using the mem::replace trick? Can you extend the lifetime of the returned value to the struct lifetime?
Thanks for the content anyway! ;)
I'm not aware of another way to achieve the same effect in safe code.
Would wrapping the mut ref in Option help? Then you can use take() to do the mem replace
The trick is in mem::replace - it makes the iterator itself an owner of this slice pointer. The split_first then is able to return two pointers that aren't connected to any lifetime (and you can do that, because if you are the only one pointing to a memory chunk, you can use split_first to make two mutable pointers to the same chunk as long as they don't overlap). Since the function is now the owner, it can do with these two owned mutable pointers whatever necessary - return first one from function and assign the second one back to the mutable iterator state.
Thanks for that awesome video. That was way better than reading any book. I think using `slice: Option
Hard to say if it's easier but if it works it's certainly not incorrect.
std::mem::replace is hard to understand
very nice !
ok :]
i used rust last time but i find it overly complex..its not much different from c++ except no more header files..other than that, its not much different from c++..its anoyying to use..is it really worth it? making linked list in rust alone make me wanna break my keyboard
Rust is certainly not the easiest language to learn or use. If it doesn't fit any of your use cases, there's no harm in not using it. Linked lists are something you almost never need in Rust and implementing one in safe code requires an intermediate knowledge of the borrower checker so it's not really beginner friendly. I made a video on this: th-cam.com/video/IiDHTIsmUi4/w-d-xo.html