I would love to watch some in-depth analysis on what currently is not feasible for a borrow-checker to accept but that is memory safe. A breakdown of various types of situations, which are likely to be fixable sooner, which later, and which perhaps never.
Because I'll keep coming back to this, here are some timestamps (please pin this!): 0:00 - Intro 1:31 - The Problem 5:34 - Cell (Cloning Available) 8:00 - RefCell (No Cloning) 10:37 - RwLock (Multithreading R/W Locks) 14:34 - Mutex (Multithreading Single Lock) 15:46 - Outro
I encountered some these before but had trouble following the differences between them and I didn't get why I would use one over another, now that I've heard them all explained at once it makes much more sense. Thanks!
7:10 I would usually write for &adj in &node.adjacent or whatever variation makes the borrow checker and type checker happy (the &adj is a pattern match, so you can use just adj both in the for declaration and the function call in the line below if that suits your needs, and I assume clippy will tell you as much). That's what I did everywhere in my Advent of Code. That's also what your error message actually recommends.
@@JosephDalrymple Sure. I can never really remember which is which of iter() and into_iter() and which one for loops call implicitly, so another repetition will not hurt.
@@MasterHigure I usually use the knowledge that most 'into' functions are consuming to remember which way round they are; for loops you just kinda have to remember that it's into_iter though
Very helpful video. I've been trying to port a simple simulation "game" with rather complicated data structures to Rust. I generally use this as a test to learn features of a new language. I originally wrote it on my Commodore 64 in BASIC (yes, I'm old, lol), then ported it to 6502 assembly, later C, C++, Java, and Python. Now I'm trying to port it to learn Rust and the borrow checker is giving me hell. I'm on my 6th refactor and I still don't have it working... I think the info from this video may help, though!
And I'll say that using Java for the last several years has nearly ruined me for trying something like Rust. Java encourages the use of many aliased references and cleans up the mess for you. I think someone who has only ever done Java and never, say, C or C++ is going to have a hard time grok-ing Rust.
nice, glad you found it valuable! That's cool that you have the same game that you use as a litmus test for new languages that you learn - what a great way of teasing out the things you like and don't like when you are learning a new language
I enjoyed this video. I had some trouble understanding Cell and Refcell, and this video is beneficial to clarify these concepts. I think the example you have chosen is excellent because simple enough to focus on the concepts and not be bothered understanding a complex code. The explanations are also really clear, so thanks for sharing this content. That is an excellent job.
Nice! I’d also be interested in a crust-of-rust-but-faster style overview of how Cell/RefCell work internally, if there’s any insights to be gained there.
I can definitely understand the desire for something like Crust of Rust but that is more concise. This video was actually heavily inspired by Jon Gjengset's coverage of the same smart pointers, in a way that I am hoping was complimentary. He covers the implementation details, so I decided to cover practical usage examples. I'm afraid I don't quite have the expertise to do a video on the implementation details at the moment, but in the meantime I'd highly recommend Jon's video on the topic if you haven't already watched it!
Thanks so much Mark! If you haven't already, check out the "Rust Demystified" video - the second part is about lifetimes. Though I'm sure there are more, it covers the main concepts and use cases.
thanks! yes the example given in this video would probably be easier to write in pretty much all other mainstream languages. it can be seen as an example of the extra burden placed on the developer in exchange for Rust's performance and safety
I realise that you're presenting a general case. I've used the Atomic classes for simple types (bool, ints, floats) which are no bigger than my architecture, as they have ideal performance when you call them with relaxed timing. When you enable any optimisation, it optimises down to the bare minimal instructions for assigning the data type.
Very helpful video! I think this will come in handy with converting some C++ code to Rust, as it might be safe but just harder to express with the borrow checker.
> There's no potential for dangling references... That's mooostly true for sure, but you could do it if you really wanted to. Suppose you added some elements to node A's adjacent Vec, and then you went through node B to grab a reference to one of those elements. That would be a &&Node double reference, which is kind of weird, but whatever suppose we grabbed it. Then, if Rust allowed us to get away with mutation here, say by going through node C somehow, we could call .clear() on that Vec, and that would make the &&Node double reference dangling! (Maybe the more realistic way to get into a situation like this would be to *iterate* over the Vec and then somehow try to clear it in the middle of your loop.)
I'd say in the final scenario, RwLock would have been better though. Because when multiple threads are reading the node structure and nothing is writing, a Mutex would block other threads from reading, while an RwLock does allow multiple readers as long as nothing is holding a write lock.
One thing is that in rust you should first try to change your API, so that you don't need interior mutability, if that doesn't work only then should you consider using it.
Using the ' character for lifetime annotations is horrid and always unsettling to look at and parse with my eyes, and annoying to implement for editors/web views with lightweight syntax highlighting. Even writing it here in a post I'm worried about clarity and if it's hard to read. Rust should have used another keyword or @ or something, why is that operator so off-limits... It's not just an aesthetic problem, using ' actually causes language problems. For example it has ambiguity with trying to use const generic parameters of type char like if you wanted to use a type Foo without some special-case scenario. Didn't Rust learn from C++ about doing crazy stuff with parsing special corner case syntaxes? It's bad like the problems C++ inherited.
@@spark_coder a tldr, RC is reference counting and ARC is *atomic* reference counting. You need atomic instructions in order to use a reference counter across threads, otherwise you would have data races.
If you're looking for topics, I think a nice related topic (regarding graphs) is IndraDB would be a great one. I never really had a need for a graph database yet, but I'm sure I will at some point, and would be cool to know an open source rust graph db.
I can understand why people love this language but if I don't need the performance I prefer living with a GC. The hoops this throws are just crazy just to implement a simple graph structure..
Your comment reminds me that as much as I detest C...I lived fine without using it for decades before I encountered Rust, haha. I guess mostly I only use Rust for embedded programming atm, anyway, so I guess I'm not entirely under Rust's spell, but still. OTOH, I hear Rust does concurrency safely, too, hmmm....
I like using cell for non-copy types too. Doing `let mut contained = cell.replace(String::new());` allows you to take the string out of the cell, and it can be put back with `cell.replace(contained);`
I don't have an ultra strong preference for macOS over Linux for development - just don't make me use Windows 😎 right now the biggest thing keeping me on macOS is Apple's Messages app for texting from my computer. Honestly if that wasn't a factor I'd probably be using Linux
I've written a C++ program that manipulates a TIN (a plane graph in which every region is a triangle) using three classes in a tangled hierarchy, at least two levels of locking, and readers-writer locks, to refine the TIN until it fits the point cloud. It took me about a year to get the multithreading working right. Every new triangle has to be created locked, and there's just one lock that controls adding new triangles, which will be a bottleneck when I run it on a 144-thread machine. (I have an idea of how to fix that.) How could this be done in Rust?
It's a bit hard for me to answer without a better understanding of the low level details of your project, so let me flip it back to you - what aspects of Rust do you think would make implementing such a thing challenging?
I'm writing a gameboy emulator right now, and the emulator is split up between the different chips in the gameboy, e.g. the cpu, the picture processing unit (ppu) and the audio processing unit (apu). My problem now is, that these chips communicate to each other by writing into a shared memory space, which effectifly is shared mutable state in my emulator. I had some trouble finding elegant solutions. Your video helped me a lot, thanks!
I think it's worth pointing out that interior mutability should be seen as kind of a last resort. I say kind of because obviously there are plenty of situations where it's the right choice - but frequently it's used as a crutch to use Rust just like you would a garbage collected language, when just a slight change in how we think about memory or control flow would lead to better results. That's probably the hardest part about really learning Rust, but the cool thing is that once you get the hang of it, it's easily applied to other languages as well and tends to lead to better project structure.
You reffer to Cell, Refcell, Mutex and RwLock as smart pointers, but they are not pointers. The value stored is stored as part of the Cell/Refcell/Mutex/Rwlock not in a seperate location on the heap. Also while you can't use get on a cell containing a non-copy type, if your type implements Default (as for example String does) you can use take. So you can write code like. let tmp = cell.take(); // do something with tmp. cell.put(tmp);
I think you'd definitely want to avoid using interior mutability when it isn't strictly necessary, and it does feel a bit like fighting the borrow checker. That said, the benefits of the borrow checker are huge and the cases where you need to work around it are likely to be few and far between.
When I started programming in Rust, I had to use interior mutability in some places, but after a while I just do things differently so I don't need this anymore.
@@carlosmspk I don't really remember why I needed it to begin with. I just don't get into situations where I would need interior mutability anymore after I learnt how to use Rust properly. For graph like structures I just store indices of arrays or maps. Often it also helps to split a struct into multiple smaller structs, and pass them around.
@@porky1118That...sorta sounds like it sucks? I'm generally of the opinion that your data should ideally be stored in the form most natural for it. If your data is a foo, it should be a foo. Specifically, if your data is a graph, it should ideally be stored as a graph, rather than bending it into some other shape. Sure, you can make it work, in other ways - but it feels to me like it would be clunky, artificial, unnatural. There might be reasons to use other forms, such as performance (indexing etc.), but I don't think I'd want to default to them. It's been while since I had to deal with graph types of any significant complexity, though, so it's possible I'd reconsider if actually tasked with doing so today.
TBH RefCell is a ductape of the Rust. Construction with whole purpose to turn of main feature of rust and which will crash your application even if there is no real reason for that. One of the worst parts of the Rust. The two another quite equaly bad parts is a macros and weird borrow checker rule which require mutable reference to actually go out of scope instead of track the last use in that scope.
yeah i agree it is kind of like turning off the main feature of Rust, which is always preferable to avoid. It comes back to the intractable problem of accepting all memory safe programs while rejecting all unsafe programs. This intractability leads to these sorts of things being necessary. Also If you still want the safety, you can avoid the potential panics by using 'try_borrow' instead
I can definitely relate to this! It feels so wrong! But unfortunately that appears to be the reality of compile time memory safety checks - there are some types of programs that, despite being memory safe, need to go around those compile time safety checks 🙁
@@codetothemoon Hm... I would rather try to pass `add_one` a `&mut Node` and still use a `RefCell` for single threading or `Arc` for threaded stuff... (?) Or rather not build the data structure of the graph like that, but instead with indexes and a global common list/hash-map... There is always a way I think one just needs to make the program work differently. A data structure linked by refs (pointers) is easy but anyway gets you in trouble also in C++ where you need to make sure these pointers point always to valid memory and do not get dangling because of reallocation of some underneath data structure which holds the nodes...
So I tried to just jump in and figure these things out as I go along (with copilot by my side), rather than to stop and study it. How hard can it be? I've used C/C++ for years, I know how to code. I'll take it as it comes Turns out that's a really good way to give yourself an aneurysm
0:39 IDK, that sounds like something you can reduce to the halting problem. Like imagine you had a program that segfaults if it discovers itself is memory safe and does nothing unsafe otherwise. That would be a contradiction.
Tried to write a trivial LinkedList in Rust and ran head first into problems when trying to modify the list, and writing methods to modify the list. Felt like I was kinda close but never quite got there….this is definitely what I missed.
In the end, the Node struct ends up with various smart pointers on different fields. I understand we got there by construction, but I wonder if there’s a way to refactor it so that the smart pointers are just on Nodes, rather than the internal fields? Could the add_urgency() function somehow lock an entire node rather than the internal field? What would that look like?
How would you go about implementing a doubly-linked list in safe Rust? I tried doing this once, but I got stuck when implementing a mutable iterator for it
At around 5:00, why did you take the parameter as `node: &Node` if you want to mutate its value ? It's an immutable reference which will not allow you in general, irrespective of tree structure or the thing (Cell) you want to explain. You should have taken `node: &mut Node` and try mutating it.
The premise of the video is what to do in situations where you'd like to take a mutable reference (or ownership), but you can't because there already exist multiple borrows of that piece of data. That's where interior mutability comes in, which is basically foregoing compile time borrow checking such that you're able to mutate things that have been borrowed "immutably".
@@codetothemoon Yes. I understood that. Thanks for saying that again. It's a great video. However, the point I was making is.. you had an opportunity to make a "full circle" argument. That is, when we beginners see a function that takes `node: &Node` as parameter, it seems immediately obvious to us that `node` cannot be modified .. and that is true, irrespective of the shape of the data-structure (that is actually wrong, as we see it later in this video itself, but that is what the beginners initially think). So logically, in order to mutate node, we have to change the parameter type to `node: &mut Node`, then we take 5 seconds and make the full circle argument that: while `&mut` does allow the function to mutate the node, we now cannot make two different nodes `b` and `c` refer to the same node `a`, as that has to be `&mut` type as well, which Rust doesn't allow. Hence, the need of `Cell` is clearly understood. We beginners tend to forget little concepts while we build our knowledge base in our mind, so such a "full circle" argument helps a lot.
When i am at this point, i know its not the right thing to do, especially in Rust, but i just use *const T or *mut T and live with the fact, that i am in an unsafe world at this context. Using the functions from outside are still safe to use, when coded propertly
Would the final lock-based solution work with any generic type for the node payload? Let's say Node was a generic struct. Btw, thanks a lot for the great content!
If you're writing an application that will be deployed on Linux, it's recommended to use Mutex in general, even if you have reads in there. You should of course benchmark and try RwLock vs Mutex in your specific application, but the Mutex implementation in the Linux kernel is just so much faster than the RwLock implementation. Also, Linux' RwLock prioritizes readers over writers, so by having a constant amount of readers, you can starve your writers and they'll never get access. On Mac, RwLock is superior to Mutex. Idk about Windows for sure, but I think Mutex was also better than RwLock. You could make a type that wraps the two of them and picks a solution depending on your operating system. There's probably already a crate for that.
One of the greatest benefits of Rust is that it makes you uncomfortable using worse patterns. Sure, you can implement it as a traditional graph, but you could also implement it as an entity component system with a vector of nodes and a vector of relationships between them. That makes it: 1. Memory safe 2. Cache-friendly 3. Without the overhead of reference counting I'm not saying people should not use RefCell, but people should ask themselves: Do I need it? Or do I want it because that is how it's done in C++/java/...? And is there a better pattern?
How would you implement an O(n) graph traversal when relationships between nodes are stored in a vector? I also don't see how this would negate the need for interior mutability, since each node would potentially have multiple shared references to it in said vector
@@codetothemoon the relationship vector would contain indexes of the children, not references. You just need to know the index of the node. Using that index in the node vector gives you the node data, and using it in the relationship vector gives you data containing the indexes of children. This gives you O(n) traversal, i believe.
Would you find that valuable? If so, what editors have you tried and which is your go-to at the moment? Trying to get a sense of what folks are interested in and where they are coming from.
@@codetothemoon I've just kind of given up on emacs and went back to nvim, but it seems like a couple of people asked about the editor in the comments already and I always find those vids interesting
@@codetothemoon There aren't a lot of resources for Rust, and you're one of the channels I love so much and am grateful to. project: Any rust project that connects to data and connects to any frontend frameworks.
see stackoverflow.com/questions/50704279/when-or-why-should-i-use-a-mutex-over-an-rwlock also in the scenario described by the video I think mutex is the better option because there isn't a need for a read lock at all
0:40 Static analysis that accepts all memory-safe programs and rejects all non-memory-safe programs is impossible as a consequence of the halting theorem, duh.
One of your assumptions for the use of borrow_mut is incorrect: you assume that there would never be two borrow muts in your add one function, but the cycle case would create a borrow_mut for each loop of the cycle.
Good point - I should have mentioned this much earlier in the video, but at 12:07 I explain that this program doesn't handle cycles. As you probably know we'd have to somehow keep track of the nodes we've already visited to avoid infinite loops (or panics, as would happen in case you reference)
i have watcched so many videos about RefCell and Rc and what all tutorials fail on is that no one shows a real world problem applying Rc or RefCell. Every one builds these simple basic graphs that is not applicable on a real life problem. My suggestion to you is that in your tutorial you set out to solve a specific problem instead, that can only be solved by using interior mutability, instead of building this fictional simple graph. Just some feedback.
@@adibhanna Emacs has very different controls (by default), is less focused on pure text editing and comes with basically every feature imaginable, even if it doesn't make any sense. Doom Emacs is a popular alternative configuration. I personally hate Emacs' approach to many things but it's still one of the best editors out there.
@@adibhanna IMO you can't go wrong with either - both are fantastic. The actual coding experience pretty much identical in both (albeit key bindings may be different), most of the differences exist outside of that. Emacs is famous for something called org mode, which is sort of like markdown but much more powerful. For example, you can include code blocks of any language in your org documents, and you can run them inside the doc similar to how you would in something like a Jupyter notebook. Much of what you can do in emacs can probably be replicated in nvim plugins, but it seems like the one thing emacs will always have over *vim is that it can be run as a terminal OR GUI application - when run as the latter, you get capabilities like having multiple fonts at the same time. Something that can really make things more aesthetically pleasing and that I don't believe is possible in a pure terminal application.
which complexity are you referring to specifically? In the video I was hoping to show how interior mutability is simple :) I think the borrow checker definitely introduces some complexity, but complexity that is well worth the memory safety it yields in return!
@@codetothemoon I've never had an issue with memory safety, its not rocket science to keep track of allocations, especially with smart pointers. This just looks like an unnecessarily complex way of doing something really simple like accessing a vector in a struct.
@@theLowestPointInMyLife If you learn to understand complex then you would easily do it the next time without falling into traps. I think to the Author this is not complex because he is experienced in Rust. Don't you want to become like him?
@@theLowestPointInMyLife "I've never had an issue with memory safety" - good for you. Unfortunately, most programmer aren't as brilliant as you are. Microsoft said that 70% of their security bugs are memory safety issues
How would it be able to detect a cycle at compile time? What would the error be? And how would we be able to get around that (since cycles are required for some applications) ?
@@codetothemoon Depends how you try to write it. The fact that graph edges are references to already constructed vertices requires that all edges go "upwards" (in source code). This guarantees there are no loops.
Well, I guess you could make a node mutable and add edges after construction. But we can do that only if it has no incoming edges (since those are immutable borrows).
@@codetothemoon Go around loops: like you did with Arcs. Or you could use an arena: keep all nodes in a single slice and refer to other nodes by their index instead of an actual reference
but they are slower than if there's no unsafe code at all... there's performance hit even for the simplest cases just because we can't express things clear enough... or I'm wrong..
Why fight the borrow checker instead of learning how to adopt rust's intended paradigm? Specific program instances that are memory safe are not necessarily memory safe in general. These exceptions are among the situations that rust avoids by design. The little "hacks" or circumstantially memory safe bits of code that appear early in the development cycle are the seeds of infinite hell down the road when system complexity increases. What could have been "clearly memory safe" code snippets at an earlier stage can grow into messy reference violations down the road -- *exactly* the kinds of headaches from C/C++ development that played part in motivating rust's new paradigm. So I ask: if the developer is going to outsmart the borrow checker whenever it's inconvenient, why not just code in C++?
I can definitely empathize with this - it does feel a bit more verbose than it needs to be. I'm hopefully that at some point explicit lifetimes won't be necessary for this scenario
I belive it is impossible to have a program that can 100% of the time guarantee that a program is memory safe or not because of the halting problem. Say you have a function S that takes a program as input and returns true if its memory safe and false if its not Then a function L that is leaks memory and is unsafe, so any other function that calls L will also leak and be unsafe. Now, construct a function C that takes some source code as input, passes that source code into S, then if S returns true then it simply ends, if S returns false then it calls L and ends. Now, pass C's source code into C: 1. C is called with C as input 2. S is called with C as input 3A. If S returns true (C is safe) 4A. C calls L 5A. L leaks memory 6A. L is unsafe 7A. C is unsafe -- Contradiction! By definition, S NEVER fails and here it predicted that C is safe 3B. If S returns false (C is unsafe) 4B. C ends 6B. C is safe -- Contradiction! By definition, S NEVER fails and here it predicted that C is unsafe S can only return true and false but both of those values are wrong meaning that it is impossible to have a function that correctly predicts if a given source code is safe or not in all cases.
Great explanation, but as a teacher myself, I would caution that if you really want your listeners to absorb what you're communicating, then you must significantly slow down your rate of speech. There are at least two reasons for this: 1) The comprehension of new concepts takes time. Therefore, although you're delivering great content, in order for your lesson to be really effective, that content cannot be delivered at conversation speed (or here, almost machine gun speed). You must consciously slow down your rate of delivery in order to give your listeners the time to absorb and comprehend these new facts and concepts 2) If a native English speaker finds it hard to keep up, how much more challenging will it be for someone whose first language is not English? Always keep of the non-native English in mind Great video otherwise!
thanks for the feedback, it is super helpful! i do indeed struggle with pacing, and the common feedback in this era was that things were too fast. I've corrected a bit but I could probably still do better!
You're welcome. What helped me the most was teaching non-native English speakers. These people are perfectly intelligent, but in relation to the subject matter I was talking about, they were simply uninformed. Therefore, in order to reduce the effect of the language barrier, I had to deliberately slow my speech down and rephrase what I wanted to say in the simplest possible terms without the use of slang or idiom. This not only helped the students comprehend the lesson, but also helped me distill my knowledge down to the core concepts. It really is a win-win situation for everyone! 🙂
New in Rust 1.69: the Vibe Checker
nice
Nice.
Nice.
Nice.
somebody might have to walk me through this one... 🤓
I would love to watch some in-depth analysis on what currently is not feasible for a borrow-checker to accept but that is memory safe. A breakdown of various types of situations, which are likely to be fixable sooner, which later, and which perhaps never.
great idea!
Because I'll keep coming back to this, here are some timestamps (please pin this!):
0:00 - Intro
1:31 - The Problem
5:34 - Cell (Cloning Available)
8:00 - RefCell (No Cloning)
10:37 - RwLock (Multithreading R/W Locks)
14:34 - Mutex (Multithreading Single Lock)
15:46 - Outro
I encountered some these before but had trouble following the differences between them and I didn't get why I would use one over another, now that I've heard them all explained at once it makes much more sense. Thanks!
nice, glad you found this approach valuable!
ngl this video is a much better introduction to interior mutability than the rust book
This is what I needed to try and implement the Monte Carlo tree search in Rust, thank you so much for the clear explanation
nice, glad you found it valuable!
7:10 I would usually write
for &adj in &node.adjacent
or whatever variation makes the borrow checker and type checker happy (the &adj is a pattern match, so you can use just adj both in the for declaration and the function call in the line below if that suits your needs, and I assume clippy will tell you as much).
That's what I did everywhere in my Advent of Code. That's also what your error message actually recommends.
That's what I initially thought was going to be done, but it's pretty interesting to see the different approaches.
@@JosephDalrymple Sure. I can never really remember which is which of iter() and into_iter() and which one for loops call implicitly, so another repetition will not hurt.
@@MasterHigure I usually use the knowledge that most 'into' functions are consuming to remember which way round they are; for loops you just kinda have to remember that it's into_iter though
Very helpful video. I've been trying to port a simple simulation "game" with rather complicated data structures to Rust. I generally use this as a test to learn features of a new language. I originally wrote it on my Commodore 64 in BASIC (yes, I'm old, lol), then ported it to 6502 assembly, later C, C++, Java, and Python. Now I'm trying to port it to learn Rust and the borrow checker is giving me hell. I'm on my 6th refactor and I still don't have it working... I think the info from this video may help, though!
And I'll say that using Java for the last several years has nearly ruined me for trying something like Rust. Java encourages the use of many aliased references and cleans up the mess for you. I think someone who has only ever done Java and never, say, C or C++ is going to have a hard time grok-ing Rust.
nice, glad you found it valuable! That's cool that you have the same game that you use as a litmus test for new languages that you learn - what a great way of teasing out the things you like and don't like when you are learning a new language
yeah I agree that having a background solely in garbage collected languages will definitely make Rust tougher to pick up...
I enjoyed this video. I had some trouble understanding Cell and Refcell, and this video is beneficial to clarify these concepts. I think the example you have chosen is excellent because simple enough to focus on the concepts and not be bothered understanding a complex code. The explanations are also really clear, so thanks for sharing this content. That is an excellent job.
Than you so much, I'm glad you found it valuable! I found these concepts confusing as well, and I tried to make the explanation I wish I had had.
Nice! I’d also be interested in a crust-of-rust-but-faster style overview of how Cell/RefCell work internally, if there’s any insights to be gained there.
Can recommend "Rust Atomics and Locks" by Mara Bos, which covers this exact topic. Available for free on her website.
I can definitely understand the desire for something like Crust of Rust but that is more concise. This video was actually heavily inspired by Jon Gjengset's coverage of the same smart pointers, in a way that I am hoping was complimentary. He covers the implementation details, so I decided to cover practical usage examples. I'm afraid I don't quite have the expertise to do a video on the implementation details at the moment, but in the meantime I'd highly recommend Jon's video on the topic if you haven't already watched it!
nice, going to put this on my list of things to check out!
The Crust of Rust, but Faster than Lime? LOL
@@codetothemoon i actually wanted to see practical usecases of these smart pointers after jons video and you posted exactly what we all wanted
Again, a really great video! Can I request a video about lifetimes, specifically knowing what to add when rust tells you it needs them.
Thanks so much Mark! If you haven't already, check out the "Rust Demystified" video - the second part is about lifetimes. Though I'm sure there are more, it covers the main concepts and use cases.
@@codetothemoon Thanks, yeah...that helped a little. How about a video on bits and bytes? Using the Bytes crate, [u8] etc...bit shifting...
Clearest explanation I've seen so far. Would have taken 2 minutes to write in c++
thanks! yes the example given in this video would probably be easier to write in pretty much all other mainstream languages. it can be seen as an example of the extra burden placed on the developer in exchange for Rust's performance and safety
I realise that you're presenting a general case.
I've used the Atomic classes for simple types (bool, ints, floats) which are no bigger than my architecture, as they have ideal performance when you call them with relaxed timing.
When you enable any optimisation, it optimises down to the bare minimal instructions for assigning the data type.
Very helpful video! I think this will come in handy with converting some C++ code to Rust, as it might be safe but just harder to express with the borrow checker.
nice, glad you found it valuable!
This amazing explanation deserves my subscription. Great job 👍
Thank you, very happy to have you onboard!
You are so good at explaining these! Your rust content is the best out there!
thank you for the kind words, really happy it's been valuable to you! 😎
Wow! This is so informative and easy to follow, although it's a complex topic for beginners like me! Great job
thank you, really glad you got something out of it!
By far the best explanation I've found, and the only one I've understood!
> There's no potential for dangling references...
That's mooostly true for sure, but you could do it if you really wanted to. Suppose you added some elements to node A's adjacent Vec, and then you went through node B to grab a reference to one of those elements. That would be a &&Node double reference, which is kind of weird, but whatever suppose we grabbed it. Then, if Rust allowed us to get away with mutation here, say by going through node C somehow, we could call .clear() on that Vec, and that would make the &&Node double reference dangling! (Maybe the more realistic way to get into a situation like this would be to *iterate* over the Vec and then somehow try to clear it in the middle of your loop.)
Going to try to implement some of this info into a doubly linked list , thanks!
nice, that is also a great scenario where interior mutability can be leveraged!
Very good and concise intro into that important Rust topic, thanks!
thanks, really glad you got something out of it!
Short and sweet. Thank you!
thanks for watching, really happy you got something out of it!
I'd say in the final scenario, RwLock would have been better though. Because when multiple threads are reading the node structure and nothing is writing, a Mutex would block other threads from reading, while an RwLock does allow multiple readers as long as nothing is holding a write lock.
Thanks a lot. Really appreciate your high quality videos
thanks for watching, glad you got something out of it!
How and why Arc is a costless abstraction?
Great video thanks. Haven't used RwLock much before so good to know about it
Thanks really, explained a lot of concepts I was having trouble with!
Great, really happy it was helpful!
This was so helpful!
nice, really happy you got something out of it!
Love the thumbnail! Thanks for the video.
thanks and thanks for watching!
Very very nicely done. Kudos.
thank you, glad you liked it!
One thing is that in rust you should first try to change your API, so that you don't need interior mutability, if that doesn't work only then should you consider using it.
agree 💯!
@sutsuj6437 Can you give an example of what this would look like in this case?
Using the ' character for lifetime annotations is horrid and always unsettling to look at and parse with my eyes, and annoying to implement for editors/web views with lightweight syntax highlighting. Even writing it here in a post I'm worried about clarity and if it's hard to read. Rust should have used another keyword or @ or something, why is that operator so off-limits... It's not just an aesthetic problem, using ' actually causes language problems. For example it has ambiguity with trying to use const generic parameters of type char like if you wanted to use a type Foo without some special-case scenario. Didn't Rust learn from C++ about doing crazy stuff with parsing special corner case syntaxes? It's bad like the problems C++ inherited.
Can you also explain the difference between Rc and Arc?
Check out my other video, “Rust’s alien data types”, for that explanation 😎
@@codetothemoon 😁 Good plug... But also thank you :)
@@spark_coder a tldr, RC is reference counting and ARC is *atomic* reference counting. You need atomic instructions in order to use a reference counter across threads, otherwise you would have data races.
@@Chex_Mex oh so kind of like std::atomic in cpp... Cool :)
That is the first time I’ve seen just a scope be useful. No if statement no loop, just a scope.
yeah they are good for ensuring deallocation once you no longer need something
Wow, very neat explanation! 👍
thanks, glad you found it valuable!
If you're looking for topics, I think a nice related topic (regarding graphs) is IndraDB would be a great one. I never really had a need for a graph database yet, but I'm sure I will at some point, and would be cool to know an open source rust graph db.
Thanks Jeff, I've put it on my list of stuff to check out!
Great video as always, I was wondering since I'm rewriting my neovim config from zero, what color scheme is this?
I'm using doom emacs for this video with the doom-monokai-pro theme - but i'm sure there is a Neovim equivalent.
Great video! Which IDE are you using?
DOOM emacs, which has been my go-to editor for the past few months github.com/doomemacs/doomemacs
very helpful video! thanks
Glad it was helpful!
Great content, and I like your "no bullshit" style. Subscribed!
thanks, very happy to have you onboard!
I can understand why people love this language but if I don't need the performance I prefer living with a GC. The hoops this throws are just crazy just to implement a simple graph structure..
totally understand the sentiment that "its not worth it" in many cases. I think there will always be a place for GC 😎
Your comment reminds me that as much as I detest C...I lived fine without using it for decades before I encountered Rust, haha. I guess mostly I only use Rust for embedded programming atm, anyway, so I guess I'm not entirely under Rust's spell, but still. OTOH, I hear Rust does concurrency safely, too, hmmm....
that was a great video mate
thanks, really glad you found it valuable!
I like using cell for non-copy types too.
Doing `let mut contained = cell.replace(String::new());` allows you to take the string out of the cell, and it can be put back with `cell.replace(contained);`
Great video. Thanks
thanks glad you got something out of it!
What makes you want to use MacOS instead of something like Linux?
I don't have an ultra strong preference for macOS over Linux for development - just don't make me use Windows 😎 right now the biggest thing keeping me on macOS is Apple's Messages app for texting from my computer. Honestly if that wasn't a factor I'd probably be using Linux
@@codetothemoon I am a FOSS activist, but I have to respect that. Well, at least we both can agree that Windows is a terrible dev environment!
I've written a C++ program that manipulates a TIN (a plane graph in which every region is a triangle) using three classes in a tangled hierarchy, at least two levels of locking, and readers-writer locks, to refine the TIN until it fits the point cloud. It took me about a year to get the multithreading working right. Every new triangle has to be created locked, and there's just one lock that controls adding new triangles, which will be a bottleneck when I run it on a 144-thread machine. (I have an idea of how to fix that.) How could this be done in Rust?
It's a bit hard for me to answer without a better understanding of the low level details of your project, so let me flip it back to you - what aspects of Rust do you think would make implementing such a thing challenging?
This vid is gold 🥇
Thanks for the kind words!
I'm writing a gameboy emulator right now, and the emulator is split up between the different chips in the gameboy, e.g. the cpu, the picture processing unit (ppu) and the audio processing unit (apu). My problem now is, that these chips communicate to each other by writing into a shared memory space, which effectifly is shared mutable state in my emulator. I had some trouble finding elegant solutions. Your video helped me a lot, thanks!
I think it's worth pointing out that interior mutability should be seen as kind of a last resort. I say kind of because obviously there are plenty of situations where it's the right choice - but frequently it's used as a crutch to use Rust just like you would a garbage collected language, when just a slight change in how we think about memory or control flow would lead to better results. That's probably the hardest part about really learning Rust, but the cool thing is that once you get the hang of it, it's easily applied to other languages as well and tends to lead to better project structure.
Agree 💯
This was awesome! Thanks
glad you liked it!
3:24 I'm confused by ` 4 j`. Normally, `` is move up half a screen, but it's not doing that. Have you remapped it?
You reffer to Cell, Refcell, Mutex and RwLock as smart pointers, but they are not pointers. The value stored is stored as part of the Cell/Refcell/Mutex/Rwlock not in a seperate location on the heap.
Also while you can't use get on a cell containing a non-copy type, if your type implements Default (as for example String does) you can use take. So you can write code like.
let tmp = cell.take();
// do something with tmp.
cell.put(tmp);
I'm wondering though, isn't a heavy usage of these kind of a red flag? Feels like fighting the borrow checker
I think you'd definitely want to avoid using interior mutability when it isn't strictly necessary, and it does feel a bit like fighting the borrow checker. That said, the benefits of the borrow checker are huge and the cases where you need to work around it are likely to be few and far between.
@@codetothemoon Cool, thanks for the reply!
Nice explanation, thanks!
glad you found it valuable, thanks for watching!
When I started programming in Rust, I had to use interior mutability in some places, but after a while I just do things differently so I don't need this anymore.
How do you get around it?
@@carlosmspk I don't really remember why I needed it to begin with. I just don't get into situations where I would need interior mutability anymore after I learnt how to use Rust properly.
For graph like structures I just store indices of arrays or maps.
Often it also helps to split a struct into multiple smaller structs, and pass them around.
do you think there is a way around it in the scenario described in the video?
@@codetothemoon This scenario is about graphs, and "For graph like structures I just store indices of arrays or maps."
@@porky1118That...sorta sounds like it sucks? I'm generally of the opinion that your data should ideally be stored in the form most natural for it. If your data is a foo, it should be a foo. Specifically, if your data is a graph, it should ideally be stored as a graph, rather than bending it into some other shape. Sure, you can make it work, in other ways - but it feels to me like it would be clunky, artificial, unnatural. There might be reasons to use other forms, such as performance (indexing etc.), but I don't think I'd want to default to them. It's been while since I had to deal with graph types of any significant complexity, though, so it's possible I'd reconsider if actually tasked with doing so today.
Wonderful!
thanks, glad you liked it!
TBH RefCell is a ductape of the Rust. Construction with whole purpose to turn of main feature of rust and which will crash your application even if there is no real reason for that. One of the worst parts of the Rust. The two another quite equaly bad parts is a macros and weird borrow checker rule which require mutable reference to actually go out of scope instead of track the last use in that scope.
yeah i agree it is kind of like turning off the main feature of Rust, which is always preferable to avoid. It comes back to the intractable problem of accepting all memory safe programs while rejecting all unsafe programs. This intractability leads to these sorts of things being necessary. Also If you still want the safety, you can avoid the potential panics by using 'try_borrow' instead
You will panic in the end because what should add_urgent() function do of it can't get Mut borrow for some reason..
I am just feeling wrong passing a non-mutable node but in the function change its members... Is that legit?? It seems like a codesmell
I can definitely relate to this! It feels so wrong! But unfortunately that appears to be the reality of compile time memory safety checks - there are some types of programs that, despite being memory safe, need to go around those compile time safety checks 🙁
@@codetothemoon Hm... I would rather try to pass `add_one` a `&mut Node` and still use a `RefCell` for single threading or `Arc` for threaded stuff... (?) Or rather not build the data structure of the graph like that, but instead with indexes and a global common list/hash-map... There is always a way I think one just needs to make the program work differently. A data structure linked by refs (pointers) is easy but anyway gets you in trouble also in C++ where you need to make sure these pointers point always to valid memory and do not get dangling because of reallocation of some underneath data structure which holds the nodes...
So I tried to just jump in and figure these things out as I go along (with copilot by my side), rather than to stop and study it. How hard can it be? I've used C/C++ for years, I know how to code. I'll take it as it comes
Turns out that's a really good way to give yourself an aneurysm
0:39 IDK, that sounds like something you can reduce to the halting problem. Like imagine you had a program that segfaults if it discovers itself is memory safe and does nothing unsafe otherwise. That would be a contradiction.
yes, I believe checking a program (that uses manual memory management) for memory safety is roughly equivalent to the halting problem.
Tried to write a trivial LinkedList in Rust and ran head first into problems when trying to modify the list, and writing methods to modify the list. Felt like I was kinda close but never quite got there….this is definitely what I missed.
In the end, the Node struct ends up with various smart pointers on different fields. I understand we got there by construction, but I wonder if there’s a way to refactor it so that the smart pointers are just on Nodes, rather than the internal fields? Could the add_urgency() function somehow lock an entire node rather than the internal field? What would that look like?
How would you go about implementing a doubly-linked list in safe Rust?
I tried doing this once, but I got stuck when implementing a mutable iterator for it
How do you the change the style of line numbers? Which theme are you using for it ?
At around 5:00, why did you take the parameter as `node: &Node` if you want to mutate its value ? It's an immutable reference which will not allow you in general, irrespective of tree structure or the thing (Cell) you want to explain. You should have taken `node: &mut Node` and try mutating it.
The premise of the video is what to do in situations where you'd like to take a mutable reference (or ownership), but you can't because there already exist multiple borrows of that piece of data. That's where interior mutability comes in, which is basically foregoing compile time borrow checking such that you're able to mutate things that have been borrowed "immutably".
@@codetothemoon Yes. I understood that. Thanks for saying that again. It's a great video.
However, the point I was making is.. you had an opportunity to make a "full circle" argument. That is, when we beginners see a function that takes `node: &Node` as parameter, it seems immediately obvious to us that `node` cannot be modified .. and that is true, irrespective of the shape of the data-structure (that is actually wrong, as we see it later in this video itself, but that is what the beginners initially think). So logically, in order to mutate node, we have to change the parameter type to `node: &mut Node`, then we take 5 seconds and make the full circle argument that: while `&mut` does allow the function to mutate the node, we now cannot make two different nodes `b` and `c` refer to the same node `a`, as that has to be `&mut` type as well, which Rust doesn't allow. Hence, the need of `Cell` is clearly understood. We beginners tend to forget little concepts while we build our knowledge base in our mind, so such a "full circle" argument helps a lot.
You are awesome !
thank you! 😎
If I had seen this video existed a month ago, it would have saved me from spending a whole day banning my head into a wall.
thanks, I'm glad you found it valuable. it's the approach that I also would have preferred when first learning this topic.
@codetothemoon Great video! And I loved you doom emacs setup, would you mind sharing your dotfiles?
thank you! sure - here they are github.com/MoonKraken/dotfiles/tree/master/.doom.d
@@codetothemoon thanks a lot!
Can You give an use case of Cow Smart pointer?
Thanks! Excelent content
I have maybe stupid question... Why not use Cell, RefCell in adjacent vector and leave values in "natural state" ?
Great content 🙂👍
When i am at this point, i know its not the right thing to do, especially in Rust, but i just use *const T or *mut T and live with the fact, that i am in an unsafe world at this context. Using the functions from outside are still safe to use, when coded propertly
yeah I'd probably avoid this if possible, seems like it'd be foregoing some of the benefits Rust provides over C/Cpp
Would the final lock-based solution work with any generic type for the node payload? Let's say Node was a generic struct. Btw, thanks a lot for the great content!
Yes, it would. Including types that are not "Sized" (i.e. whose size is not known at compilation time)
12:38 Arc could be avoided with thread::scope
I wasn't aware of thread::scope, thanks for pointing this out!
If you're writing an application that will be deployed on Linux, it's recommended to use Mutex in general, even if you have reads in there. You should of course benchmark and try RwLock vs Mutex in your specific application, but the Mutex implementation in the Linux kernel is just so much faster than the RwLock implementation. Also, Linux' RwLock prioritizes readers over writers, so by having a constant amount of readers, you can starve your writers and they'll never get access.
On Mac, RwLock is superior to Mutex.
Idk about Windows for sure, but I think Mutex was also better than RwLock.
You could make a type that wraps the two of them and picks a solution depending on your operating system. There's probably already a crate for that.
One of the greatest benefits of Rust is that it makes you uncomfortable using worse patterns.
Sure, you can implement it as a traditional graph, but you could also implement it as an entity component system with a vector of nodes and a vector of relationships between them.
That makes it:
1. Memory safe
2. Cache-friendly
3. Without the overhead of reference counting
I'm not saying people should not use RefCell, but people should ask themselves: Do I need it? Or do I want it because that is how it's done in C++/java/...? And is there a better pattern?
How would you implement an O(n) graph traversal when relationships between nodes are stored in a vector? I also don't see how this would negate the need for interior mutability, since each node would potentially have multiple shared references to it in said vector
@@codetothemoon the relationship vector would contain indexes of the children, not references. You just need to know the index of the node. Using that index in the node vector gives you the node data, and using it in the relationship vector gives you data containing the indexes of children. This gives you O(n) traversal, i believe.
Will you be doing a Doom Emacs video ?
Would you find that valuable? If so, what editors have you tried and which is your go-to at the moment? Trying to get a sense of what folks are interested in and where they are coming from.
@@codetothemoon I've just kind of given up on emacs and went back to nvim, but it seems like a couple of people asked about the editor in the comments already and I always find those vids interesting
At the time this videos made or because of Allan Turing?
Both!
Please can you make one big huge tutorial on with projects
I'm not sure what you mean - what sort of projects?
@@codetothemoon There aren't a lot of resources for Rust, and you're one of the channels I love so much and am grateful to.
project: Any rust project that connects to data and connects to any frontend frameworks.
Is there a reason to use mutex instead of rwlock?
see stackoverflow.com/questions/50704279/when-or-why-should-i-use-a-mutex-over-an-rwlock
also in the scenario described by the video I think mutex is the better option because there isn't a need for a read lock at all
Which keys are you using. Browns?
Gateron Red Pros in a Corne! They feel so nice...
@@codetothemoon Gateron is great! Keep using it. Very satisfying.
This sort of thing makes me want to go back to cpp but there are too many other advantages to using rust.
yeah there are some hoops to jump through when you can't live by the rules of the borrow checker...
0:40 Static analysis that accepts all memory-safe programs and rejects all non-memory-safe programs is impossible as a consequence of the halting theorem, duh.
😎
One of your assumptions for the use of borrow_mut is incorrect: you assume that there would never be two borrow muts in your add one function, but the cycle case would create a borrow_mut for each loop of the cycle.
Good point - I should have mentioned this much earlier in the video, but at 12:07 I explain that this program doesn't handle cycles. As you probably know we'd have to somehow keep track of the nodes we've already visited to avoid infinite loops (or panics, as would happen in case you reference)
Enooms
Yeah I talk funny 🙃
How my rocks, would a RwLock lock, if a RwLock could lock rocs.
Hah! this would have been a fantastic line to open the video with....
i have watcched so many videos about RefCell and Rc and what all tutorials fail on is that no one shows a real world problem applying Rc or RefCell. Every one builds these simple basic graphs that is not applicable on a real life problem. My suggestion to you is that in your tutorial you set out to solve a specific problem instead, that can only be solved by using interior mutability, instead of building this fictional simple graph. Just some feedback.
What editor are you using?
Doom Emacs, you can see it in the title bar at the top of the window.
yep DOOM emacs! been using it for a few months now and I don't see myself switching to something else anytime soon.
@@codetothemoon I never used Emacs! how does it compare to neovim?
@@adibhanna Emacs has very different controls (by default), is less focused on pure text editing and comes with basically every feature imaginable, even if it doesn't make any sense.
Doom Emacs is a popular alternative configuration.
I personally hate Emacs' approach to many things but it's still one of the best editors out there.
@@adibhanna IMO you can't go wrong with either - both are fantastic. The actual coding experience pretty much identical in both (albeit key bindings may be different), most of the differences exist outside of that. Emacs is famous for something called org mode, which is sort of like markdown but much more powerful. For example, you can include code blocks of any language in your org documents, and you can run them inside the doc similar to how you would in something like a Jupyter notebook. Much of what you can do in emacs can probably be replicated in nvim plugins, but it seems like the one thing emacs will always have over *vim is that it can be run as a terminal OR GUI application - when run as the latter, you get capabilities like having multiple fonts at the same time. Something that can really make things more aesthetically pleasing and that I don't believe is possible in a pure terminal application.
in 4:51, why didn't we try `&mut Node`?
So what's the benefit of adding all this complexity?
which complexity are you referring to specifically? In the video I was hoping to show how interior mutability is simple :) I think the borrow checker definitely introduces some complexity, but complexity that is well worth the memory safety it yields in return!
@@codetothemoon I've never had an issue with memory safety, its not rocket science to keep track of allocations, especially with smart pointers. This just looks like an unnecessarily complex way of doing something really simple like accessing a vector in a struct.
@@theLowestPointInMyLife If you learn to understand complex then you would easily do it the next time without falling into traps. I think to the Author this is not complex because he is experienced in Rust. Don't you want to become like him?
@@TON-vz3pe not particularly, Im very happy with C++, use rust very minimally in the rare case it has a better library than is available in c++
@@theLowestPointInMyLife "I've never had an issue with memory safety" - good for you. Unfortunately, most programmer aren't as brilliant as you are. Microsoft said that 70% of their security bugs are memory safety issues
What is that status line?
I was really expecting the comment section to be 95% rust nerds just saying "I already knew that"
12:07 I wouldn't call it "doesn't handle". Graphs with cycles will be rejected at compile time.
How would it be able to detect a cycle at compile time? What would the error be? And how would we be able to get around that (since cycles are required for some applications) ?
@@codetothemoon Depends how you try to write it. The fact that graph edges are references to already constructed vertices requires that all edges go "upwards" (in source code). This guarantees there are no loops.
Well, I guess you could make a node mutable and add edges after construction. But we can do that only if it has no incoming edges (since those are immutable borrows).
@@codetothemoon Go around loops: like you did with Arcs. Or you could use an arena: keep all nodes in a single slice and refer to other nodes by their index instead of an actual reference
but they are slower than if there's no unsafe code at all... there's performance hit even for the simplest cases just because we can't express things clear enough... or I'm wrong..
Why fight the borrow checker instead of learning how to adopt rust's intended paradigm?
Specific program instances that are memory safe are not necessarily memory safe in general. These exceptions are among the situations that rust avoids by design.
The little "hacks" or circumstantially memory safe bits of code that appear early in the development cycle are the seeds of infinite hell down the road when system complexity increases.
What could have been "clearly memory safe" code snippets at an earlier stage can grow into messy reference violations down the road -- *exactly* the kinds of headaches from C/C++ development that played part in motivating rust's new paradigm.
So I ask: if the developer is going to outsmart the borrow checker whenever it's inconvenient, why not just code in C++?
That snippet at 0:26 is why I haven't learned C++ or Rust
Syntax is getting out of hand
I can definitely empathize with this - it does feel a bit more verbose than it needs to be. I'm hopefully that at some point explicit lifetimes won't be necessary for this scenario
I belive it is impossible to have a program that can 100% of the time guarantee that a program is memory safe or not because of the halting problem.
Say you have a function S that takes a program as input and returns true if its memory safe and false if its not
Then a function L that is leaks memory and is unsafe, so any other function that calls L will also leak and be unsafe.
Now, construct a function C that takes some source code as input, passes that source code into S, then if S returns true then it simply ends, if S returns false then it calls L and ends.
Now, pass C's source code into C:
1. C is called with C as input
2. S is called with C as input
3A. If S returns true (C is safe)
4A. C calls L
5A. L leaks memory
6A. L is unsafe
7A. C is unsafe -- Contradiction! By definition, S NEVER fails and here it predicted that C is safe
3B. If S returns false (C is unsafe)
4B. C ends
6B. C is safe -- Contradiction! By definition, S NEVER fails and here it predicted that C is unsafe
S can only return true and false but both of those values are wrong meaning that it is impossible to have a function that correctly predicts if a given source code is safe or not in all cases.
Great explanation, but as a teacher myself, I would caution that if you really want your listeners to absorb what you're communicating, then you must significantly slow down your rate of speech. There are at least two reasons for this:
1) The comprehension of new concepts takes time. Therefore, although you're delivering great content, in order for your lesson to be really effective, that content cannot be delivered at conversation speed (or here, almost machine gun speed). You must consciously slow down your rate of delivery in order to give your listeners the time to absorb and comprehend these new facts and concepts
2) If a native English speaker finds it hard to keep up, how much more challenging will it be for someone whose first language is not English? Always keep of the non-native English in mind
Great video otherwise!
thanks for the feedback, it is super helpful! i do indeed struggle with pacing, and the common feedback in this era was that things were too fast. I've corrected a bit but I could probably still do better!
You're welcome.
What helped me the most was teaching non-native English speakers. These people are perfectly intelligent, but in relation to the subject matter I was talking about, they were simply uninformed. Therefore, in order to reduce the effect of the language barrier, I had to deliberately slow my speech down and rephrase what I wanted to say in the simplest possible terms without the use of slang or idiom.
This not only helped the students comprehend the lesson, but also helped me distill my knowledge down to the core concepts.
It really is a win-win situation for everyone!
🙂
Yep, probably not going to become a Rust dev
definitely not for everyone!