Solving distributed systems challenges in Rust
ฝัง
- เผยแพร่เมื่อ 1 ก.ค. 2024
- In this stream we work through the fly.io distributed systems challenges (fly.io/dist-sys/) in Rust, and solve all the way up to challenge 3c.
You can find the code at github.com/jonhoo/rustengan.
0:00:00 Introduction
0:05:57 Maelstrom protocol and echo challenge
0:41:34 Unique ID generation
1:00:08 Improving initialization
1:31:10 Single-node broadcast
1:50:28 Multi-node broadcast and gossip
2:49:15 Don't send all values
2:57:39 Improve efficiency of gossip
Live version with chat: th-cam.com/users/liveUgf-EgWBExs - วิทยาศาสตร์และเทคโนโลยี
Still the best Rust related stuff! Really appreciate what you are doing!
I hope you do more of these, especially with diagramming. It really helped to visualize the problems you were solving.
Please do more distributed system videos like consensus algorithms, distributed databases etc
I would second this motion. Consensus algorithms in rust. Also Byzantine Fault Tolerant State Machine Replication
I too want the same
Yeah, he even has a phd in distributed systems, pls do more of it 🙏
Your content has always been fantastic, but this was my favorite video and the one I got the most out of. I think it’s because of the clear and digestibly sized problems. I don’t feel like I need to focus for a whole 3 hour block of time to fully get everything out of it. I can start and stop the video at my leisure. It also gives me a clear project I can do if I feel I want to practice what was just learned. Similar vibes to the time you worked through the rust macro book. Obviously, make the content you want to make, but you’ve definitely left me wanting more in this genre.
It'd be great to see more videos on distributed systems - like implementing RAFT and other consensus algorithms, query routing, distributed hash tables, distributed storage, etc.
I always learn many new things from your videos. Thank you so much!
Awesome as always Jon, enjoyed this thoroughly. I appreciate that you are a master at articulating your thought process and reasoning behind various changes taken (and the pros/cons at those ultimately taken).
I'd be extremely interested to see how someone like you--with your literal doctorate in distributed systems--would build up a similar challenge.
Now I have something to watch through easter! Ty Jon
Funny how enthusiastic you are about "Rustengan" :D
I APPROVE!
Amazing. Looking forward to more similar contents.
Appreciate your rust coding video. I am a beginner for the rust and I really learn really practical rust knowledge by following your coding line by line. Thank you !!! Wish you a good life.
Thank you for this amazing walkthrough!
Thanks for the stuff, high high quality, not sure what the 20 dislikers were expecting from this video.
Thanks for sharing! Just got interested in learning Rust again and found your book on Kindle and via that your TH-cam!
It was so satisfying to see you pass the 3c without changing anything.
Great video as usual!
i hope you provide us a video which is continue to solving rest of the problems
Hey! I remember Jepsen for their Consistency Models page! It was really helpful to get me started on distributed systems and understanding data consistency, along with potentially building hand-wavy reasoning/logical system around it! Happy to see fly-io take the author far far away onto actual courses for distributed systems!
Great vid, loved it!
Glad to see you appreciate DnD 🐉🙌🏾
Never took Jon for a Naruto guy
rustengan was an epic choice!
I literally asked the question “is there any material for distributed systems from the Rust perspective?” On LinkedIn, and was told that I was asking a stupid question! Can you believe that?
Yes. It's linked in. It's like the worst parts of Reddit and Twitter combined
You can ask questions on LinkedIn? Thought it was just for listing your certs and degrees.
Thank you so much for doing this ❤
Very cool video! Thanks :)
Would love to see the next challenges
One thing that would have been interesting to see is what kind of an effect changing the gossip interval has on the latency to consistent, in combination with the 10% extra items sent per gossip!
Common need something like these, Bring rust to Backend stuff !!
Love this content. More of this.
Hi, I woke up and this video was on. Nice stuff
In relation to distributed (heterogeneous, asynchronus) systems, you might be interested in Petrinets. If you are actually building a system, this gives you a tool for catching some of the problems before you design them in.
It is a bit like the relationship between ER modelling and database implementaion.
I'm using Rust for my 'nodes' (made from 'Place' 'Transition' and 'Arc' in PN language) and Rust fits perfectly.
Very helpful, thanks。
I think your gossip optimization has a bug - the topology doesn't need to be symmetric, so a node doesn't necessarily know about messages that nodes in its neighborhood know about.
Had a blast, thanks!
I like that this guy looks up cool words to name his cargo
He does this regularly and comes up with some really interesting ones. Defenestration was one he picked (I think). Do you realise how hard it is to name a lib well?
"There are only two hard things in Computer Science: cache invalidation and naming things."
@@bsodmike I think you forgot off by one errors ;P
@@TNH91that and runtime segfaults!
39:13 OMG the amount of times I've had this problem, and being a beginner had no clue how to fix, so I would end up trying to shuffle lines around or clone or god knows what, when dereferencing was all that's needed...
This is why I love your vids, they contain so much helpful practical knowledge, they are just a joy to watch.
Man, I missed your content! :)
Great video as always! I think you are still seeing big Gossips in the end because you are making an assumption that all Gossiping happens bi-directionnally. If n1 Gossips to n2 but n2 never gossips to n1, n1 will always want to send the full thing as it doesn't knonw that n2 now knows the stuff it sent to it previously. That's where the GossipOk messages might come in handy so that n2 can tell n1 it saw the messages n1 Gossiped to it.
Hii John,
Love your work !! Hugh respect!!
What is your take on decisions made by The Foundation on "Rust" trademark and logo usage ??
I felt little disappointed..
Thanks for the inspiration
I loved this stream ! Very interesting.
Have you in mind to do an other stream focusing on consensus protocol, traditional leader based (Paxos or Raft) or even new leaderless alternatives (Epaxos, Atlas, Tempo, Accord) ? That would be awesome !
Thank you so much!
39:09 when even Jon get stuck with the borrow checker... i think i am safe
Hi Jon, amazing as always!
I had an idea for sync across nodes - can you form a probabilistic data structure(like a bloom filter) from your node’s values and send it to neighbours, so they can test for known values, to figure out the data that we might be interested in?
Sure, there may be false positives(we don’t know about the value, but the bloom filter says we do), but I feel like they can be dealt with.
Or use a lossy hash table for inverse behaviour(only false negatives)
It's a cool idea - worth trying out!
It's really an excellent stuff.
3:01:30 instead of random you can use a module counter so you send n, n+1 and n+2 additional alreadyknowns. Or you add the first x of all already now since in the next message they won’t be contained
This is gold.
When you defined Body with a field rest: B and then rejected it, you could actually keep Body and use Body for two-step deserialization and Body for one step deserialization if you happen to know B should be. There's a slight bit of weirdness in the second case you end up with a redundant string but you could make a type with two fields, a hashmap of serde_json::Values and a string for the type: struct DynBody { ty: String, fields: HashMap}. But deserialization to a KnownType might need a little bit of custom code to check the type field and reject if it's not the expected type.
Just some random thoughts.
There is a crate that implements fully RPC/KV spec if you need any kind of support to fully focus the most interesting challenges - Kafka / Tx.
Hey Jon,
I think it would very interesting to implement eye tracking. For example, at 22:39, you tab back to the serde_json docs for just a moment, then find the answer of what you're looking for. Seeing exactly what you looked at on the page would clue us in to your thought process, and add another layer.
-Tom
That would be neat but I don’t know how easy eye tracking is to set up without dedicated hardware
1:01:27 - hehe, Beavis, he said "generic over some ASS"
I kinda understand the gist of the code, but i feel like it's still way over my level at the moment. Awesome content anyway :)
Amazing !!!
Amazing content, Jon!
I have a little (maybe dummy) question: At the end you added an eprintln! for debug the size of messages sent. Why that operation didnt break the reading from stdout? How Maelstrom knows what is a message response and what is a log from stdout? Sorry if I missed something 😅
Ah, so, `eprintln!` (note the `e`) prints to STDERR, not STDOUT. It's `println!` (no `e`) that prints to STDOUT. Since Maelstrom only checks STDOUT, you can safely use STDERR for other kind of printing/logging.
@@jonhoo Oh, that makes sense! Thanks! Keep up the excellent content!
Great work! However, just wonder why not just return a Result, and let nodes forget serialization stuffs?
Nice!
Thanks. Can you reupload eng subscription version?
Ths s so cool !!
hi, i was wondering on 39:14 in terms of ownership and borrowing is there any difference between
`&mut *output`
changing the function param to
`... mut output: &mut StdoutLock...`
and just call
`... serde_json::to_writer(&mut output, &reply).context("serialize response")...`
anyway great content, thanks!
Hi Jon, awesome stream as always! Got a question about the "improving efficiency" part. Why dont you just keep track of which messages you've already sent to the peer and stop sending those? Is it because you are not sure whether those have arrived or failed due to network conditions?
That's exactly right!
@@jonhoo In this case, implementing a perfect link to mitigate this problem wouldnt benefit you in terms of performance ( not to mention that the program would have to be async ) right ? Watching you program/explain these distributed system challenges is giving me a new love for this topic!
@@guilhermesoares7857 really perfect link can never exist
In the echo example, would `println!("{}", serde_json::to_string(&reply)?);` have worked instead of explicitly working with the `StdoutLock`?
I only remembered Rasengan after 3 hour of video :)))))
Hey I know this is not in the topic of Rust exactly but can you post a video of your coding setup? (ide, vim scripts ...)
Hi Jon, thank you for the great streaming.
At around 2:23:49 I notice there are proc macro errors after you gd into other crates. Are there ways to avoid those?
I believe no. probably changes in rust-analyzer needed.
Very nice content. I think it's the first time I watch a live for so long. Does anyone know someone doing similar content in go?
Gosh this setup, I had nothing predownloaded. That took a while
I'm not sure what happened. I created the project like you and my rust-analyzer broke. I debuged for 3 hours. Deleted the repo. Recreated it without the --bin and it works.
Not sure that happened there, but I'll write that in case it helps anyone else
Working on 4. Im not sure I understand but they provide a service for kv storage so to replicate this in Rust it means I would just use a database like Redis right?
Does anyone know if there is still a suggestions website for future streams?
whats that Panda symbol that shows up next to payload: Payload popup ? what vim plugin is this ? and whats the on the go errors from ?
For the gossip section getting longer was there also the possibility based on provided topology that ‘n1’ talks to ‘n2’ but ‘n2’ doesn’t talk to ‘n1’? Or were they always bi directional without the need for the sync?
love the name
What plugin is that in vim? What's a good code completion plugin?
What drawing program is being used?
Do you have any interest in trying to implement a chess engine in Rust? I do and watching you do it would be very instructional.
Can someone tell me what drawing tool is being used here?
Of all the streams for me to miss....
Best quality content at 7:30 😅
Rust devil!
What I'm wonder is, what's there to gain by making everything this generic? I love the generic stuff in this video, as it's a particularly weak point in my skillset, so any practicing on that is great, but every time I am faced with problems I need to solve i just so rarely find myself going for a generic solution - because it's so rare that it actually needs that _level_ of re-use (and in generics case it's not so much re-use, there's some, but not entirely, like in the case of C++ templates for instance).
r#type? we have a lot of that.
I don’t like the “reverse dst and src” logic in responses - because you’re given your node id in the init, my assumption would be that you might see a message intended for somebody else and should have some logic for ignoring it (so you should check if the dest field is actually your own id before replying to it).
I guess the person who wrote the “you’re still reversing dst and src” comment in the chat at some point had something similar in mind. Since you know your node_id you should use that as the src of the response, instead of just blindly reversing the message’s fields.
Of course if maelstorm only delivers messages to the intended destinations and there’s no multicast, it doesn’t really matter.
Would you continue this challenge in future?
Missed opportunity for “chidori”😅
In the unique ID generation it looks like you did a lot of work to capture your own ID from the "init" message and store in state. But... don't you get your own node ID with every message, including the "generate" message itself? Couldn't you have just used `input.dst`?
This is as far as I got in the video, so maybe this will be addressed later on.
Oh, yeah, I suppose you could just grab out the dst field in the message! That only works under the assumption that there are no multicast messages though :p The more relevant response is that we need more of the init state for later challenges, so we might as well construct the flow for grabbing init state properly anyway.
@@jonhoo Fair enough. Great topic, btw! I really enjoyed the video this far.
(Noob Q) why are stack traces in Java ? Tyia
Jon, what you think about create new video about Tokio and same field stuff? In many jobs requires Tokio, and still in youtube no cool video about Tokio and async in Rust? Pls))
I call it seeeered
Be careful Jon, we don't want you sued... 😒
No! You will not use Erlang!
Won't the unique challenge fail? You ran it with only 1 node. Are the different nodes always creating unique ID:s regardless of if they have no knowledge of what the other nodes are spitting back to Maelstrom?
missed opportunity to name it Charybdis, for completing charybDistributed system challenges